Configuration Reference

Chainset Configuration

Chainset Properties

Property	Description	Required	Default

| Name | Unique identifier for the chainset | Yes | - | | Enabled | Whether the chainset is active | No | true | | Roles | Roles that can access the chainset | No | [] | | Workgroups | Workgroups associated with the chainset | No | [] | | Imports | Other chainsets to import | No | [] |

Creating a Chainset

Via ETL Management UI: 1. Click “Add” button 2. Enter unique name 3. Optionally copy from existing chainset 4. Click “OK”

Via Upload: 1. Click “Upload” button 2. Select JSON file (.chainset.json) 3. Enter name or select “Overwrite” 4. Click “OK”

Enabling/Disabling Chainsets

Enabled Chainset: - Available to users with appropriate roles - Can be executed manually, by scheduler, workflows, datasets - Indicated by green checkmark in management UI

Disabled Chainset: - Not available to non-owners - Owner can still access via ETL Designer - Cannot be executed by scheduler, workflows, datasets - Indicated by red cross in management UI

To Enable/Disable: 1. Click “Edit” icon for chainset 2. Check/uncheck “Enabled” checkbox 3. Click “Save”

Chain Imports

Chainsets can import other chainsets to reuse their chains.

Configuring Imports: 1. Click “Edit” icon for chainset 2. In “Import Chainsets” section, search for chainset 3. Select chainset from dropdown 4. Click “Save”

Important Rules: - Avoid circular dependencies (A imports B, B imports C, C imports A) - Imported chains can be called using Chain Call steps - Changes to imported chainset affect all importers - Import relationships are not versioned

Example Structure:

CommonUtilities (no imports)
  - FormatAddress
  - ValidateEmail
  - SendEmail

CustomerManagement (imports CommonUtilities)
  - ImportCustomers (can call FormatAddress, ValidateEmail)
  - ExportCustomers (can call SendEmail)

OrderProcessing (imports CommonUtilities)
  - ProcessOrders (can call ValidateEmail, SendEmail)

Role-Based Access Control

Access Levels

Level	Capability	Required
Read Access	View and execute chains	User’s role in chainset roles OR ownership
Write Access	Modify chainset and chains	Ownership
Management	Enable/disable, set roles	Ownership

Assigning Roles

Single Chainset: 1. Click “Edit” icon for chainset 2. In “Roles” section, select roles 3. Click “Save”

Multiple Chainsets: 1. Select chainsets (checkboxes) 2. Click “More Actions” → “Set Roles” 3. Select roles 4. Click “OK”

Ownership

Chainset creator is the owner
Owner has full access regardless of enabled/disabled status
Owner can modify chainset even if disabled
Ownership cannot be transferred via UI (requires database update)

Workgroup Integration

Workgroup Association

Workgroups provide organizational grouping and filtering.

Assigning Workgroups:

Single Chainset: 1. Click “Edit” icon for chainset 2. In “Workgroups” section, select workgroups 3. Click “Save”

Multiple Chainsets: 1. Select chainsets (checkboxes) 2. Click “More Actions” → “Set Workgroups” 3. Select workgroups 4. Click “OK”

Workgroup Filtering

In ETL Management, use the “Workgroup” dropdown to filter: - Shows only chainsets associated with selected workgroup - Useful for organizing chainsets by department, project, or client

Auto-Assignment

When a workgroup is selected in the filter: - New chainsets automatically get that workgroup assigned - Helps maintain organization

Chain Configuration

Chain Properties

Property	Description	Required	Default

| Name | Unique name within chainset | Yes | - | | Description | Documentation of chain purpose | No | "“ | | Group | Organizational group | No | - | | Icon | Visual icon | No | ”chain“ | | Color | Visual color | No | ”Transparent" | | Steps | Ordered list of steps | Yes | [] | | Test Inputs | Sample data for testing | No | [] |

Chain Groups

Groups organize chains within a chainset.

Creating Groups: 1. Select chains (checkboxes) 2. Click “More Actions” → “Set Group” 3. Enter group name 4. Click “OK”

Filtering by Group: - Use “Group” dropdown in Chains panel - Shows only chains in selected group

Group Rules: - Each chain belongs to at most one group - Moving chain to new group removes from old group - Groups are purely organizational (no execution impact)

Step Configuration

Step Properties

Property	Description	Required	Default

Step Parameters

Each step type has different parameters. See step help in Dictionary for details.

Common Parameter Patterns: - Field names - Reference fields in incoming records - Values - Literal values or expressions - Queries - Database queries with variable substitution - Expressions - Formulas using ${fieldName} syntax

Variable Substitution

The string.Substitute step rewrites a string field in-place, replacing each ${fieldName} placeholder with the value of that field from the current record. This is the mechanism ETL chains use to inject runtime values into strings — most commonly MongoDB aggregation pipelines.

Syntax: ${fieldName} inside a string field value

Example: building a dynamic MongoDB query

1. mongodb.AggregationDefinition
   Pool: your-pool-name
   Collection: customers
   Aggregation: [{"$match": {"customerId": "${customerId}", "date": "${reportDate}"}}]

2. string.Substitute
   Field: aggregation

3. mongodb.Reader

If the incoming record is { "customerId": "C12345", "reportDate": "2026-03-03" }, then after step 2 the aggregation field becomes:

[{"$match": {"customerId": "C12345", "date": "2026-03-03"}}]

Step 3 (mongodb.Reader) executes the substituted aggregation and emits one output record per result document.

When the aggregation pipeline contains no ${placeholders} (static query), the string.Substitute step can be omitted.

Test Input Configuration

Test Input Properties

Property	Description	Required
Name	Unique name for test	Yes
Description	Purpose of test	No
Test JSON	Sample input data	Yes

Creating Test Inputs

Select chain
Click “Test Input” dropdown → “More Actions” → “Add Test”
Enter name, description, and JSON
Click “OK”

Test Input Best Practices

Create multiple test inputs per chain
Cover happy path, edge cases, and error cases
Use descriptive names: “Valid Order”, “Missing Customer”, “Large Dataset”
Include comments in description explaining what test validates
Keep test data realistic but minimal

Using Test Inputs

Select test from “Test Input” dropdown
Click “Run Steps”
Chain executes with test data as initial input
View results in Results panel

Export/Import Configuration

Export Chainset

Single Chainset: 1. Click “More Actions” → “Download” 2. Chainset downloads as <name>.chainset.json

Multiple Chainsets: 1. Select chainsets (checkboxes) 2. Click “More Actions” → “Download” 3. Enter zip file name 4. Chainsets download as zip file

Import Chainset

Single File: 1. Click “Upload” button 2. Select .chainset.json file 3. Enter name or select “Overwrite” 4. Click “OK”

Zip File: 1. Click “Upload” button 2. Select .zip file containing multiple chainsets 3. Click “OK” 4. View upload results (success/failure for each)

File Format

Chainset JSON structure:

{
  "name": "MyChainset",
  "enabled": true,
  "chains": [
    {
      "name": "MyChain",
      "description": "Chain description",
      "steps": [
        {
          "stepType": { "id": "json.Record" },
          "name": "Create Data",
          "json": { ... }
        }
      ],
      "testInputs": [
        {
          "name": "Test 1",
          "description": "Test description",
          "testJSON": { ... }
        }
      ]
    }
  ]
}

Database Configuration

Database Connections

ETL chains use database connection pools configured in Ambience.

MongoDB Connections: - Configured in application.conf - Referenced by pool name in steps - Connection pooling managed automatically

In the ETL Designer, available pools are shown as a dropdown in each step dialog. In examples throughout this documentation, your-pool-name is used as a placeholder for whichever pool appears in your deployment.

JDBC Connections: - Configured in application.conf - Referenced by connection name in steps - Support various databases (PostgreSQL, MySQL, Oracle, etc.)

Privileges

Required Privileges

Interface	Privilege	Description
ETL Management	`mod-etl`	Access management interface
ETL Management (modify)	Ownership	Modify/delete chainsets
ETL Designer	`mod-etl-designer`	Access designer interface
ETL Designer (read)	Role access OR ownership	View and run chains
ETL Designer (write)	Ownership	Modify chains and steps

Checking Privileges

Users see only chainsets they have access to: - Owned chainsets (always visible) - Enabled chainsets with user’s role assigned - Disabled chainsets only if owner

Performance Configuration

Job Queue

ETL chains execute through a job queue: - One job at a time per chainset - Multiple chainsets can run concurrently - Long-running jobs don’t block other chainsets

Implications: - Multiple users can trigger same chainset → jobs queue - Scheduler jobs queue with manual executions - Consider chainset design for concurrent needs

Optimization Settings

Chain-Level: - Use efficient step patterns - Filter early, aggregate late - Minimize record expansion (1 => N operations)

Step-Level: - Use specific database queries - Create appropriate indexes - Use projections to limit fields - Batch operations when possible

Backup and Recovery

Backup Strategy

Regular Exports: - Export chainsets regularly - Store in version control - Tag releases and versions - Document changes

Automated Backup: - Schedule ETL chain to export all chainsets - Store exports in backup location - Retain historical versions

Recovery

Restore Chainset: 1. Locate backup file (.chainset.json) 2. Click “Upload” in ETL Management 3. Select file 4. Choose “Overwrite” if replacing existing 5. Click “OK”

Restore Multiple Chainsets: 1. Create zip file of chainset JSON files 2. Click “Upload” in ETL Management 3. Select zip file 4. Click “OK” 5. Review upload results

Environment Management

Development/Test/Production

Recommended Approach:

Development Environment:

Create and test chainsets
Use test databases
Iterate rapidly

Test Environment:

Import from development
Test with realistic data
Validate integrations

Production Environment:

Import tested chainsets
Use production databases
Monitor execution

Migration Process

Export chainset from source environment
Review and update database references if needed
Import to target environment
Update roles and workgroups if needed
Test thoroughly before enabling
Enable and monitor

Configuration Differences

When moving between environments, update: - Database connection names - API endpoints - File paths - Email recipients - Scheduler configurations

Troubleshooting Configuration Issues

Chainset Not Visible

Check: - Chainset is enabled - User has appropriate role OR is owner - Workgroup filter not hiding chainset

Cannot Modify Chainset

Check: - User is owner (only owners can modify) - Chainset is not locked by another user

Chain Fails to Execute

Check: - All step parameters are valid - Database connections are configured - Required privileges are granted - Test with debug logging enabled

Import Fails

Check: - JSON file format is correct - Chainset name is unique (or Overwrite selected) - No circular dependencies in imports - File is not corrupted

Next Steps

ETL Management - Manage chainsets
ETL Designer - Design chains
Best Practices - Design guidelines
FAQ - Common questions

Next: Glossary