• ETL
  • Quick Start Guide

Quick Start Guide

Create Your First ETL Chain in 10 Minutes

This guide walks you through creating a simple ETL chain that reads data, transforms it, and writes the results.

Prerequisites

  • Access to ETL Management interface (requires mod-etl privilege)
  • Access to ETL Designer interface (requires mod-etl-designer privilege)

Step 1: Create a Chainset (2 minutes)

  1. Navigate to the ETL Management page
  2. Click the Add button in the upper right corner
  3. In the “Add ETL Chainset” dialog:
  • Name: Enter “MyFirstChainset”
  • Copy From: Select “Blank ETL”
  1. Click OK

The ETL Designer will open automatically with your new chainset.

Step 2: Create a Chain (1 minute)

  1. In the Chains panel (left side), click + Add Chain
  2. In the “Add Chain” dialog:
  • Name: Enter “HelloWorld”
  • Copy From: Select “Blank Chain”
  1. Click OK

The chain is now selected and ready for adding steps.

Step 3: Add Steps (4 minutes)

Add a JSON Record Step (creates sample data)

  1. In the Steps panel (middle), click + Add Step
  2. In the step selection dialog:
  • Search for “JSON Record” or navigate to JSON category
  • Select JSON Record
  • Click Add
  1. In the “Edit Step” dialog:
  • Step Name: Keep as “JSON Record” or rename to “Create Sample Data”
  • JSON: Enter the following: json { "name": "Alice", "age": 30, "city": "New York" }
  1. Click Save

Add a String Upper Step (transforms data)

  1. Click + Add Step again
  2. Search for “String Upper” or navigate to String category
  3. Select String Upper and click Add
  4. In the “Edit Step” dialog:
  • Step Name: Keep as “String Upper” or rename to “Uppercase Name”
  • Field: Enter “name”
  • To Field: Enter “name” (to replace the original)
  1. Click Save

Add an integer.Add Step (modifies data)

  1. Click + Add Step again
  2. Search for “Integer Add” or navigate to Number > Integer category
  3. Select Integer Add (integer.Add) and click Add
  4. In the “Edit Step” dialog:
  • Step Name: Keep as “Integer Add” or rename to “Increment Age”
  • Field: Enter “age”
  • Value: Enter “1”
  • To Field: Enter “age”
  1. Click Save

Step 4: Run the Chain (1 minute)

  1. In the Steps panel, click the Run Steps button (play icon)
  2. The chain executes and results appear in the Results panel (right side)
  3. Click on the run record to see the output

Expected Output:

{
  "name": "ALICE",
  "age": 31,
  "city": "New York"
}

Step 5: Test with Different Input (2 minutes)

Create a Test Input

  1. In the Steps panel, find the Test Input dropdown
  2. Click the More Actions button (three dots) next to it
  3. Select Add Test
  4. In the “Add Test” dialog:
  • Name: “Multiple People”
  • Description: “Test with array of people”
  • Test JSON: Enter: json [ {"name": "Bob", "age": 25, "city": "Boston"}, {"name": "Carol", "age": 35, "city": "Chicago"}, {"name": "Dave", "age": 40, "city": "Denver"} ]
  1. Click OK

Run with Test Input

  1. Select “Multiple People” from the Test Input dropdown
  2. Click Run Steps
  3. View the results - you should see three records, each with uppercase name and age incremented by 1

What You’ve Learned

✅ Created a chainset - Container for your chains
✅ Created a chain - Sequence of transformation steps
✅ Added steps - Individual operations (JSON Record, String Upper, integer.Add)
✅ Ran the chain - Executed and viewed results
✅ Used test inputs - Tested with different data

Understanding What Happened

  1. JSON Record step created sample data (one or more records)
  2. String Upper step transformed the “name” field to uppercase
  3. integer.Add step incremented the “age” field by 1
  4. Records streamed through the steps sequentially
  5. Results appeared in the Results panel

Next Steps

Add More Transformations

Try adding these steps to your chain:

  • String Concatenate - Combine name and city into a “fullInfo” field
  • integer.Multiply - Calculate age in months
  • structure.AddFields - Add new fields like { "processed": true }
  • Filter - Only keep records where age > 30

Save to Database

Add a MongoDB Writer step to save results:

  1. Add MongoDB Writer step at the end of your chain
  2. Configure:
  • Pool: Select your database connection
  • Collection: Enter a collection name (e.g., “people”)
  1. Run the chain to save records to MongoDB

Create Reusable Chains

  1. Create a new chain called “FormatPerson”
  2. Move the String Upper and integer.Add steps to this chain
  3. In “HelloWorld” chain, replace those steps with a Chain Call step
  4. Configure Chain Call to call “FormatPerson”
  5. Run to see the same results using composition

Explore Step Categories

Open the Dictionary (button in upper right) to browse available steps:

  • Array - Work with arrays and lists
  • String - Text manipulation
  • Number - Mathematical operations
  • Date - Date and time operations
  • MongoDB - Database operations
  • Structure - Control flow and composition
  • And 60+ more categories…

Common Next Tasks

Troubleshooting

Chain doesn’t run? - Check that all steps have required parameters filled in - Look for error messages in the Results panel - Enable debug logging on steps to see detailed execution

Unexpected results? - Enable debug logging to see input/output for each step - Check step parameters are correct - Verify field names match your data

Can’t find a step? - Use the search bar in the Add Step dialog - Browse categories in the Dictionary - Check step help for parameter details

Tips for Success

💡 Start simple - Begin with a few steps, add complexity gradually
💡 Test frequently - Run the chain after adding each step
💡 Use test inputs - Create multiple test cases for different scenarios
💡 Enable debug logging - See exactly what each step does
💡 Name steps clearly - Use descriptive names like “Validate Email” not just “Validation”
💡 Break into chains - Use Chain Call for reusable logic
💡 Document with descriptions - Add descriptions to chains and test inputs

Congratulations! You’ve created your first ETL chain. Continue exploring to build powerful data transformation pipelines.

Next: ETL Management