ETL Integration

The Workflow module provides deep integration with the ETL system, allowing workflows to execute data transformations at various points in the workflow lifecycle and enabling ETL chains to create and control workflow instances.

Overview

Integration Points: - OnEntry - Execute ETL chain when entering a state - OnExit - Execute ETL chain when leaving a state - OnTransition - Execute ETL chain during state transition - Guard - Use ETL chain to evaluate transition conditions - ETL Steps - 18 workflow-specific ETL steps for workflow operations

Workflow ETL Steps Reference

The workflow module provides the following ETL steps for use in ETL chains:

Instance Management

workflow.CreateInstance

Creates a new workflow instance.

Configuration:

{
  "workflowId": "leave-request"
}

Input: BSON document with initial data (optional)

Output: BSON document with complete workflow instance state

Use Cases: - Create workflow from ETL pipeline - Batch create multiple instances - Automated workflow triggering

Example ETL Chain:

1. Source: MongoDB query for new employee records
2. Transform: Map employee data to workflow format
3. workflow.CreateInstance: workflowId = "employee-onboarding"
4. Sink: Log created instances

workflow.SendEvent

Sends an event to an existing workflow instance.

Configuration:

{
  "eventName": "approve",
  "paramsField": "eventData"
}

Input: BSON document containing workflow instance state and event parameters

Output: BSON document with updated workflow instance state

Use Cases: - Progress workflows from ETL pipelines - Automated event triggering based on external data - Batch processing of workflow events

Example:

1. Source: Load workflow instances
2. Filter: Select instances pending timeout
3. Transform: Add event parameters
4. workflow.SendEvent: eventName = "timeout"
5. Sink: Update instances

workflow.GetCurrentState

Retrieves the current state of a workflow instance.

Configuration:

{}

Input: BSON document with workflow instance ID

Output: BSON document with complete instance state

Use Cases: - Query workflow state in ETL chains - Conditional logic based on workflow state - Reporting and analytics

workflow.GetAllInstances

Retrieves all instances of a workflow.

Configuration:

{
  "workflowId": "leave-request"
}

Input: None (or filter criteria)

Output: Stream of BSON documents, one per instance

Use Cases: - Batch processing of workflow instances - Reporting and analytics - Data export

Data Manipulation

workflow.AddToElxPublic

Adds fields to the elxPublic section of workflow instance.

Configuration:

{
  "field": "newField",
  "value": "newValue"
}

Input: Workflow instance state

Output: Updated workflow instance state

Use Cases: - Add calculated fields - Enrich instance data - Update display information

workflow.AddToElxPrivate

Adds fields to the elxPrivate section of workflow instance.

Configuration:

{
  "field": "internalField",
  "value": "internalValue"
}

Input: Workflow instance state

Output: Updated workflow instance state

Use Cases: - Store sensitive data - Add server-side processing data - Track internal state

workflow.MergeToElxPublic

Merges data into the elxPublic section.

Configuration:

{
  "sourceField": "mergeData"
}

Input: Workflow instance state with merge data

Output: Updated workflow instance state

Use Cases: - Bulk update public data - Merge form submissions - Update multiple fields at once

workflow.MergeToElxPrivate

Merges data into the elxPrivate section.

Configuration:

{
  "sourceField": "privateData"
}

Input: Workflow instance state with merge data

Output: Updated workflow instance state

Use Cases: - Bulk update private data - Store calculation results - Update internal state

workflow.RemoveElxPrivate

Removes fields from the elxPrivate section.

Configuration:

{
  "field": "fieldToRemove"
}

Input: Workflow instance state

Output: Updated workflow instance state

Use Cases: - Clean up temporary data - Remove sensitive data after processing - Data minimization

Navigation Control

workflow.SetElxRedirect

Sets the elxRedirect field to control browser navigation.

Configuration:

{
  "redirectUrl": "/confirmation"
}

Input: Workflow instance state

Output: Updated workflow instance state with redirect

Use Cases: - Navigate to confirmation page after submission - Redirect to dashboard after approval - Custom navigation flows

Example:

OnEntry chain for "Approved" state:
1. workflow.SetElxRedirect: redirectUrl = "/leave-requests/approved"
2. SendEmail: Notify employee of approval

Guard Evaluation

workflow.GuardEndPoint

Marks the evaluation point for guard conditions. Must be present in guard chains.

Configuration:

{}

Input: Workflow instance state

Output: Same as input (pass-through)

Use Cases: - Mark end of guard evaluation - Required in all guard chains

Example Guard Chain:

1. GetCurrentState: Load instance
2. Transform: Check approval rules
3. Conditional: If rules pass, continue; else fail
4. workflow.GuardEndPoint: Mark evaluation complete

Utilities

workflow.LookupStates

Provides state lookup utilities for workflow instances.

Configuration:

{
  "stateMachineId": "main"
}

Input: Workflow instance state

Output: State information

Use Cases: - Query current state - Check state machine status - Conditional logic based on state

workflow.CleanInstances

Maintenance operation to clean up workflow instances.

Configuration:

{
  "workflowId": "leave-request",
  "olderThan": "90d"
}

Input: None

Output: Cleanup results

Use Cases: - Archive old instances - Delete completed workflows - Database maintenance

workflow.DuplicateWorkflow

Duplicates a workflow definition.

Configuration:

{
  "sourceWorkflowId": "template-workflow",
  "newWorkflowName": "New Workflow"
}

Input: None

Output: New workflow definition

Use Cases: - Create workflow templates - Clone workflows for different departments - Workflow versioning

workflow.CreateInstanceField

Creates or updates a specific field in a workflow instance.

Configuration:

{
  "fieldPath": "elxPublic.status",
  "value": "Pending"
}

Input: Workflow instance state

Output: Updated workflow instance state

Use Cases: - Targeted field updates - Nested field manipulation - Dynamic field creation

Common ETL Patterns

Pattern 1: Send Notification on State Entry

Scenario: Send email when workflow enters “Pending Approval” state

OnEntry Chain for “Pending Approval” State:

1. GetCurrentState
   - Loads current instance state

2. Transform: Extract Email Data
   - Get employee email from elxPublic.employeeEmail
   - Get manager email from elxPublic.managerEmail
   - Format email subject and body

3. SendEmail
   - To: managerEmail
   - Subject: "Leave Request Pending Approval"
   - Body: Include employee name, leave dates, reason

4. workflow.AddToElxPrivate
   - field: "notificationSent"
   - value: current timestamp

Pattern 2: Validate Data Before Transition

Scenario: Check approval rules before allowing approval transition

Guard Chain for “approve” Transition:

1. GetCurrentState
   - Load instance state

2. Transform: Extract Approval Data
   - Get leave days from elxPublic.days
   - Get employee grade from elxPublic.employeeGrade
   - Get approver role from context

3. Conditional: Check Approval Authority
   - If days <= 3: Any manager can approve
   - If days > 3 and days <= 7: Senior manager required
   - If days > 7: Director approval required
   - Check if current user has required role

4. workflow.GuardEndPoint
   - If checks pass, transition fires
   - If checks fail, transition blocked

Pattern 3: Calculate Derived Values

Scenario: Calculate total cost when workflow enters “Approved” state

OnEntry Chain for “Approved” State:

1. GetCurrentState
   - Load instance state

2. Transform: Calculate Cost
   - Get daily rate from employee record
   - Get leave days from elxPublic.days
   - Calculate total cost = daily rate * days

3. workflow.MergeToElxPrivate
   - Add calculated cost to private data
   - Add calculation timestamp
   - Add calculation user

4. Transform: Update Budget System
   - Call external API to reserve budget
   - Get budget confirmation number

5. workflow.AddToElxPrivate
   - field: "budgetConfirmation"
   - value: confirmation number

Pattern 4: Update External System

Scenario: Update HR system when leave is approved

OnTransition Chain for “approve” Transition:

1. GetCurrentState
   - Load instance state

2. Transform: Prepare HR Update
   - Extract employee ID
   - Extract leave dates
   - Format for HR system API

3. HTTP Request: Update HR System
   - POST to HR API
   - Include leave details
   - Get confirmation

4. Conditional: Check HR Response
   - If success: Continue
   - If failure: Log error and notify admin

5. workflow.MergeToElxPublic
   - Add HR confirmation number
   - Update status to "Synced with HR"

Pattern 5: Timeout Handling

Scenario: Escalate if no approval within 3 days

OnAfter Configuration (Pending Approval State): - Units: Days - Time: 3 - Event: escalate

OnTransition Chain for “escalate” Transition:

1. GetCurrentState
   - Load instance state

2. Transform: Prepare Escalation
   - Get manager email
   - Get senior manager email
   - Get leave details

3. SendEmail: Escalation Notice
   - To: senior manager
   - CC: manager
   - Subject: "ESCALATION: Leave Request Pending"
   - Body: Include all details + overdue notice

4. workflow.MergeToElxPublic
   - Add escalation flag
   - Add escalation timestamp
   - Increment escalation count

5. Conditional: Check Escalation Count
   - If count >= 3: Auto-approve or auto-reject
   - Else: Wait for next escalation

Pattern 6: Batch Create Instances

Scenario: Create workflow instances for all new employees

ETL Chain:

1. Source: MongoDB
   - Query: New employees from HR database
   - Filter: startDate = today

2. Transform: Map to Workflow Format
   - Map employee fields to elxPublic
   - Add onboarding checklist items
   - Set initial state data

3. workflow.CreateInstance
   - workflowId: "employee-onboarding"
   - Creates one instance per employee

4. Transform: Extract Instance IDs
   - Get created instance IDs
   - Map to employee records

5. Sink: Update Employee Records
   - Store workflow instance ID in employee record
   - Mark as "onboarding initiated"

Pattern 7: Conditional Routing with Guards

Scenario: Route to different states based on amount

Guard Chain for “high-value” Transition:

1. GetCurrentState
   - Load instance state

2. Transform: Check Amount
   - Get amount from elxPublic.amount
   - Get threshold from configuration

3. Conditional: Amount Check
   - If amount > threshold: Success
   - Else: Fail

4. workflow.GuardEndPoint
   - Marks evaluation complete

Guard Chain for “standard” Transition:

1. GetCurrentState
   - Load instance state

2. Transform: Check Amount
   - Get amount from elxPublic.amount
   - Get threshold from configuration

3. Conditional: Amount Check
   - If amount <= threshold: Success
   - Else: Fail

4. workflow.GuardEndPoint
   - Marks evaluation complete

Result: High-value requests go to senior approval, standard requests go to manager approval.

Using Workflows in ETL Chains

Triggering Workflows from ETL

ETL chains can create and control workflow instances:

Example: Process Incoming Orders

1. Source: Order Queue
   - Read new orders from message queue

2. Transform: Validate Order
   - Check inventory
   - Validate customer
   - Calculate totals

3. workflow.CreateInstance
   - workflowId: "order-fulfillment"
   - Initial data: order details

4. Conditional: Check Inventory
   - If in stock: Send "process" event
   - If out of stock: Send "backorder" event

5. workflow.SendEvent
   - Event based on inventory check
   - Progress workflow automatically

6. Sink: Log Results
   - Record workflow instance IDs
   - Update order tracking

Accessing Workflow State in ETL

ETL chains can query workflow state for reporting and processing:

Example: Daily Approval Report

1. workflow.GetAllInstances
   - workflowId: "leave-request"
   - Streams all instances

2. Filter: Pending Approvals
   - Select instances in "Pending Approval" state
   - Filter by date range

3. Transform: Format Report
   - Extract employee names
   - Calculate pending days
   - Group by manager

4. Aggregate: Summary Statistics
   - Count by manager
   - Average pending time
   - Identify overdue

5. SendEmail: Daily Report
   - To: HR team
   - Attach formatted report

Best Practices

ETL Chain Design

Keep Chains Focused - One chain per responsibility (notification, validation, calculation) - Reuse chains across multiple workflows - Name chains descriptively

Error Handling - Always handle ETL chain failures gracefully - Log errors for debugging - Consider retry logic for transient failures - Use conditional steps for error paths

Performance - Keep OnEntry/OnExit chains lightweight - Move heavy processing to async operations - Use batch operations where possible - Cache external data when appropriate

Guard Chains

Always Include GuardEndPoint - Required for proper guard evaluation - Place at end of guard chain - Marks evaluation complete

Return Clear Results - Success = transition fires - Failure = transition blocked - No side effects in guard chains

Keep Guards Simple - Pure evaluation logic only - No data modifications - No external system updates - Fast execution

Data Management

Public vs Private - Use elxPublic for display data only - Use elxPrivate for sensitive/internal data - Never expose credentials in elxPublic - Clean up temporary data in elxPrivate

State Modifications - Prefer Merge over Add for bulk updates - Use Remove to clean up old data - Document expected data structure - Validate data before adding

Troubleshooting

Common Issues

Guard Chain Not Firing - Ensure workflow.GuardEndPoint is present - Check chain returns success/failure correctly - Verify chain is assigned to correct transition - Check ETL logs for errors

OnEntry Chain Not Executing - Verify chain is assigned to state - Check user has permission to execute chain - Review ETL logs for errors - Ensure chainset is correct

Data Not Updating - Check field paths are correct - Verify data types match - Review ETL chain logs - Ensure workflow instance is being saved

Performance Issues - Profile ETL chain execution time - Identify slow steps - Consider async processing - Optimize external API calls

ETL Integration

Overview

Workflow ETL Steps Reference

Instance Management

workflow.CreateInstance

workflow.SendEvent

workflow.GetCurrentState

workflow.GetAllInstances

Data Manipulation

workflow.AddToElxPublic

workflow.AddToElxPrivate

workflow.MergeToElxPublic

workflow.MergeToElxPrivate

workflow.RemoveElxPrivate

Navigation Control

workflow.SetElxRedirect

Guard Evaluation

workflow.GuardEndPoint

Utilities

workflow.LookupStates

workflow.CleanInstances

workflow.DuplicateWorkflow

workflow.CreateInstanceField

Common ETL Patterns

Pattern 1: Send Notification on State Entry

Pattern 2: Validate Data Before Transition

Pattern 3: Calculate Derived Values

Pattern 4: Update External System

Pattern 5: Timeout Handling

Pattern 6: Batch Create Instances

Pattern 7: Conditional Routing with Guards

Using Workflows in ETL Chains

Triggering Workflows from ETL

Accessing Workflow State in ETL

Best Practices

ETL Chain Design

Guard Chains

Data Management

Troubleshooting

Common Issues

See Also