Workflow Integration

Overview

ETL chains integrate seamlessly with the Workflow module, enabling data transformation and business logic execution at various points in workflow state machines. This combination is powerful: - Workflows manage state-dependent business processes - ETL chains handle data transformation and integration

Integration Points

ETL chains can be called from workflows at multiple points:

Integration Point	When It Executes	Use Cases
OnEntry	When entering a state	Initialize data, send notifications, update systems
OnExit	When leaving a state	Cleanup, validation, logging
OnTransition	During state transition	Transform data, validate transition, update records
Guard	Before transition fires	Evaluate complex conditions, check business rules

Workflow ETL Steps

The ETL module provides workflow-specific steps for interacting with workflow instances:

• workflow.CreateInstance - Create new workflow instance
• workflow.SendEvent - Send event to workflow instance
• workflow.GetCurrentState - Get workflow instance state
• workflow.AddToElxPublic - Add data to public workflow data
• workflow.AddToElxPrivate - Add data to private workflow data
• workflow.MergeToElxPublic - Merge data into public workflow data
• workflow.MergeToElxPrivate - Merge data into private workflow data
• workflow.SetElxRedirect - Set browser redirect URL

OnEntry ETL Chains

OnEntry chains execute when a workflow instance enters a state.

Use Cases

• Send notification emails
• Update external systems
• Calculate derived values
• Initialize state-specific data
• Log state entry

Example: Send Approval Request Email

Workflow State: "Pending Approval"
  OnEntry Chain: "SendApprovalRequestEmail"

ETL Chain: "SendApprovalRequestEmail"
  1. Get workflow instance data
     - Access elxPublic for request details
     - Access elxPrivate for approver information
  
  2. Look up approver details
     - MongoDB Reader to get approver email
     - Get approver name
  
  3. Compose email
     - To: approver email
     - Subject: "Approval Required: Leave Request"
     - Body: Include request details from elxPublic
  
  4. Send email
     - Send notification to approver
  
  5. Log notification
     - Write to audit log
     - Record notification sent

Accessing Workflow Data

In OnEntry chains, workflow instance data is available:

{
  "elxPublic": {
    "employeeName": "Jane Doe",
    "leaveType": "Annual",
    "startDate": "2026-03-10",
    "days": 5
  },
  "elxPrivate": {
    "approverId": "MGR123",
    "approverEmail": "manager@example.com"
  },
  "elxEvent": {
    "eventName": "submit"
  }
}

OnExit ETL Chains

OnExit chains execute when a workflow instance leaves a state.

Use Cases

• Cleanup temporary data
• Validate data before transition
• Log state exit
• Archive state-specific information

Example: Validate Before Approval

Workflow State: "Pending Approval"
  OnExit Chain: "ValidateApprovalData"

ETL Chain: "ValidateApprovalData"
  1. Get workflow instance data
     - Check elxEvent for approval decision
  
  2. Validate required fields
     - If approved: check approval comments exist
     - If rejected: check rejection reason exists
  
  3. Validate business rules
     - Check approver has authority
     - Check request is still valid
  
  4. On validation failure:
     - Add error details to elxPrivate
     - Drop the record (emit no output) — chain produces no records, preventing transition
  
  5. On validation success:
     - Add validation timestamp
     - Record passes through — chain produces output, transition continues

OnTransition ETL Chains

OnTransition chains execute during a state transition.

Use Cases

• Transform data for next state
• Update related records
• Send notifications
• Integrate with external systems

Example: Process Approval Decision

Workflow Transition: "Pending Approval" → "Approved"
  Event: "approve"
  OnTransition Chain: "ProcessApproval"

ETL Chain: "ProcessApproval"
  1. Get approval details from elxEvent
     - Approver ID
     - Approval comments
     - Approval timestamp
  
  2. Update leave balance
     - MongoDB Reader: get employee leave balance
     - Subtract approved days
     - MongoDB Writer: update balance
  
  3. Add approval info to elxPublic
     - workflow.MergeToElxPublic
     - Add approver name, timestamp, comments
  
  4. Update external HR system
     - REST API call
     - Send leave approval notification
  
  5. Send confirmation email
     - Email to employee
     - Confirm leave approval

Guard ETL Chains

Guard chains evaluate conditions to determine if a transition should fire.

How Guards Work

1. Event arrives at workflow instance
2. Transition with matching event is found
3. If transition has guard, guard ETL chain executes
4. If chain emits at least one record, transition fires
5. If chain emits no records, transition doesn’t fire

Guards work through the standard record-drop mechanism: a step that filters out a record (or a mongodb.Reader that finds nothing) produces no output, so nothing reaches the end of the chain and the transition is blocked. No exceptions are thrown.

Use Cases

• Complex business rule evaluation
• External system checks
• Data validation
• Authorization checks

Example: Check Leave Balance

Workflow Transition: "Draft" → "Pending Approval"
  Event: "submit"
  Guard Chain: "CheckLeaveBalance"

ETL Chain: "CheckLeaveBalance"
  1. Get requested days from elxPublic
     - Extract days field
  
  2. mongodb.AggregationDefinition
     Pool: your-pool-name
     Collection: employees
     Aggregation: [{"$match": {"employeeId": "${employeeId}"}}]

  3. string.Substitute
     Field: aggregation

  4. mongodb.Reader (get employee leave balance)
  
  5. Check sufficient balance
     - Compare requested days to available balance
  
  6. If insufficient balance:
     - Drop the record (emit no output) — guard sees no records, transition doesn't fire
     - Workflow stays in "Draft" state
  
  7. If sufficient balance:
     - Record passes through — guard sees output, transition fires
     - Workflow moves to "Pending Approval"

Creating Workflow Instances from ETL

ETL chains can create workflow instances using the workflow.CreateInstance step.

Example: Create Leave Request from Form Submission

ETL Chain: "CreateLeaveRequest"
  1. Receive form submission data
     - Employee ID
     - Leave type
     - Start date, end date
  
  2. Validate form data
     - Check required fields
     - Validate date range
  
  3. Calculate leave days
     - Number calculation
     - Exclude weekends/holidays
  
  4. Create workflow instance
     - workflow.CreateInstance
     - Workflow: "LeaveRequestApproval"
     - Initial data in elxPublic:
       {
         "employeeId": "...",
         "leaveType": "...",
         "startDate": "...",
         "endDate": "...",
         "days": ...
       }
  
  5. Return workflow instance ID
     - For tracking and reference

Sending Events to Workflows from ETL

ETL chains can send events to workflow instances using the workflow.SendEvent step.

Example: Auto-Approve After Timeout

Scheduled ETL Chain: "ProcessPendingApprovals"
  (Runs hourly)
  
  1. MongoDB Reader
     - Query workflow instances
     - State: "Pending Approval"
     - Created > 48 hours ago
  
  2. For each pending instance:
     - workflow.SendEvent
     - Event: "auto_approve"
     - Data: { reason: "Auto-approved after timeout" }
  
  3. Log auto-approvals
     - Write to audit log
     - Record auto-approval action

Common Patterns

Pattern 1: Notification on State Entry

Workflow State: "Approved"
  OnEntry Chain: "SendApprovalNotification"

Chain sends email to employee confirming approval

Pattern 2: Data Enrichment on Transition

Workflow Transition: "Draft" → "Pending Approval"
  OnTransition Chain: "EnrichRequestData"

Chain adds manager info, leave balance, approval deadline to elxPrivate

Pattern 3: External System Integration

Workflow State: "Approved"
  OnEntry Chain: "UpdateHRSystem"

Chain calls external HR system API to record approved leave

Pattern 4: Conditional Routing with Guards

Workflow State: "Pending Approval"
  Transition 1: → "Approved" (event: "approve")
    Guard: "CheckApproverAuthority"
  
  Transition 2: → "Pending Senior Approval" (event: "approve")
    Guard: "CheckRequiresSeniorApproval"

Guards determine which transition fires based on business rules

Pattern 5: Scheduled Workflow Processing

Scheduled ETL Chain: "ProcessExpiredRequests"
  1. Find expired workflow instances
  2. Send "expire" event to each
  3. Workflow transitions to "Expired" state

Accessing Workflow Data in ETL Chains

When an ETL chain is called from a workflow, the workflow instance data is available as the input record:

Available Fields

{
  "elxPublic": {
    // Data shared with browser/client
  },
  "elxPrivate": {
    // Server-side only data
  },
  "elxEvent": {
    // Current event data
  },
  "elxHistory": [
    // Array of state transitions
  ],
  "elxAppId": "...",
  "workflowId": "...",
  "instanceId": "..."
}

Modifying Workflow Data

Use workflow-specific ETL steps to modify instance data:

Add to elxPublic:

workflow.AddToElxPublic
  Field: "approvalDate"
  Value: "${currentDate}"

Merge into elxPrivate:

1. timestamp.Now
   To Field: now

2. workflow.MergeToElxPrivate
   JSON: {
     "internalNotes": "Processed by ETL",
     "processingTimestamp": "${now}"
   }

Set redirect:

workflow.SetElxRedirect
  URL: "/leave-requests/confirmation"

Best Practices

Keep Workflow ETL Chains Fast

Workflow chains should execute quickly: - Avoid long-running operations - Don’t process large datasets - Use async operations for slow tasks - Consider moving slow operations to scheduled ETL

Handle Errors Gracefully

Workflow ETL chains communicate outcomes through the record stream, not exceptions: - OnEntry/OnExit: If the chain emits no records or emits an error record, the transition may be affected - Guard chains: A guard blocks a transition by emitting no records — this is intentional, not an error - OnTransition: An unexpected empty result or error record may leave the workflow in an inconsistent state

Design chains to: - Validate inputs early - Drop records explicitly (via a filter step) to signal “condition not met” in guards - Emit error records ({ "error": "..." }) when downstream handling is needed - Log error records to a collection for debugging

Use Guards for Validation

Use guard chains for: - Business rule validation - Authorization checks - Data validation - External system checks

Don’t use guards for: - Data transformation (use OnTransition) - Notifications (use OnEntry/OnExit) - Logging (use OnEntry/OnExit)

Separate Concerns

• OnEntry: Initialization, notifications
• OnExit: Cleanup, validation
• OnTransition: Data transformation
• Guard: Condition evaluation only

Test Workflow ETL Chains

Test chains both: 1. Standalone - In ETL Designer with test inputs 2. In workflow context - With actual workflow instances

Document Workflow Dependencies

Document which workflows use which ETL chains: - Which states call which chains - What data the chains expect - What the chains modify - Error handling behavior

Example: Complete Leave Request Workflow

Here’s a complete example showing ETL integration at multiple points:

Workflow Definition

Workflow: "LeaveRequestApproval"

State: "Draft"
  Transition → "Pending Approval"
    Event: "submit"
    Guard: "CheckLeaveBalance"
    OnTransition: "EnrichRequestData"

State: "Pending Approval"
  OnEntry: "SendApprovalRequestEmail"
  OnAfter: 48 hours, event "auto_approve"
  
  Transition → "Approved"
    Event: "approve"
    OnTransition: "ProcessApproval"
  
  Transition → "Rejected"
    Event: "reject"
    OnTransition: "ProcessRejection"

State: "Approved"
  OnEntry: "SendApprovalNotification"
  OnEntry: "UpdateHRSystem"

State: "Rejected"
  OnEntry: "SendRejectionNotification"

ETL Chains

CheckLeaveBalance (Guard):

1. Get requested days from elxPublic
2. mongodb.AggregationDefinition / string.Substitute / mongodb.Reader
   (look up employee leave balance record)
3. Compare requested days to available balance
4. If insufficient: drop the record — no output, transition does not fire
5. If sufficient: record passes through — transition fires

EnrichRequestData (OnTransition):

1. Get manager information
2. Calculate approval deadline
3. Add to elxPrivate

SendApprovalRequestEmail (OnEntry):

1. Get approver details
2. Compose email
3. Send to approver
4. Log notification

ProcessApproval (OnTransition):

1. Update leave balance
2. Add approval info to elxPublic
3. Update external HR system

SendApprovalNotification (OnEntry):

1. Compose confirmation email
2. Send to employee
3. Log notification

UpdateHRSystem (OnEntry):

1. Prepare HR system payload
2. Call HR system API
3. Log integration result

This example shows how ETL chains handle validation, data enrichment, notifications, and external system integration throughout the workflow lifecycle.

Next Steps

• Scheduler Integration - Automate ETL chains with scheduling
• Dataset Integration - Provide data for visualizations
• Examples - More complete examples
• Best Practices - Design effective ETL chains