ETL

Introduction

The ETL module enables writing powerful and versatile high-level functions that can be used for various purposes including but not limited to:

  • ETL (extract, transform, load) for preparing and enriching data
  • log monitoring
  • report generation
  • maintenance or cleanup
  • sending notifications
  • making external API calls

A record or item in the ETL Management interface is called a chainset. Each chainset can contain as many chains as desired. Each chain is made up of a set of steps. These steps are the building blocks in the ETL module.

The chains built in this module can be used with other Ambience modules to create more functional solutions. The following are the most common usages:

  • Creating scheduled jobs for various purposes, such as data polling, file maintenance, report generation and so on
  • Using on-demand data retrieval as datasets on dashboards

The ETL module also has a job queue to allows multiple jobs to be triggered on the same ETL chainset. Some delay may be experienced on the return results (if any) if there are jobs that need to be completed before yours could begin.

Granting read-access to other users through roles and modifying other ETL chain properties are easily done through the ETL Management interface.

Features

Below is an overview of the available features for each interface comprising the Dashboard module.

  • ETL Management

    • Add chainset
    • Edit chainset
    • Enable and disable chainset
    • Add and remove roles
    • Delete chainset
    • Refresh list
    • Upload chainset (JSON)
    • Download chainset (JSON)
    • Search

  • ETL Designer

    • Manage chains
      • Add chain
      • Copy chain
      • Cut chain
      • Paste chain
      • Download chain (JSON)
      • Edit chain
      • Delete chain
      • Explain chain
    • Manage steps
      • Add step
      • Copy step
      • Cut step
      • Paste step
      • Disable step
      • Move step
      • Edit step
      • Delete step
    • Run chain
    • View results

Module Interfaces and Privileges

Interface Description Required Privilege
ETL Management Allows adding, modifying and deleting ETL chainsets. Chainset properties, including access rights and display names, are set here. Mandatory:
- mod-etl

For modifying and deleting existing chainsets:
- ownership of the chainset
ETL Designer Provides an interface for configuring ETL chainsets, running chains and viewing results. Mandatory:
- mod-etl-designer

At least one of the following for read access:
- at least one of the user’s roles must be granted access to the chainset
- ownership of the chainset

For write access:
- ownership of the chainset

Next: ETL Management