ETL Daemon
Description
An ETL Daemon identifies an ETL chainset that is intended to run continuously while the Ambience server is alive. This means that it does not need to be scheduled, that it will automatically restart upon failure and that it will switch from machine to machine in a cluster if the machine it is running on fails.
For example, an ETL daemon can be used to continuously reads messages from a broker (MQTT, Kafka etc.) or scan a directory for changes.
The ETL Daemon module allows you to create and manage ETL daemons.
A set of ETL steps for controlling ETL daemons are available under the “ETL Daemon” category in the ETL module. These steps provide the ability to start-stop ETL daemons which enables them to be run for scheduled periods of time rather than continuously. This allows the ETL daemons to be scheduled to run at high load and stop in quieter periods.
To run ETL daemons, the Daemon User must be set in the ETL Daemon option in the System Settings module.
All actions and who performed them, and other important information are sent to the Audit Log.
This module is available in both Ambience and Repertoire software suite.
Module Interfaces
| Interface | Description | Required Privilege |
|---|---|---|
| ETL Daemon | Allowed to view ETL Daemon | mod-etl-daemon |
| ETL Daemon | Allowed to call ETL Daemon API | mod-etl-daemon-api |
| ETL Daemon | Allowed to edit ETL Daemon | mod-etl-daemon-edit |
| ETL Daemon | Able to receive from ETL Daemon | mod-etl-daemon-notifications |
UI and Status
UI
There are several columns in the ETL daemon management page.
| Column | Description |
|---|---|
| Name | Name of ETL daemon. |
| Description | Brief description of ETL daemon. |
| Affinity | It is green when the values of “Actual” and “Preferred” are equal, otherwise it is red. |
| Actual | Actual Ambience instance that is running the ETL daemon. |
| Preferred | Preferred Ambience instance to run the ETL daemon, set in System Settings module. |
| Status | Current status of ETL daemon. |
| Started | When the ETL daemon was started. |
| Duration | Indicates how long the ETL daemon has been running. |
| Autoplay | Indicates a green tick if autoplay is turned on, otherwise a red cross if autoplay is turned off. |
| Retry | Current retry logic state Spinning icon indicates current retrying. “2/3” indicates 2 of 3 retries remaining. |
| Enabled | Shows a green tick if enabled, otherwise a red cross if the ETL daemon is disabled. |
| Actions | List of actions to be performed on the ETL daemon. |
Retry
The retry iteration includes a retry strategy with a count of 3. If a daemon fails, it will automatically started again up to three times.
Depending on the retries remaining, the delay before the daemon is automatically started will differ as follows:
- 3 retries remaining = 10 sec delay
- 2 retries remaining = 30 sec delay
- 1 retry remaining = 60 sec delay
Each time a daemon fails, a warning notification will be sent if there ar retries remaining. If there are no retries remaining, then an error notification will be sent. Notification are sent to whoever has the mod-etl-daemon-notifications privilege.
Status
The status of the ETL daemons are as follows:
| Status | Description |
|---|---|
| Idle | ETL daemon not running. |
| Starting | ETL daemon was started by the run action but has not yet been picked up by an Ambience instance. |
| Running | ETL daemon is running |
| Stopping | ETL daemon was stopped by the stop action but has yet been stopped. Stop is not always immediate as the ETL chain will await for a safe place to stop in a managed way. |
| Restarting | ETL daemon was restarted by the restart action and is still in the process of restarting. |
| Reclaiming | ETL daemon was running in a non-preferred Ambience instance and the preferred instance was started causing the ETL daemon to be reclaimed automatically. |
| Failed | ETL chain that the ETL daemon was running has failed and all retries wre exhausted. |
The checkboxes beside the “Select All” field and status allow users to filter the ETL daemon that are listed in the page.
Add
To add a ETL daemon, start by clicking on the “Add” button on the upper right corner of the page.
The “Add Panel” will appear.
| Field | Description | Mandatory | Input Type | Constraints | Default Value |
|---|---|---|---|---|---|
| Properties | |||||
| Name | Unique name assigned to the ETL daemon. | Y | Text field | Any text (case-sensitive) | Empty string |
| Description | Brief description for the ETL daemon. | N | Text field | Any text | Empty string |
| Preferred | Select the preferred Ambience instance to run the ETL daemon. | Y | Drop-down list | Select from list | Not selected |
| Enabled | If selected, enables the ETL daemon. | Y | Checkbox | True (selected) or false (unselected) | Selected |
| Job Properties | |||||
| Chainset | Select the ETL chainset to be used. | N | Drop-down list | Select from list | Not selected |
| Chain | Select the ETL chain to be used. | N | Drop-down list | Select from list | Not selected |
| Parameters | JSON parameters to pass to the ETL daemon. | Y | Text field | Any text | Empty string |
In the “Properties” section, key in an unique name for the new ETL daemon in the “Name” field. It is case-sensitive. Provide a brief description on the ETL daemon in the “Description” field. Select the desired system ID from the drop-down list in the “Preferred” field. The “Enabled” field is selected by default. Do note that only enabled ETL daemons can be run.
In the “Job Properties” section, select the desired ETL chainset and chain for the ETL daemon in the “Chainset” and “Chain” fields respectively. In the “Parameters” field, key in the desired parameters in JSON format for the ETL daemon.
For the ETL chain to appear in the drop-down list, the ETL chain must contain the ETL Daemon Signature step and include an ETL step that performs a continuous task that has been designed to work with the ETL daemon. Currently, these steps are :
- Daemon MQTT Reader
- Daemon Timer
Click on the “Save” button at the top right corner of the panel to create the new ETL daemon. Clicking on the “Cancel” button aborts the action.
If the “Preferred” field is left empty, an error message will appear.
If the name already exist or the name is missing, the relevant error message will appear.
Upon successful saving the ETL daemon, a message will appear.
Bulk Actions
This function allows you to perform the same action on multiple daemons.
To perform a bulk action, select one or more checkboxes next to desired daemons. Do note that disabled ETL daemons are not selectable. Click on the “Bulk Actions” button at the upper right corner of the page.
Select the desired action to perform from the list.
The available options that appear in the list depends on the status of the ETL daemon. If the selected daemons are running, the below options will be available.
If no daemon is selected, only the first three options will appear, which allows you to either select all, select none or invert selection.
Run
To run a ETL daemon, click on the 
“Run” icon under the “Actions” column corresponding the desired daemon.
If any error occurs, an error message will appear and the daemon will stop running.
Once the ETL daemons runs, the status of the daemon will change from “Idle” to “Starting”. The 
“Busy” icon will appear in the “Actions” column. Once the Ambience instance picks up the ETL daemon, the status will change to “Running”. Do note that the “Busy” icon may not be noticeable, as the duration may be too short.
The available icons will change according to the status of the ETL daemon.
Kill
Killing a running ETL daemon will stop the ETL daemon anywhere in the ETL chain (unsafe). This is potentially unsafe as the chain may be at any step. For example, in the middle of a complex action that should be allowed to complete. Instead you should use “Stop” as described below. Autoplay that is set prior will be set to false automatically.
To kill a running ETL daemon, click on the 
“Kill” icon under the “Actions” column corresponding the desired daemon that is running.
A “Kill Daemon” dialog box will appear.
Click on the “OK” button to kill the ETL daemon. To abort, click on the “Cancel” button.
Stop
Stopping a running ETL daemon will stop the ETL daemon at a managed place in the ETL chain (safe). This ensures that the chain is not in the middle of a complex action that needs to complete. Autoplay that is set prior will be set to false automatically.
To stop a running ETL daemon, click on the 
“Stop” icon under the “Actions” column corresponding the desired daemon that is running.
The “Busy” icon will appear with its status changed to “Stopping”. Until the ETL daemon stops running, its status will change to “Idle”. Do note that the icon may not be noticeable, as the time to stop can be short, depending on which step in the chain is currently executing.
Restart
Restarting a running ETL daemon stops (safe) and then starts the ETL daemon.
To restart a running ETL daemon, click on the 
“Restart” icon under the “Actions” column corresponding the desired daemon.
Autoplay On/Off
When autoplay is set to true, the ETL daemon will run automatically when the Ambience instance starts. The autoplay is set to off by default.
To set autoplay to on (true), click on the 
“Toggle On” icon under the “Actions” column corresponding the desired daemon. A green tick will appear under the “Autoplay” column.
To set autoplay to off (false), click on the 
“Toggle Off” icon under the “Actions” column corresponding the desired daemon. A red cross will appear under the “Autoplay” column.
Edit
To modify an existing ETL daemon, click on the 
“Edit” icon under the “Actions” column corresponding the desired daemon.
The “Edit” panel will appear.
Edit as desired and click on the “Save” button to save the changes. Once the daemon is saved, a message will appear.
You can enable or disable an ETL daemon. Disabled ETL daemons cannot be triggered or run.
Do note that if the ETL daemon is running, the icon will not appear.
Delete
Deleting a ETL daemon removes it from the list.
To delete an existing daemon, click on the 
“Delete” icon under the “Actions” column corresponding the desired daemon.
If the daemon is running, the icon will not appear.
There is an option to undo the deletion. A notification with an “Undo” button appears right after clicking on the “Delete” icon.
Upon clicking on the “Undo” button, the deleted daemon is restored and is added back to the list. Do note that the notification will disappear after a few seconds.
Upload
To upload a ETL daemon, start by clicking on the “Upload” button on the upper right corner of the page.
The “Upload” dialog box appears on the page.
Browse to the location of the file (<filename>.daemon.json) to be uploaded and set the “Name” field. By default, the “Name” field uses the name of the upload file. You can change it to an appropriate name if desired. If the file to be uploaded is to replace an existing daemon, select the “Overwrite” field. Do note that the “Name” field must be unique, or an error message will appear.
| Field | Description | Mandatory | Input Type | Constraints | Default Value |
|---|---|---|---|---|---|
| File | Browse to select file to be uploaded. | Y | “Choose File” button | Any text | Empty |
| Name | Unique name assigned to the file to be uploaded. When a file is specified in the above field, this field will automatically takes in the file name of the specified file. |
Y | Text field | Any text | Empty string |
| Overwrite | If selected, overwrites existing daemon. | N | Checkbox | Select or not selected | Not selected |
Click on the “OK” button to upload the selected file. Click on the “Cancel” button to abort the action. A message will appear after the file has been successfully uploaded.
If the name of the daemon to be uploaded already exist, an error message will appear.
To overwrite the existing daemon, select the “Overwrite” field in the dialog box, then click on the “OK” button.
You can arrange/sort the list of daemons in ascending or descending order by clicking on the name of the desired field. By default, the list is sorted by the “Name” field in ascending order.
Download
You can download an existing ETL daemon by clicking on the 
“Download” icon under the “Actions” column corresponding the desired daemon.
The downloaded daemon has the following filename format <filename>.daemon.json.
Refresh
After performing actions on the browser/tab, the list is reloaded to display the ETL daemons. The manual “Refresh” button is available and is particularly useful if there are others making changes on the same page.
The “Refresh” button is found on the upper right corner of the page. Clicking on it reloads the list.
Search
The search bar appears on the upper left corner of the page.
This provides an easy way to search through the list. It is case-insensitive and displays records that have the entered search value in any of the values of the fields below:
- Name
- Description
- Affinity
- Preferred
- Status
- Started
- Retry
