ETL Management

Description

ETL Management provides an interface to manage ETL chainset properties. Users can add, modify and delete chainsets here. This interface also leads to the ETL designer. Uploading and downloading chainsets are also supported. Controlling the audience of the chainset is also managed through this interface.

In larger systems, it is useful to be able to share common chains across multiple chainsets. This is achievable by creating a chainset and then importing that chainset into others. This allows those importing to be able to call on any of the imported chains as if they were locally defined.

Add ETL Chainset

To add a new chainset, start by clicking on the “Add” button on the upper right corner of the page.

A dialog box prompting the user to key in the new chainset’s name is displayed. The user also has the option to copy another chainset, which includes all its components, such as chains, steps in each chain and formatting, and use it as a starting point.

Field	Description	Mandatory	Input Type	Constraints	Default Value
Name	Unique name assigned to the chainset that would be used as its identifier.	Y	Text field	Any text. The value must not exist in the current list of names, regardless of status (enabled or disabled) and ownership. The value is case-sensitive.	Empty string
Copy From	Indicates what template the chainset would be based on. Selecting “Blank ETL” results in using a blank canvass as a starting point while selecting an existing chainset would result in having the selected chainset as the starting template.	Y	Select	1 value can be selected. Options are: - “Blank ETL” - names of all enabled chainsets the user currently has read access to	“Blank ETL”

Note that in copying another chainset, any change made to the chainset in the “Copy From” field would not be applied to the new chainset after its creation. Copying does not link any of the chainsets.

Clicking on the “OK” button on the dialog above redirects the user to the ETL Designer, where the user can modify the chainset, given that the user has appropriate access rights, and run the chains or steps. Clicking on the “Cancel” button aborts the action.

Do note that if an option is selected from the drop-down list for the “Workgroup” field at the top of the page, any ETL chainset added to the management page will have the selected workgroup added to it. For example, if “Workgroup A” is selected, a new ETL chainet “Chainset XYZ” is added. “Chainset XYZ” will have “Workgroup A” added to its properties.

Edit ETL Chainset

To modify an existing chainset’s attributes, click on the “Edit” icon under the “Actions” column corresponding the said chainset.

The “Edit ETL CHainset” dialog box displays the selected chainset’s attributes.

Field	Description	Mandatory	Input Type	Constraints	Default Value
Name	Unique name assigned to the chainset that would be used as its identifier.	Y	Text field	Any text. The value must not exist in the current list, regardless of status (enabled or disabled) and ownership. The value is case-sensitive.	Empty string
Enabled	Indicates if the chainset is enabled (selected) or disabled (unselected).	Y	Checkbox	True (selected) or false (unselected)	True (selected)
Import Chainsets
Search Function	Allows you to search for the ETL chainsets that starts with the string defined in this field.	N	Text field	Any text. The value is upper case and is not case-sensitive.	Blank
Chainsets	List of ETL chainsets.	N	Drop-down list	Multiple values can be selected.	Not selected

Enable / Disable Chainset

Disabling a chainset temporarily makes it unavailable for all allowed read users.

To enable a disabled chainset, select the “Enabled” checkbox in the “Edit ETL Chainset” dialog box. The enabled ETL chainset will be indicated with a green tick under the “Enabled” column in the management page.

Similarly, deselect the checkbox to disable an enabled chainset. Disabling a chainset also disables using it with the ETL Designer interface for the designated roles. The owner can still access the chainset via the ETL Designer interface. The disabled ETL chainset will be indicated with a red cross under the “Enabled” column in the management page.

Import Chainsets

In the “Import Chainsets” panel, the search function allows you to define the string to filter the ETL chainsets in the the drop-down list that starts with the string defined. The characters in the string appears as upper case and is not case sensitive. For example, key in “ETL” and all ETL chainsets that starts with “ETL” will appear in the drop-down list.

To select the ETL chainset, click on the desired ETL chainset in the drop-down list and it will appear above the search function.

To remove any of the selected ETL chainset, click on the “x” on the right of the ETL chainset.

Note

When importing the chainsets, ensure the chainsets are structured sensibly and logically into a hierarchy and must not include circular dependencies. The following scenario must not occur:

A imports B

B imports C

C imports A

Open ETL Chainset

Click on the name of the ETL chainset under the “Name” column to load the ETL Designer.

More details on the interface can be found below:

ETL Designer.

Upload ETL Chainset

Clicking on the “Upload” button on the upper right corner of the interface prompts the user to upload a JSON (.json) file.

Single File

To upload a single file, select the JSON file to upload by clicking on the “Choose File” button in the dialog box. THe selected file must have the filename format: <filename>.chainset.json.

For the “Name” field, the same constraints for “Name” in the add chainset dialog are applied to the “Name” field here. Verification on name uniqueness is not applied if “Overwrite” is selected.

After clicking on the “OK” button on the upload dialog, a record of a new chainset is uploaded (new name given) or an existing chainset with the same name (existing name with “Overwrite” selected) is updated. Clicking on the “Cancel” button aborts the action.

Zip File

Browse to the location of the zip file to be uploaded. The name of the chainsets will be extracted from the zip file. Click on the “OK” button to upload the chainsets. The “Upload Results” dialog box will display the results of the uploading.

Those results in black are uploaded successfully while those in red indicates upload failures with their relevant messages. Click on the “Close” button to close the dialog box.

Delete ETL Chainset

Deleting an ETL chainset removes it from the records, which means another chainset with the same name as the deleted one can be added.

To delete an existing chainset from Ambience, click on the “Delete” icon under the “Actions” column corresponding the said user.

There is an option to undo the deletion. A notification with an “Undo” button appears right after clicking on the “Delete” icon.

Upon clicking on the “Undo” button, the deleted chainset is restored and is added back to the list of chainsets.

Dictionary

The “Dictionary” button at the upper right corner of the page when clicked, displays a page which allows you to search for the ETL step along with their description.

The page can be divided into several parts: The search function at the top of the page allows user to key in keyword to search for the ETL step. The left panel displays the categories of ETL. The centre panel displays the ETL steps of the category selected. The right panel displays the description of the ETL selected.

Keyword Search

To use the search function to search for a ETL step, key in a keyword, e.g. “array”. The left panel will display the categories that have ETL step(s) containing the keyword “array”. Select the desired category and the centre panel will display the list of ETL steps containing the keyword “array”. Select the desired ETL step in the centre panel. Some ETL steps may have a number in square brackets on its right. This number indicates the number of usage of the ETL step. The description for the ETL step will appear in the right panel.

Category Search

To search using category, scroll and select the desired categroy from the list of categories in the left panel. The list of ETL steps for that category will appear in the centre panel. Scroll and select the desired ETL step. The description for the selected ETL step will appear in the right panel.

Workgroup

The “Workgroup” field at the top left corner of the page allows you to filter the ETLs associated with the selected workgroup.

Simply select the desired workgroup from the drop-down list and the ETLs associated with the selected workgroup will appear.

Do note that if no workgroup is created in the Workgroups module, the “Workgroup” field at the top left of the page will not appear.

More Actions

This feature allows user to perform actions on multiple ETL chainsets at the same time.

When the “More Actions” button located at the upper right corner of the page is clicked, a list of available actions are displayed.

If no chainset is selected prior clicking the button, there will be fewer actions available. To select a chainset, select the checkbox next to the name of the chainset.

Option	Description
Select All	Selects all chainsets in the list.
Select None	Unselects all chainsets.
Invert Selection	Inverts the current selection. That is, any chainsets selected will be unselected and vice versa.
Set Roles	Selects the roles to be assigned to the chainsets.
Set Workgroups	Selects the workgroups to be associated with the chainsets.
Download	Downloads multiple chainsets as a zip file. If only one chainset is selected, the selected chainset will be downloaded as a JSON file.
Set Enabled True	When selected, sets the selected chainsets to enable.
Set Enabled False	When selected, sets the selected chainsets to disable.

Set Roles

To set the roles for the selected chainsets, click on the “More Actions” button and select the “Set Roles” option. The “Set Roles” dialog box will appear.

Select the roles to have access to the chainsets by selecting the checkboxes. Alternatively, you can search for the desired role by keying in the keyword in the search function at the top of the dialog box. You can also choose to select all, select none or invert selection by clicking on their respective icons on the right of the search function.

Click on the “OK” button to save the changes or click on the “Cancel” button to abort the action.

Set Workgroups

To set the workgroups to be associated with the selected chainsets, click on the “More Actions” button and select the “Set Workgroups” option. The “Set Workgroups” dialog box will appear.

Select the workgroups to be associated with the chainsets by selecting the checkboxes. Alternatively, you can search for the desired workgroup by keying in the keyword in the search function at the top of the dialog box. You can also choose to select all, select none or invert selection by clicking on their respective icons on the right of the search function.

Click on the “OK” button to save the changes or click on the “Cancel” button to abort the action.

Do note that the available options that appear in the list, the user is either owner of the workgroups or are part of the role listed in the workgroups. If the user is not the owner of any workgroup or not part of the roles of a workgroup, no options will appear in the “Workgroups” panel.

Download

To download the selected chainsets, click on the “More Actions” button and select the “Download” option.

If only one chainset is selected, the downloaded chainset has a filename of <filename>.chainset.json.

If more than one chainset are selected prior, the “Download” dialog box will appear.

Key in the desired name for the zip file and click on the “OK” button. The selected chainsets will be downloaded as a zip file with the filename provided earlier. The individual chainset in the zip file will have the filename format as mentioned above.

Set Enabled True / False

To enable or disable the chainsets, click on the “More Actions” button and select the “Set Enabled True” or “Set Enabled False” option respectively.

The selected chainsets that are owned by the user will be enabled or disabled as selected. ETL chainsets that are not owned by the user, will not be affected by the enable/disable action.

An enabled chainset is represented with a green tick under the “Enabled” column, while a disabled chainset is represented with a red cross.

Refresh

After performing actions (e.g. add chainset, delete chainset) on the browser window/tab, the list is reloaded to display the updated data. The manual “Refresh” button is available and is particularly useful if there are others making changes on the same page.

The “Refresh” button is available on the upper right corner of the page. Clicking on it reloads the list of chainsets.

Search

The search bar appears on the upper left corner of the page.

This provides an easy way to search through the chainset list. It is case-insensitive and displays records that have the entered search value in any of the values of the fields below:

• Name
• Owner
• Workgroups
• Roles
• Last Modified

Alternatively, you can click on the any owner, orkgroup or owner under their respective columns to aid the search for the chainsets in the page.

In the example above, chainsets that are assigned the role “login” are shown.

These two search methods can be combined together, with each criteria separated by a comma.