Working with Properties DataSources

The Properties DataSource can be used directly or indirectly through a parameter processor. The following examples illustrate the direct use of Properties DataSource.

Testing the Data Flow

The Employee details are maintained in a text file which includes the department field of the employees. Use the following process to derive a new column using the department field of the text data source and check the validity of the derived column using the Properties DataSource:

  1. Use the filesystem or folder popup menu to select Add -> DataSource.

  2. The DataSource Wizard appears.

  3. Select Text DataSource and click Next. In the screen that appears, enter a unique name for the data source, such as Employee.

  4. Enter the URL (file path) in the text box or click the browse button to the right of the text box and select the EmpInfo.txt file from the Open dialog window.

    Select the First line is header option.

  5. Select the Qualifier as " and separator as Semicolon. Click Infer Schema to infer the columns. After setting the properties, the screen appears as shown in Figure 8.3, “Text DataSource Parameters”.Click Finish to add the text data source to the repository.

    Figure 8.3. Text DataSource Parameters

    Text DataSource Parameters
  6. Select the same location, and use the popup menu to select Add -> DataSource.

  7. The DataSource Wizard appears. Select Properties DataSource and click Next.

  8. Enter a unique name such as Prop_Emp.

  9. Click Add Column to invoke the Add Column dialog window. Enter the column name as Dept_Name (same column name as in the text data source), Data type as String and the value as Computer. After entering the values, the Add Column dialog appears.

  10. Click Ok to add the column to the Wizard. The screen appears as shown in Figure 8.4, “Properties DataSource”.

    Figure 8.4. Properties DataSource

    Properties DataSource
  11. Click Finish to add the Prop_Emp data source to the repository.

  12. Select Properties Samp, and use the popup menu to add a Composite DataSource called Comp1.

  13. Double click the Comp1.ds data source to open the composite designer window.

  14. Select and place the properties data source Prop_Emp on the diagram. Select the derivative processor from the Action bar of the designer window and place it on the diagram.

  15. Connect the output of Prop_Emp to the input of the Derivative processor.

  16. Open the Derivative Properties and select the Derived tab. Click Add Column.

  17. In the Add Column dialog, enter the name as Department, data type as String and the value as dept. Click Ok to add the column to the wizard. After entering the properties, the processor window appears as shown in Figure 8.5, “Derivative Wizard”.

    Figure 8.5. Derivative Wizard

    Derivative Wizard
  18. Select the JavaScript tab and enter the following script in the window.

    if (Dept_Name== "Computer") dept = "CPT";
    else dept = "Others";
  19. Click Finish.

  20. Connect the output of the Derive-Dept processor to the Result. Select Result and choose View from the popup menu. The output will be displayed in the data window as shown in Figure 8.6, “Test Output”. Notice that the department Computer is mapped to CPT.

    Figure 8.6. Test Output

    Test Output
  21. Now delete the flow between the Prop_Emp data source and the Derive-Dept processor.

  22. Select and place the Employee.ds data source in the designer window. Using the flow processor, connect the output of the Employee.ds data source to the input of the Derive-Dept processor. Select Result, and choose View from the popup menu. The output will be displayed in the data window as shown in Figure 8.7, “Full Output”. Notice that all the records containing the Computer department are mapped to "CPT", and the Other departments are mapped to others. The above procedure makes it easy to check the validity of the derived columns.

    Figure 8.7. Full Output

    Full Output

    Note

    Therefore, if you want to test a flow with specific values, you can use a Properties data source to provide them, then switch back to the real data source for release.

Passing Parameters to the Flow

The following procedure illustrates how to use a Properties DataSource along with the Parameter processor, to pass parameter values to check a Filter condition:

  1. Add a Properties DataSource named Prop_Emp1.

  2. Add a column called Department with data type String and value Electronics.

  3. Add a column called Salary with data type Integer and value 7500. The values are shown in Figure 8.8, “Prop_Emp1”.

    Figure 8.8. Prop_Emp1

    Prop_Emp1
  4. Click Finish to add the data source to the repository.

  5. Add a Composite DataSource called Comp2.

  6. Open the Composite diagram and and drag the Employee.ds file onto it. Add a Filter processor to the diagram as well.

  7. Connect the output of the Employee DataSource to the input of the Filter processor.

  8. Open the filter properties. In the row corresponding to the Dept_Name field, select the Equals option from the combo box of the When column. Enter the condition as ${Department}.

  9. In the row corresponding to the Emp_Sal field, select the More Than option from the combo box of the When column. Enter the condition as ${Salary}. After entering the values, the page appears as shown in Figure 8.9, “Filter Wizard”. Click Finish.

    Figure 8.9. Filter Wizard

    Filter Wizard
  10. Select the Parameters processor from the action bar and place it in the workspace of the designer window. Change the colour of the processor.

  11. Connect the output of the Filter processor to the input of the Parameters processor

  12. Select and place the Prop_Emp1 data source in the designer window workspace.

  13. Connect the Prop_Emp1 DataSource with the params input of the Parameters processor. Also, connect the output of the Parameters processor to the Result.

  14. Select Result, and choose View from the popup menu. The output is displayed as shown in Figure 8.10, “Output”.

    Figure 8.10. Output

    Output
  15. The values passed by the records of the Prop_Emp1 data source are matched with the dynamic parameters in the filter condition. Based on this, the records of the text data source satisfying the filter condition are fetched.

As the flows are pull-push, the data store pulls data from the source, and this is when the parameters get added. The pull command propagates back to the filter which uses the properties to control the filter process. The pull command terminates at the datasource, which pushes the data back along the flow, through the filter which has already been configured.