Data Transformation Package (DTP)
Introduction¶
The Data Transformation Package is a calculation engine that runs data transformation processing in the context of Business Workflow tasks that can be used in processes. In particular, every DTP is made up of one or several activities comprising one or several operations.
To define a DTP, it is therefore necessary to create the relative activities, divided into operations. The DTP performs calculations on dedicated physical tables identified by typologies with the logical structure defined by the dataset.
Activity details¶
Each DTP can contain several tasks that are executed in the order defined by the user. The following elements must be defined for every activity:
- the target dataset, or the dataset in which the operations insert, edit or check data,
- a target dataset typology
- a priority level, or the order in which the operations will be run on the reference dataset (for example, a level 3 activity is run after the level 2 activity and before the level 4 activity).
Operation types¶
| Operation type | Description |
|---|---|
| Insertion | Inserts data from a source dataset into a target dataset. |
| Enrichment | Enriches a target dataset with data from a source dataset. |
| Check | Runs checks on the target dataset data according to criteria defined by the user |
| Deletion | Deletes data from the target dataset according to user-defined criteria |
Based on the operation type, it is possible to define calculation criteria for specific data. The operations are run in the order in which they are listed in the DTP. For details of the available operations, see Operations.
Parametric DTPs¶
When defining a DTP, it is possible to use parametric elements that are resolved at runtime.
For more details, see Parametric DTPs.
Restrictions on the duplication of activities¶
To create activities with similar properties, it is possible to duplicate activities present in the current DTP. Below are some restrictions and behaviours:
- If there are errors in the duplication of the activity or one of the operations that it contains, the entire data processing will be cancelled.
- Operations which contain Matching fields cannot bring those relationships with them if the target dataset is changed in the duplication operation. In such cases, the system searches the new dataset for any columns with the same name as the original target dataset. It then only matches columns with the same name as those saved in the matching fields.
- With Loading to Analytical Workspace operations, the operation is only copied if the target dataset matches the source dataset.
- With Special Method operations, the values are not copied in the operation.
- For Allocation operations, the details are not copied in the operation.
- Fields containing filters or formulas such as Target filter, SQL formula, or Run condition are copied even if they contain references to columns of the target dataset.
- The following duplications are not supported:
- Enrichment with formula field operation
- fields matched for the Enrichment with reference table operation
- Field relationships related to operations will be duplicated if the target dataset matches the source dataset or if the target dataset contains a fields with the same code as a field of the source dataset.
- Relationship fields with special N values, which automatically relate the remaining fields, and F values, which apply the formula to the remaining fields, are copied in any case.
- In case of upload > Write to another dataset or Export Reporting layer to file in the activity, the upload will only be copied if the target dataset/type pair matches the source dataset/type pair.
- In case of upload > Export to Financial Workspace in the activity, the upload will be copied even if no Financial Workspace mapping is configured for the target dataset.