Existence check
Functionality and purpose of the operation¶
The Check - existence operation checks for the presence of a value on the target dataset fields. This check uses the condition columns and existence columns specified on a reference (existence) dataset that has been specifically prepared (see Reference dataset).
The existence columns on the reference dataset contain an M (Mandatory) for the fields to be checked on the target dataset, whereas they are empty for fields that should not be considered.
The checks and conditions are applied to the target dataset using column names, which will be the same as the corresponding columns in the reference dataset.
Operation behaviour¶
The following scenarios may occur during execution:
- No null value in the dataset: the operation is completed successfully.
- Null values detected: the operation generates a warning. The audit shows the error message and details of the top 10 rows in the dataset with anomalies.
IMPORTANT: If the DTP is launched from the cockpit and there are anomalies, an anomaly report file is created. For further details see DTP Report.
If the Dataset to be checked is the following:
Dataset¶
| Category | Product | Price | Currency |
|---|---|---|---|
| Electronics | Smartphone | 699 | USD |
| Fashion | Sneakers | USD | |
| Home | Coffee Maker | 89 | USD |
| Fashion | Leather Jacket | 159 | |
| Toys | Puzzle Game | USD | |
| Books | Mystery Novel | 18 |
Example 1: Check with mandatory fields with values
Reference dataset - Existence¶
| Category | Product | Price | Currency |
|---|---|---|---|
| Home | Coffee Maker | M | M |
| Electronics | Smartphone | M | M |
¶
The operation is successful and no anomaly is reported as all fields defined as mandatory have a value in the Dataset.
Example 2: Check with mandatory fields without any value
Reference dataset - Existence¶
| Category | Product | Price | Currency |
|---|---|---|---|
| Home | Coffee Maker | M | M |
| Electronics | Smartphone | M | M |
| Books | Mystery Novel | M | M |
¶
The operation reports an anomaly for the row that has no value for the Currency field defined as mandatory in the existence type reference dataset.
The data processing ends with a warning, highlighting the top 10 differences in the audit:
| Category | Product | Price | Currency |
|---|---|---|---|
| Books | Mystery Novel | 18 |
Example 3: check with optional fields
Reference dataset - Existence¶
| Category | Product | Price | Currency |
|---|---|---|---|
| Home | Coffee Maker | M | M |
| Electronics | Smartphone | M | M |
| Books | Mystery Novel | M |
¶
The operation does not check the Currency field for the Books - Mystery Novel condition as the field on the existence type reference dataset is empty.
The operation is successful and no anomaly is reported as all fields defined as mandatory have a value in the Dataset.