Deep Learning for Anomaly, Forecast, and Analysis Model
The Deep Learning for Anomaly, Forecast, and Analysis model can be divided into three steps: Data Preparation, Training, and Execution.
The goal is to prepare data for the forecasting task (Data Preparation), then fit the model with minimal error (Training), and predict the target variable values (Execution).
This model is designed to train only one Deep Learning architecture once; it performs a single training pass over all historical series. That same model then provides predictions for all dimensions and targets in the data.
The solution relies on the OData API Service, and the user setting it up must have the necessary permissions.
Data Preparation¶
The goal of this step is to prepare data for the forecasting task.
- Duplicated values are managed by summing the corresponding values (aggregation of the rows with the same dimensionality ID and time ID).
- Exogenous features that are not associated with any target are excluded: they must have the same dimensionality as the target you aim to influence.
- Numerical data of the exogenous variables are scaled to a mean of 0 and a variance of 1.
- The seasonality for each feature is calculated.
- Optionally, seasonality features for the target and exogenous variables can be created using a rolling mean at seasonal intervals. There is a parameter that controls whether to generate seasonal variables for the target and exogenous features.
- For each feature, the script creates lagged features by shifting the original data from 1 up to the smaller of the maximum seasonality of that feature (across different dimensions) and MAX_LAG, if it is set. After generating the lagged features, some missing values will appear at the start of the data set. All of these must be removed to ensure proper model training. As a result, the data set will lose several rows equal to the maximum lag across all features, multiplied by the number of dimensions.
Therefore, to retain as much data as possible, we suggest keeping MAX_LAG relatively low.
- If the flag for scaling is set to Y, during the pre-processing step, we apply the standard scaler to the external features and targets.
Example:
The features in the input training data set are:
COVID_RATE
MARKETING
While VOLUME is the target variable.
The script calculates the seasonality of the target variable, which is 2, and the seasonality of each feature.
COVID_RATE - 3
MARKETING - 2
Given a MAX_LAG of 5, the script calculates the shift up to the minimum between 5 and the maximum seasonality of each feature (minus 1):
| COVID_RATE | COVID_RATE-1 | COVID_RATE-2 | MARKETING | MARKETING-1 |
|---|---|---|---|---|
| 0.8 | 95 | |||
| 0.7 | 0.8 | 100 | 95 | |
| 0.75 | 0.7 | 0.8 | 90 | 100 |
| 0.9 | 0.75 | 0.75 | 110 | 90 |
| … | … | … | … | … |
As you can see, this step produces some missing values in the data set, so the first two rows will be deleted
| COVID_RATE | COVID_RATE-1 | COVID_RATE-2 | MARKETING | MARKETING-1 |
|---|---|---|---|---|
| 0.75 | 0.7 | 0.8 | 90 | 100 |
| 0.9 | 0.75 | 0.75 | 110 | 90 |
| … | … | … | … | … |
Training¶
The goal of this step is to develop a custom Multi-Task Deep Neural Network architecture.
The last 20% of the data is reserved for validation, allowing the model’s error to be evaluated by comparing actual and forecasted values.
The model is optimized by minimizing the Mean Absolute Percentage Error (MAPE).
The contribution of each driver is calculated at each time point for every target. Then, the importances are determined as the average contribution per target and normalized so that they sum to 1 for each target. If the contributions are set to N, the importances are not derived from these contributions but are instead calculated from the final gradients associated with each input variable.
This phase creates four data sets:
- Fit data set – containing the fitted values
- Importance data set – summarizing driver importances
- Contribution data set – detailing contributions over time
- Metrics data set – reporting performance metrics
In the metrics data set, for each target, four metrics (R², MSE, MAE, and MAPE) are provided, calculated on the validation data set. Additionally, the standard deviation of the residuals is included.
Execution¶
The goal of this step is to predict the target variable values using the trained model.
If the forecasting horizon extends beyond one step (meaning more than one point to predict), a rolling forecast strategy is used to generate predictions for the entire required period. In this method, previously forecasted values serve as inputs for the subsequent predictions.
This phase produces three data sets:
- Forecast data set – includes predicted values with their confidence intervals
- Importance data set – summarizes driver importances
- Contribution data set – shows contributions over time
If anomaly detection is performed on the actual target values provided as input, the forecast data set includes an additional column showing the outlier/inlier classification and the associated outlier probability.