Data Preparation

It is possible to map either a single description or a set of descriptions (made up of several dimensions) onto a single dimension or a set of dimensions. The resulting description we want to map and the combination of dimensions on which we want it mapped are considered as a single row. For easier reading, we won’t talk about codes here, but we specify now that it’s necessary to associate each description with its related code field.

There are two types of fields where we can enter our descriptions: level fields and concat fields.

The difference is that the level description fields are treated separately, while the concat description fields are concatenated to form a new description. The number of the Level fields must be the same between train and execute. The number of concat fields can change between train and execute. The level fields are compared one by one, whereas the concat field are compared by concatenating all the concat fields together.

With the field level, therefore, we give a greater weight to the single field, with the concat field we give the same weight to all the fields between them concatenated.

The user defines a set of configuration parameters (see Appendix 2), including the language of the descriptions and whether to enable automatic translation prior to text comparison.

The system uses a single multilingual NLP embedding model, capable of processing and comparing texts in multiple languages natively. As a result, text comparison works out of the box for all supported languages, without requiring any preliminary translation. During data preparation, the software automatically cleans the descriptive fields by removing missing values and special characters and concatenates the descriptions into a single textual representation.

An optional parameter allows all descriptions to be translated into English before the embedding generation step. When this parameter is set to Y, the texts are first translated into English and then compared using the embedding model.

The use of translation is not mandatory and does not affect the functional correctness of the mapping process. It is provided solely as an optional optimization. In some scenarios, translating texts into English may lead to a small improvement in comparison accuracy, as embedding models may generally perform slightly better in English than in other languages.

However, automatic translation has a significant computational cost and increases execution time, since it requires a full Transformer model for language generation rather than a lightweight embedding-only model. For this reason, enabling translation is generally discouraged for large data sets and should be considered only when working with a limited volume of data.

When enabled, the translation process is fully handled internally by the system and is completely transparent to the user. Descriptions are always displayed in their original language, both during training and execution.