Installation¶

To install the Tagetik Data Loader, simply unzip the file TagetikDataLoader_ApacheKaraf.zip into a directory (e.g. /TagetikDataLoader/).

Version 9 of Tagetik Data Loader uses the latest version of Apache Karaf container, 4.4.7

Inside are two directories:

apache-karaf-&lt;version>: contains the Apache Karaf container, the Tagetik Data Loader jars and the default JDBC drivers. The directory can be renamed to apache-karaf.
tgk-configuration-&lt;version>: contains the externalisation of the Karaf configuration (/etc) in which the configuration files of the Tagetik Data Loader are also located.

Configuration¶

Tagetik Data Loader (TDL) requires the configuration of a set of parameters necessary for it to function correctly and fully:

Java Home configuration
definition of the other Karaf settings (memory, etc.)
definition of Karaf configuration
general configuration and configuration of the connection to CCH Tagetik
authentication configuration
logs configuration
configuration of datasources for the databases access
configuration of paths for accessing file system directories

Java Home configuration¶

In order to configure Java Home, it is necessary to access the setenv file(setenv.bat if installation is carried out on Windows operating systems) located in /bin and define JAVA_HOME.

The JDK version to be used is 17.0.12. The Tagetik Data Loader 9 has been built and tested with JDK17.0.12 : upgrading to subsequent patches is not discouraged in principle, as they generally contain fixes that do not disrupt compatibility. However, it should be noted that the Tagetik Data Loader is not beforehand tested with every JDK patch beyond version 17.0.12. This occurs only in response to specific requests or reports of issues identified after a JDK update.

E.g. linux: export JAVA_HOME="/usr/lib/jvm/openjdk-17.0.12"

E.g. windows set JAVA_HOME=C:\Progra~1\Java\openjdk-17.0.12

List of tested and supported JDKs:

Amazon Corrected 17.0.12
Microsoft 17.0.12
Oracle 17.0.12
SapMachine 17.0.12
IBM Semeru 17.0.12
Liberica 17.0.12
Temurin 17.0.12

Definition of other Java and Karaf settings¶

To define the other Java and Karaf settings, it is necessary to access the setenv file (setenv.bat if installing on Windows operating systems) located in /bin and add the options. If Karaf is run via Windows service, after running the wrapper installation command, configure the wrapper.conf file available inside the configuration directory. If a custom name is specified inside the wrapper installation command, the file will be karaf-.conf

E.g. linux:

export JAVA_MIN_MEM=256M # Minimum memory for the JVM

export JAVA_MAX_MEM=1024M # Maximum memory for the JVM

export JAVA_PERM_MEM=128M # Minimum perm memory for the JVM

export JAVA_MAX_PERM_MEM=256M # Maximum memory for the JVM

E.g. windows

set JAVA_MIN_MEM=256M # Minimum memory for the JVM

set JAVA_MAX_MEM=1024M # Maximum memory for the JVM

set JAVA_PERM_MEM=128M # Minimum perm memory for the JVM

set JAVA_MAX_PERM_MEM=256M # Maximum memory for the JVM

Karaf configuration definition¶

It is recommended to externalise the Karaf configuration from the /etc folder to another directory so that all customisations can be managed when updating the Karaf container. Once outsourced, it is possible to set up the configuration directory on Karaf by accessing the setenv file (setenv.bat if installing on Windows operating systems) located in /bin and defining the KARAF_ETC.

E.g. linux: export KARAF_ETC="/home/user/TagetikDataLoader/tgk-configuration"

E.g. windows set KARAF_ETC=C:\Progra~1\TagetikDataLoader\tgk-configuration

General setting and connection to CCH Tagetik¶

To configure the TDL for both the connection to CCH Tagetik and to change certain parameters relating to its functionality, it is necessary to access the tgk.properties.cfg file in the Karaf configuration directory.

The parameters to be configured are:

tgk.agent.polling.interval. Indicates at what intervals (in seconds) the TDL verifies requests for transactions to be taken over. It is recommended not to set up a value lower than 30 seconds
tgk.agent.tmpfolder. Indicates the path reserved for the TDL to save temporary resources used for data manipulation. If the parameter is not specified, or the specified folder does not exist, then Tagetik Data Loader uses the Karaf's temporary folder, /data/tmp, as the back-up folder. It is strongly recommended to specify this parameter, to avoid relying on temporary folders within the Karaf container.
tgk.agent.write.db.batchsize. Indicates the number of statements sent simultaneously to the database to be executed at write time (default 10000)
tgk.agent.read.db.fetchsize. Indicates the number of records to be retrieved simultaneously from the database during the reading phase (if not specified, the default value specified by the jdbc driver in use will be used). Note: the ideal value of this parameter may vary considerably depending on the database vendor and the complexity of the queries being executed.
tgk.endpoint.address. CCH Tagetik endpoint URL (e.g. https://alpha.saastagetik.com/prod)

If secure data transfer is active, i.e. the Tagetik Endpoint URL has https as protocol, in the absence of specific parameterisation, the TDL will use the default JDK trustor in use. Should you wish to specify a custom server certificate, you can do so via the following parameters:

tgk.server.ks.password. Password chosen when generating the server-side certificate. It must be written following the syntax specified in Security and encryption of sensitive data
tgk.server.ks.path. Complete path of the server-side certificate

If the communication between TDL and CCH Tagetik is managed by a proxy server, the following can also be configured:

tgk.proxy.server. Indicates the IP of the proxy server
tgk.proxy.port. Indicates the proxy server port
tgk.proxy.user. Indicates the user accessing the proxy server
tgk.proxy.password. Indicates the password with which the user defined in the previous property can access the proxy server. It must be written following the syntax specified in Security and encryption of sensitive data

It is possible to enable payload encryption of data exchanged between the TDL and CCH Tagetik via PGP algorithm.

To do this, it is necessary to configure:

tgk.pgp.publickey indicates the full path to the file containing the public key
tgk.pgp.privatekey indicates the full path to the file containing the private key
tgk.pgp. passphrase indicates the passphrase to use the private key. It must be written following the syntax specified in Security and encryption of sensitive data

If the Endpoint is only used to read data, only the public key needs to be configured; if it is only used to write data, only the secret key and secret passphrase need to be configured. If the TDL is used to write data, only the privatekey with the passphrase on the TDL and the publickey on the Endpoint defined on CCH Tagetik need to be configured.

It is also possible to specify more technical properties to guide the operation of the Tagetik Data Loader, which can be particularly useful when analysing incorrect behaviour.

tgk.debug.polling.log.disabled: This boolean property, when set to 'true', disables logging of successful poll requests that did not trigger TDL actions. Failed poll requests, regardless of the error, and all other TDL actions, will continue to be logged. Enabling this property therefore makes the tagetik.log file much less verbose. The default value is 'false'.
tgk.debug.tmp.cleanup.disabled: This boolean property, when set to 'true', disables the automatic deletion of temporary files generated by certain TDL operations. This option was introduced for diagnostic purposes, and allows you to analyse problems concerning the format or content of files generated by failed operations. The default value is 'false'. Warning: This property must be used with caution, as enabling it may lead to the retention of numerous temporary files and this may cause considerable use of disk space.

Example 1 *(HTTPS secure transfer without specified custom server certificates)***

tgk.agent.polling.interval=30

tgk.agent.tmpfolder=/home/user/TagetikDataLoader/tgk-configuration/tmp/

tgk.agent.write.db.batchsize=10000

tgk.agent.read.db.fetchsize=

tgk.endpoint.address=https://tagetik.installation.url.com/tagetikcpm

tgk.server.ks.path=

tgk.server.ks.password=

Example 2 *(HTTPS secure transfer with specified custom server certificates)***

tgk.agent.polling.interval=30

tgk.agent.tmpfolder=/home/user/TagetikDataLoader/tgk-configuration/tmp/

tgk.agent.write.db.batchsize=10000

tgk.agent.read.db.fetchsize=

tgk.endpoint.address=https://tagetik.installation.url.com/tagetikcpm

tgk.server.ks.path=/home/TDL/certificates/server.truststore

tgk.server.ks.password={TGK_ENCRYPT}server_side_certificate_password{TGK_ENCRYPT}

Example 3 (HTTP transfer)

tgk.agent.polling.interval=30

tgk.agent.tmpfolder=/home/user/TagetikDataLoader/tgk-configuration/tmp/

tgk.agent.write.db.batchsize=10000

tgk.agent.read.db.fetchsize=

tgk.endpoint.address=http://tagetik.installation.url.com/tagetikcpm

Example 4 (transfer with PGP)

tgk.agent.polling.interval=30

tgk.agent.tmpfolder=/home/user/TagetikDataLoader/tgk-configuration/tmp/

tgk.agent.write.db.batchsize=10000

tgk.agent.read.db.fetchsize=

tgk.endpoint.address=http://tagetik.installation.url.com/tagetikcpm

tgk.pgp.publickey=/home/user/TagetikDataLoade/publickey.asc

tgk.pgp.privatekey=/home/user/TagetikDataLoade/privatekey.asc

tgk.pgp.passphrase={TGK_ENCRYPT}123_pippo{TGK_ENCRYPT}

Authentication configuration¶

To enable the communication between TDL and CCH Tagetik, it is necessary to define the authentication parameters. They can be found in the file tgk.agent.cfg in the Karaf configuration directory.

The parameters are as follows:

tgk.agent.id. Unique ID associated with the TDL instance. It is automatically generated upon the first launch of Karaf.
tgk.agent.secret. SecretKey to authenticate the TDL on CCH Tagetik. It must be defined after the first launch of Karaf, once the ID has been automatically generated, and must be defined using the syntax specified in Security and encryption of sensitive data

With these two parameters it is possible to register the TDL on CCH Tagetik and authenticate it upon each request.

Log configuration¶

The TDL log system is nothing more than a customization of the log system of Apache Karaf. The log configuration can be found in the file org.ops4j.pax.logging.cfg in the Karaf configuration directory and is a standard Log4j configuration file.

This file contains the following section with the specified values:

Tagetik agent appender¶

log4j2.logger.tagetik.name = com.tagetik.cpm

log4j2.logger.tagetik.level=INFO

log4j2.logger.tagetik.additivity=false

log4j2.logger.tagetik.appenderRef.TagetikRollingFile.ref = TagetikRollingFile

log4j2.appender.tagetik.type = RollingRandomAccessFile

log4j2.appender.tagetik.name = TagetikRollingFile

log4j2.appender.tagetik.fileName = ${karaf.data}/log/tagetik.log

log4j2.appender.tagetik.filePattern = ${karaf.data}/log/tagetik.log.%i

log4j2.appender.tagetik.append = true

log4j2.appender.tagetik.layout.type = PatternLayout

log4j2.appender.tagetik.layout.pattern = ${log4j2.pattern}

log4j2.appender.tagetik.policies.type = Policies

log4j2.appender.tagetik.policies.size.type = SizeBasedTriggeringPolicy

log4j2.appender.tagetik.policies.size.size = 16MB

For the meaning of the various options and of the values to be assigned, refer to the Log4j specifications.

Log files are saved by default in the /data/log/ folder

Based on the default settings above, the TDL log file is named tagetik.log and is generated on a daily basis. Moreover, every day the file generated on the previous day is renamed by adding a suffix given by the production date.

Datasource configuration¶

In case database operations are required such as:

reading data to be transferred to CCH Tagetik
writing of data extracted from CCH Tagetik

the TDL requires the creation and configuration of tgk.datasource-.cfg files in the Karaf configuration directory.

The Karaf configuration folder (tgk-configuration) also contains the vendor templates officially supported by the Tagetik Data Loader.

For authentication with basic authentication:

SqlServer (JTDS driver): _tgk.datasource-jtds_template.cfg
SqlServer (Microsoft driver): _tgk.datasource-microsoft_template.cfg
Oracle: _tgk.datasource-oracle_template.cfg
PostgreSQL: _tgk.datasource-postgresql_template.cfg
SAP Hana: _tgk.datasource-hana_template.cfg
IBM DB2: _tgk.datasource-db2_template.cfg
Netsuite: _tgk.datasource-netsuite_template.cfg
Snowflake: _tgk.datasource-snowflake-basicAuth_template.cfg

For the Snowflake vendor, authentication is also managed via:

Key Pair: _tgk.datasource-snowflake-keyPair_template.cfg
OAuth2.0: _tgk.datasource-snowflake-oAuth_template.cfg

Warning: With regard to authentication with Key Pair, please refer to the Snowflake documentation for certificate generation and management. As far as OAuth2.0 is concerned, only the grant type 'client credentials' is accepted. For configuration and further details on the Snowflake side for both types of authentication, please refer to the official Snowflake documentation

The password type fields of the datasources, ('password' for basic authentication, 'private_key_pwd' and 'oauthClientSecret' for Snowflake authentication) must be entered (both when defining and editing) with the syntax specified in Security and encryption of sensitive data

Adding a custom datasource¶

From the Karaf command line via the command service:list DataSourceFactory it is possible to display the installed JDBC drivers.

To add a new JDBC driver it is necessary to:

copy the [driver_name].jar into the /lib/endorsed folder
run the install -s wrap:file:./lib/endorsed/[driver_name].jar command from the Karaf command line
add a .cfg file relating to the new datasource into the Karaf configuration directory (like those described above)
restart Karaf

The simultaneous presence of two JDBC drivers from the same vendor causes problems at runtime. If it is necessary to install a new JDBC driver (E.g. : a later version), the previous one has to be uninstalled.

Path configuration¶

In case operations are needed on files such as:

file transfer to CCH Tagetik
writing files produced by CCH Tagetik

TDL requires the tgk.repository.cfg file to be created and configured in the Karaf configuration directory.

Inside, a mapping list between the logical path name and the real absolute path can be configured.

The syntax is typical of the *.properties files, thus:

=

Security and encryption of sensitive data¶

For keystore and datasource passwords, there is a mechanism for encrypting the values.

To define the value of the password, the following syntax as to be used: ={TGK_ENCRYPT}{TGK_ENCRYPT}

After saving the configuration file, Tagetik Data Loader will automatically encrypt the value of the property and replace it in the same configuration file.

TDL can only handle passwords containing 'ASCII Printable Characters', i.e. ASCII characters 32 to 126 with the following exceptions:

The '$' character (ASCII 36) cannot be used as the last character of the password
The character '\' (ASCII 92) cannot be used in the password.

Starting Karaf¶

Once installation and configuration have been carried out, Tagetik Data Loader can be used by starting Karaf via the karaf command(karaf.bat if installation is carried out on Windows operating systems) present in /bin.

Starting as an operational system service¶

It is also possible to start Karaf as an OS service by executing the command wrapper:install --name "TDL" --display "TDL" --description "TDL" from the Karaf command line and afterwards:

for Windows: run from the command prompt the command TDL-service.bat install (or karaf-service.bat install), present in the /bin directory
for Linux: follow the instructions shown once you run the wrapper:install command .