BPM Admin Guide

Customizing ETL

This section describes the steps you need to follow to customize ETL.

General Rules

  • All customizations to ETL (that is, jobs or transformations) should be done by new jobs and transformations dedicated to handle a certain customization.
  • These custom jobs and transformations must be placed in a folder called "custom" or "Custom" in the repository.
  • No standard jobs or transformations should be changed.
  • The folder structure (aside from that in the "custom" folder) should remain unchanged.
  • New jobs/transformations for loading custom-defined data should be scheduled or manually run after the standard LoadData or a custom version of LoadData can be made.
  • When upgrading, customizations may need to be reconsidered:
    • An upgrade may introduce standard functionality that previously has been implemented by customizations. It is recommended to include the standard functionality in favor of the customizations, if they fully satisfy the needs of the customer.
    • An upgrade may change field names or semantics of fields. Consequently, customizations need to be adjusted carefully.
    • An upgrade may declare fields or tables as deprecated (for example, if they are replaced by others or not included in Maconomy anymore). Customizations that rely on such fields and tables should be considered changed. In some situations it may be convenient for the customer to change the customizations such that they still rely on old functionality, for example, deprecated fields. Deprecated fields or tables that are removed by the upgrading process can then be reintroduced in custom tables. Note that a dump of the content of the deprecated table then must be done as part of the upgrade process.
    • Carefully document the steps performed in the customization such that these can easily be reconsidered and possibly redone after an upgrade. Primarily, document the following:
      • Which standard tables the custom tables are copies of.
      • Which fields have been added to the copies of the standard tables.
      • The links that have been made.
Note: When running ETL jobs that load data to the Data Warehouse, one of the Number fields on Maconomy may fail while being converted to string (data type) before loading. We recommend temporarily disabling the cache on steps that fail until this issue gets resolved.

Transaction Time Stamps for Customized ETL

If the data warehouse ETL has been customized, additional tables may need to have transaction time stamps enabled. For example, if additional tables are maintained by the customized ETL, and incremental load is defined for these tables, the corresponding tables in Maconomy must have transaction time stamp triggers enabled.

Check Enable Transaction Time Stamps in MConfig for the database short name. This action enables transaction time stamp triggers for all Maconomy tables, and thus also the tables used in the ETL customization.