In this chapter the Data Logistics of the TEDAMOH ExMeX Framework are described.

The Batch & LPS Orchestration section describes how batches can be created, configured and deactivated. Batches are the basis for executing and orchestrating Load Process Steps (LPS). The section also shows how to configure LPS in batches.

After that, the Batch & LPS Execution section shows how the previously configured batches or LPSs can be executed.

The section Data Pipeline describes and highlights specific data pipelines, which are described by using examples for a better understanding.

The section DAPI gives an overview of all DAPIs: the DAPI istself, the asociated attributes and the data they provide. In the sub-section Common Attributes are all columns descibed which most DAPIs has in common.

The section Stored Procedures gives an overview of all stored procedures used for data logistics.

(Photo by CHUTTERSNAP on Unsplash)

For the implementation of a data pipeline within the data warehouse, batches were introduced in the ExMeX Framework Core from the beginning.

Each data pipeline may start with it’s own Master-Batch or is part of a Child-Batch and can exists of four Levels of children-batches, aka batch hierarchy.

There are various possibilities to influence the behavior of the batches via global variables or batch-specific settings.

By assigning Load Process Steps (LPSs) to each batch, a finer orchestration of the LPSs with the batches is already possible. There are various possibilities to influence the behavior of the LPSs via global variables, batch settings or even LPS-specific settings.

(Photo by Reproductive Health Supplies Coalition on Unsplash)

A pile of old, used pallets for transporting heavy goods in trucks, trains etc. Just as data is transported (LPS as pallets) in one or more batches (truck or train) via hyperloop in a data pipeline.

After the TEDAMOH ExMeX Framework Core has loaded data into the data warehouse for many days, weeks, months and perhaps even years, questions arise for the data modeler or the operator: What is the state of individual streams or Load Process Steps (LPS) over time or what does the system do on the first four days of each month? Or how can I present a continuous monitoring of my current load?

With the help of the Data Logistic Data Mart of the ExMeX Logging Framework, a uniform and stable access layer exists in the form of a star schema to access all this information and to be able to provide answers to the most diverse questions.

In the following sections the dimensions and facts of the Data Logistic Data Mart of the ExMeX Logging Framework are described in detail.

(Photo by Luke Chesser on Unsplash)