Release Notes

Release Notes 3.0.0

We are happy to bring TEDAMOH ExMeX Release 3.0.0 to you. With this major version we have introduced the ExMeX…


We are happy to bring TEDAMOH ExMeX Release 3.0.0 to you.

With this major version we have introduced the ExMeX compiler to be prepared for the future. In the future it should be possible to compile and execute the most different (target) languages for modules (formerly LPS).

Furthermore, we provide the basis for additional design patterns for point-in-time tables and bridge tables. The templates introduced with this release and the resulting compiled modules are currently very simple. We have retired some of the previous design patterns.

We recommend to upgrade also ExMeX - Tools, ExMeX - Plugins and ExMeX - Companion App to the newest versions! 

As always we believe this version will lead you to a better data solution!

Have fun with the new relase,

Dirk and his Team


Pay attention to the following installation steps!

Documentation

Companion App

v0.5.0 new features, enhancements & bug-fixes. We changed the overall look & feel.

  • Fixed: Several smaller issues
  • Fixed:  0000567: Deleting Filter not working
  • Added:  0000360: Remember previous server at login
  • Added:  0000539: Search field for batch names
  • Updated: 0000543: Change feedback for Settings-Save

Framework Plugins

PowerDesigner - ExMeX Plugin - Data Vault 

v2.4.0 new features, enhancements & bug-fixes - Changelog

To use all features of TEDAMOH ExMeX Release 2.13.0 this plugin version is mandatory.

  • Added:Small changes due to 0000468: SQL extension - new templating
  • Updated: 0000534: Wrong comment (in configuration scripts)
  • Updated: 0000507: Union Stereotype for Reference

Go to Documentation

The names of the objects in the data model are configured in the plug-in, NOT the code for database objects (DDL for table, view, PK, FK, etc.)! The derivation of the code from the name is a setting in the corresponding physical data model (.pdm) and is not part or task of the plugin!

Framework Toolbox

LPS Side Injection

v1.3.0 new features, enhancements & bug-fixes - Changelog

To use ExMeX Tool LPS Side Injection with ExMeX Framework Core Release 2.13.0 or later this version is mandatory. Older versions of LPS Side Injection will not work any longer!

  • Fixed: 0000531: Bulk Update Load Batch Settings

Go to Documentation

Set Batch Setting

v1.1.0 new features, enhancements & bug-fixes - Changelog

To use ExMeX Tool Set Batch Setting with ExMeX Framework Core Release 2.13.0 or later this version is mandatory.

  • Added: New

Go to Documentation

Metadata Install

v2.2.0 new features, enhancements & bug-fixes - Changelog

Redesigned version, not compatible with version prior 2.x.x!

  • 0000560: New parameter for Metadatainstall
    • Old and new arguments now in POSIX style
    • Suiteable for CI/CD
    • Changed type of arguments
    • Additional arguments
    • Extended build in help
  • Redesigned code
  • Several smaler enhancements
  • 0000557: Reduce output.log
  • 0000575: New features for MetadataInstall.bat
  • 0000577: Config-File also with argument changeable
  • 0000584: Get settings and extended help
  • 0000581: Exit, when other error occured
  • Fixed some smaller issues

Go to Documentation

Framework Core

In addition to some internal optimizations and minor bug fixes, the following tickets have been resolved:

Added

  • 0000542: DQ Check wrong combinations of modes in 1:1 Data Copy
  • Historization of modules (LPS) if new, updated or deleted
  • 0000447: Erkennen von Fehlern in output.log
  • 0000554: Prevent Main Install if already installed
  • 0000468: SQL extension - new templating
  • 0000469: Extend CDC LPS with SQL override
  • 0000550: SQL-Override: Recreate LoadPatternId = 80
  • 0000219: Implement compile module (LPS) for PIT table
  •  0000218: Implement compile module (LPS) for Bridge table

 Updated

  • 0000568: Timestamp DeltaExportOverride-Function
  • 0000544: DQ Test on Partial CDC in 1:1 Data Copy
  • 0000561: More info at 'Waiting for free slot in chunk'

 Fixed

  • 0000551: DQ Check for using HiZ-Table in CDC -> Warning no Error
  • 0000562: Fix AppGetListOfMapping
  • 0000553: Performance Execute Views
  • 0000563: pk violation in edSatLoadBatchContainsLoadProcessStep
  • 0000586: Dependency for a LPS multiple times created
  • 0000589: Ticket HelpDesk #120 - Dependencies between lps not set correctly

 Removed

  • Some deprecated code (e.g. Aligns with bitemporal)
  • 0000556: DELETE SSIS Script

Download

The downloads for a new installation of ExMeX Framework Core can be found here:

The downloads for an update of the ExMeX Framework Core can be found here:

Installation Notes

Before upgrading any existing TEDAMOH ExMeX Metadata framework, it is recommend to backup your Database and TEDAMOH ExMeX framework folder! If you do not backup, well it’s up to you. Follow all steps!

New Installation

See chapter Install Framework.

Update from Version 2.13.x to 3.0.0

  1. Extract 7z-file and copy files to your root directory.
  2. Change InitPath in file '/install/update/Update TEDAMOH ExMeX Database Framework Release v2.13.x to v3.0.0.sql' to your local ProjectRoot.
  3. Double check if correct database for installation is set in ./usr/config/UsrConfigExMeXFramework.sql (if it is used!).
  4. Double check and ensure that all tmp_*-tables in Schema MetadataZone are deleted.
  5. Merge latest version off './etc/config/ExMeX Set Config Variable.orig.sql' (provided with this release) with your script './etc/config/ExMeX Set Config Variable.sql'.
  6. Copy the ".\usr\compiler\UsrExMeXCompilerInit-mssqlserver.orig.sql" file to ".\usr\compiler\UsrExMeXCompilerInit-mssqlserver.sql" and proceed with the following steps.
  7. Execute file '/install/update/Update TEDAMOH ExMeX Database Framework Release v2.13.x to v3.0.0.sql' in SSMS in SQLCMD Mode.
  8. Reinstall Metadata!
Posted 8 months agoby Dirk.Lerner

Release Notes 2.13.0

We are happy to bring TEDAMOH ExMeX Release 2.13.0 to you. This version includes several key features that we believe…


We are happy to bring TEDAMOH ExMeX Release 2.13.0 to you.

This version includes several key features that we believe will lead you to a better data warehouse solution:

Now the One To One Copy Data Data Pipeline is enhanced with historisation and first LPSs are ready for sequences as Data Vault Surrogate Key (Hub, Link, Hub-Satellite and Hub-Satellite with HashDiff).

Additional global configurations such as setting column names for ColumnStatusCDC, ColumnAuditTrailId or ColumnEntityId. This configuration is now separated from other ExMeX Framework Core settings in a separate configuration file for easier deployment and setup of new environments.

And last but not least: The batch configuration made in the Companion App can be downloaded as a DML file and used for deployment.

We recommend to upgrade also ExMeX - Tools, ExMeX - Plugins and ExMeX - Companion App to the newest versions! 

Have fun with the new relase,

Dirk and his Team


Pay attention to the following installation steps!

Documentation

o One To One Copy Data Pipeline with historization: Link
o Several topics added and updated

Companion App

v0.3.0 new features, enhancements & bug-fixes:

o Added: ExMeX Configuration

v0.4.0 new features, enhancements & bug-fixes:

o Added: 0000531: Bulk Update Load Batch Settings
o Added: 0000540: usp for download Batch DML

Framework Plugins

PowerDesigner - ExMeX Plugin - Data Vault v2.3.0 new features, enhancements & bug-fixes:

To use all features of TEDAMOH ExMeX Release 2.13.0 this plugin version is mandatory.

o Added: 0000521: Add Column Stereotype for Temporal Mode
o Added: 0000400: Create Name of Relations
o Added: 0000523: Order of columns
o Added: 0000364: Default for new Variables
o Fixed: 0000525: Ordering in Multi Active Satellites
o Updated: 0000522: HashDiff and StateTimeBefore not mandatory

Go to Documentation

The names of the objects in the data model are configured in the plug-in, NOT the code for database objects (DDL for table, view, PK, FK, etc.)! The derivation of the code from the name is a setting in the corresponding physical data model (.pdm) and is not part or task of the plugin!

Framework Toolbox

LPS Side Injection v1.3.0 new features, enhancements & bug-fixes:

To use ExMeX Tool LPS Side Injection with TEDAMOH ExMeX Release 2.13.0 this version is mandatory. Older versions of LPS Side Injection will not work any longer!

o Fixed: 0000531: Bulk Update Load Batch Settings

Got to Documentation

Set Batch Setting v1.1.0 new features, enhancements & bug-fixes:

To use ExMeX Tool Set Batch Setting with TEDAMOH ExMeX Release 2.13.0 this version is mandatory.

o Added: New

Got to Documentation

Metadata Install v1.4.0 new features, enhancements & bug-fixes:

To use ExMeX Tool Metadata Install with TEDAMOH ExMeX Release 2.13.0 this version is mandatory.

o Added: 0000533: Build in paramter for database in MetadataInstall
o Fixed: Some minor enhancements
o Updated: Separated from ExMeX Framework Core, from now on as standalone ExMeX tool

Got to Documentation

Framework Core

In addition to some internal optimizations and minor bug fixes, the following tickets have been resolved:

o Added: 0000520: 1:1 Copy LPS with historization
o Added: 0000526: Performing some preliminary error checks in MetadataInstall.bat
o Added: 0000533: Build in paramter for database in MetadataInstall.bat
o Added: 0000529: DQ Check for using HiZ-Table in CDC
o Added: 0000531: Bulk Update Load Batch Settings
o Added: 0000533: Build in paramter for database in MetadataInstall.bat
o Added: 0000515: Sequence for Hub, Link, Sat (HashDiff) generators

o Updated: 0000253: Document MetadataInstall.bat
o Updated: 0000535: Tuning Access in Dimension

o Fixed: 0000505: Performance of DAPIs
o Fixed: 0000518: Change Logging of Reference
o Fixed: 0000536: Restart when ERROR 87
o Fixed: 0000530: One more NULL value in BatchRunVapourTrailId
o Fixed: 0000537: Trailing Restart log entries when ERROR 87
o Fixed: 0000536: Restart when ERROR 87

o Removed: 0000484: Remove StatusCDC from JSON-usps

Installation Notes

Download

The downloads for a new installation of ExMeX Framework Core can be found here:

The downloads for an update of the ExMeX Framework Core can be found here:

New Installation

See chapter Install Framework.

Update from Version 2.12.x to 2.13.0

Follow all steps!

Before upgrading any existing TEDAMOH ExMeX Metadata framework, it is recommend to backup your Database and TEDAMOH ExMeX framework folder! If you do not backup, well it’s up to you.

1. Extract 7z-file and copy files to your root directory.

2. Change InitPath in file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.12.x to v2.13.0.sql” to your local ProjectRoot.

3. Double check if correct database for installation is set in ./usr/config/UsrConfigExMeXFramework.sql (if it is used!).

4. Double check and ensure that all tmp_*-tables in Schema MetadataZone are deleted.

5. Copy the "./etc/config/ExMeX Set Config Variable.orig.sql"  file to "./etc/config/ExMeX Set Config Variable.sql" and proceed with the following steps:

  • Adopt your existing configuration from the dataObject 'MetadataZone.ConfigExMeXFramework' into the script './etc/config/ExMeX Set Config Variable.sql'. Especially in the section between line 104 to end of file. 
    Or configure the values as you wish.
    The configuration made here is installed directly during a new installation of ExMeX Framework Core.
    New global variables are installed during rollout with the values configured here.
    Attention: Configured values already existing in the database (dataObject 'MetadataZone.ConfigExMeXFramework') will not be changed by the configuration in this file!
  • Transfer your existing configuration from the file './etc/config/ConfigExMeXFramework.sql' into the script './etc/config/ExMeX Set Config Variable.sql'. Especially in the section between line 30 to 103.
    If the variable no longer exists in the new file, please do not add it!
  • Later, after the release update, you can edit global variables for ExMeX Framework Core configuration with the ExMeX Framework Companion App or with the stored procedure. For more details see also section Getting Started -> Configuration -> How do you set global variables and what is their purpose? and Getting Started -> Install Framework -> How do you set global variables for initial installation?.

6. Merge latest version off './etc/config/ConfigExMeXFramework.orig.sql' (provided with this release) with your script './etc/config/ConfigExMeXFramework.sql'.

7. Execute file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.12.x to v2.13.0.sql” in SSMS in SQLCMD Mode.

8. Check if the settings made in step 5 match the values set in the ExMeX Framework Core database:

SELECT *
FROM MetadataZone.ConfigExMeXFramework

9. Reinstall Metadata!

Posted 1 year agoby Dirk.Lerner

Release Notes 2.12.0

We are happy to bring TEDAMOH ExMeX Release 2.12.0 to you. This release contains several main features which we believe…

We are happy to bring TEDAMOH ExMeX Release 2.12.0 to you.

This release contains several main features which we believe will drive you towards a better Data Warehouse solution: More global configurations like (UseTargetEntityDB, FKConstraintOnSatelliteLPS or FKConstraintOnLinkLPS), One To One Copy Data Data Pipeline and enhanced CDC Data Pipeline.

We recommend to upgrade also ExMeX - Plugins and ExMeX - Companion App to the newest versions! 

Have fun with the new relase,

Dirk and his Team


Pay attention to the following installation steps!

Documentation

o One To One Copy Data Pipeline: Link
o Batch Orchestration: Link
o Stored Procedures moved to one places in section Framework Core in Depth -> Store Procedures
o Several topics added and updated

Companion App

o Added: ExMeX Configuration (0.3.0)

Framework Plugins

o Updated: PowerDesigner - ExMeX Plugin - Data Vault (2.3.0)
o Updated: PowerDesigner - ExMeX Plugin - Metadata Export (1.7.0)

Framework Toolbox

o Updated: LPS Side Injection Tool (Download / Documentation)

Framework Core

In addition to some internal optimizations and minor bug fixes, the following tickets have been resolved:

o Added: 0000444: 1:1 Copy LPS
o Added: 0000341: Create ExMeX Framework variables for FK Check
o Added: 0000430: USE DB zu Beginn des LPS
o Added: 0000495: Add Stereotype 'Volatile Zone View' to CDC generator

o Updated: 0000406: Return Value of [MetadataZoneCore].uspSetLoadBatchJson

o Fixed: 0000498: Missing columns in the loading script for CDC
o Fixed: 0000501: Compare on NULL in Reference Generator
o Fixed: 0000497: #71 - Fehler bei...
o Fixed: 0000503: Compare on NULL in CDC Generator
o Fixed: 0000502: Bitemporal LPS does not identify changes in context attributes
o Fixed: 0000458: Performance Lack saving Batches from App
o Fixed: 0000457: Batch Dependency Circle chain

o Removed: 0000405: Remove deprecated table: TEDAMOH Config ExMeX Framework

Installation Notes

Download

The downloads for new installations of TEDAMOH ExMeX Framework are here.

The downloads for updating existing TEDAMOH ExMeX Frameworks are here.

New Installation

See chapter Install Framework.

Update from Version 2.11.x to 2.12.0

Follow all steps!

Before upgrading any existing TEDAMOH ExMeX Metadata framework, it is recommend to backup your Database and TEDAMOH ExMeX framework folder! If you do not backup, well it’s up to you.

1. Extract 7z-file and copy files to your root directory.

2. Change InitPath in file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.11.x to v2.12.0.sql” to your local ProjectRoot.

3. Double check if correct database for installation is set in ./usr/config/UsrConfigExMeXFramework.sql (if it is used!).

4. Double check and ensure that all tmp_*-tables in Schema MetadataZone are deleted.

5. Execute file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.11.x to v2.12.0.sql” in SSMS in SQLCMD Mode.

6. Edit global variables for ExMeX configuration if needed.

For more details see chapter Configuration in the ExMeX documentation:

How do you set global variables and what is their purpose?

7. Reinstall Metadata!

 

Posted 1 year agoby Dirk.Lerner

Release Notes 2.11.0

We are happy to bring TEDAMOH ExMeX Release 2.11.0 to you. This release contains several main features which we believe…

We are happy to bring TEDAMOH ExMeX Release 2.11.0 to you. This release contains several main features which we believe will drive you towards a better Data Warehouse solution: Reference LPS, Enhanced CDC LPS, Access Layer for Data Logistic Statistics. In Batch configuration (with ExMeX - Companion App) it will be possible to set for each batch specific values for ExitOnError and MaxSqlCommandInChunk.

We recommend to upgrade also ExMeX - Plugins and ExMeX - Companion App to the newest versions!

Have fun with the new relase,

Dirk and his Team


Pay attention to the following installation steps!

Documentation

o Change Data Capture - Data Pipeline: Link
o Reference Data Stage - Data Pipeline: Link
o Data Logistic Data Mart: Link
o Batch Orchestration: Link
o Several topics added and updated

Companion App

The ExMeX Gui was previously part of the ExMeX Framework Core. For a better overview and easier release cycles, the ExMeX Gui available as ExMeX Companion App as separate downloads beginning with ExMeX Framework Core Release 2.11.0.

Framework Plugins

The extensions for PowerDesigner were previously part of the ExMeX Framework Core. For a better overview and easier release cycles, the previous extensions are available as ExMeX Framework Plugins as separate downloads beginning with ExMeX Framework Core Release 2.11.0.

PowerDesigner Extension - TEDAMOH ExMeX Data Vault

With version 2.0.x (available with ExMeX Framework Core Release 2.11.0) this plugin was completely rewritten. The main features are a simpler integration of the Data Vault naming conventions and rules and a simplification of the modelling tasks.

Framework Tools

o Added: LPS Side Injection Tool (Download)

Framework Core

In general changes of ExMeX Framework Core data architecture. As well as data logistic is moved to Data Vault.

o Added: 0000387: Generating LPS (does not work) with LPS Side Injection
o Added: 0000192: Implementing Reference Zone in Stage Layer
o Added: 0000352: #53 [R] Eigener Stereotype um AuditId [...]
o Added: 0000321: Ignore column in CDC
o Added: 0000394: Add User Logging to all db-objects associated to companion app
o Added: 0000336: Override Proxy for SSIS execution
o Added: 0000279: Enable different parallel max chunk for batches
o Added: 0000320: Define if a batch exits on error or not
o Added: 0000373: Enhance LPS-Logging for a better reporting
o Added: 0000201: Provide Star Schema for data logistic statistics
o Added: 0000354: Documentation CDC
o Added: 0000413: Restart of SSIS-SQL Jobs (Agents) in case of Error 89

o Updated: 0000380: Move LoadBatch and LoadBatchDependency to DV
o Updated: 0000414: Update DeleteConfigLoadBatchLogisticDependency.sql and document delete Dependency

o Fixed: 0000357: Deadlock in uspParallelLPSWaitForChunk
o Fixed: 0000353: Bug with Customer Version
o Fixed: 0000356: Deadlock caused by uspParallelAsyncLPSChunkLog
o Fixed: Some minor things.
o Fixed: 0000398: Oracle can not handle =<
o Fixed: 0000399: Error-Message if LPS Side Injection is used
o Fixed: 0000350: #54 - [B] Partial CDC issue - List of Change field
o Fixed: 0000351: #55 - Bug in Installation file LoadConfigExMeXVariables.sql
o Fixed: 0000390: Show Housekeeping in DAPI only for History
o Fixed: 0000412: #63 - Issue EndDating Satellite in FULL CDC mode

o Removed: Deprecated file .\library\query\LoadBatchProcessStep.sql
o Removed: 0000282: Columns in DAPIs are deprecated. Available at least until Release 2.11.0.

Installation Notes

Download

The downloads for new installations of TEDAMOH ExMeX Framework are here.

The downloads for updating existing TEDAMOH ExMeX Frameworks are here.

New Installation

See chapter Install Framework.

Update from Version 2.10.x to 2.11.0

Follow all steps!

Before upgrading any existing TEDAMOH ExMeX Metadata framework, it is recommend to backup your Database and TEDAMOH ExMeX framework folder! If you do not backup, well it’s up to you.

1. Extract 7z-file and copy files to your root directory.

2. Remove existing files or folders in your root directory:

  • library\generate\Miscellaneous\mssqlserver\CDC-Stage.sql
  • library\generate\Miscellaneous\mssqlserver\CDC-Partially-Stage.sql
  • library\generate\DataVault20\mssqlserver\Link-Satellite EndDating OneToOne.sql
  • library\query\LoadBatchProcessStep.sql
  • library\query\AddForeignKeyConstraints.sql
  • library\query\DropForeignKeyConstraints.sql
  • library\query\DeleteOrphansProcessStepVapourTrail.sql
  • library\query\MetadataQualityCheck.sql
  • library\query\StageToCore (remove folder)
  • library\generate\DataVault20\deprecated (remove folder)

3. Change InitPath in file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.10.x to v2.11.0.sql” to your local ProjectRoot.

4. Double check if correct database for installation is set in ./usr/config/UsrConfigExMeXFramework.sql (if it is used!).

5. Double check and ensure that all tmp_*-tables in Schema MetadataZone are deleted.

6. Edit script MetadataInstall.bat as described below. See also MetadataInstall.orig.bat

1. REMOVE the following lines completly!

    echo ***********************************************************************
    echo Drop Foreign Key Constraints
    echo ***********************************************************************

    echo .
    sqlcmd -S %ServerInstance% -d %Database% -i "%ExMeXRootPath%\library\query\DropForeignKeyConstraints.sql" %DatabaseCredentials% -o "%ExMeXTmpPath%key.log"
    copy "%ExMeXTmpPath%output.log"+"%ExMeXTmpPath%key.log" "%ExMeXTmpPath%output.log" > NUL
    del "%ExMeXTmpPath%key.log"
    echo done

2a. REPLACE the following lines

    echo ***********************************************************************
    echo Installing Metadata (Prepare some things :-) )
    echo Selected section: %1
    echo ***********************************************************************

2b. WITH these lines:

    echo ***********************************************************************
    echo Installing Metadata (Prepare some things :-) )
    echo Selected section          : %1
    echo Selected database         : %Database%
    echo Selected ExMeXRootPath    : %ExMeXRootPath%
    echo Selected ExMeXMetadataPath: %ExMeXMetadataPath%
    echo ***********************************************************************

7. Edit script \etc\config\ConfigExMeXFramework.sql as described below. See also \etc\config\ConfigExMeXFramework.orig.sql

REMOVE the following lines completly!

/*+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*/
-- Schema                                                                    --
--                                                                           --
-------------------------------------------------------------------------------
-- Define schemas which are used in TEDAMOH ExMeX Framwork                   --
-- Mandantory                                                                --
--                                                                           --
-------------------------------------------------------------------------------
:setvar MetadataCoreSchema	MetadataZone
:setvar MetadataStageSchema	MetadataZoneStage
:setvar MetadataMartSchema	MetadataZoneMart
/*+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*/

8. Edit script \Manual Batch.sql as described below. See also \Manual Batch.orig.sql

1. Add section for override SSIS environment

--#############################################################################
--# Override SSIS environment
--# Please note the global settings of the ExMeX variables in the table 
--# MetadataZone.TEDAMOHConfigExMeXFramework !
-- SET @SSISLinkedServer               = NULL;
-- SET @SSISProjectName                = NULL;
-- SET @SSISFolderName                 = NULL;
-- SET @SSISProxyAccountName           = NULL;
--#
--#############################################################################

2. Add SSIS environment variables to usp

EXEC @ReturnStatus = MetadataZone.uspBatchExecuteLoadProcessStep 
                                   @BatchLabel                     = @BatchLabel
                                  ,@BatchExec                      = @BatchExec
                                  ,@BatchRestartRunVapourTrailId   = @BatchRestartRunVapourTrailId
                                  ,@SingleProcessStepId            = @SingleProcessStepId
                                  ,@ParallelExecution              = @ParallelExecution
                                  ,@MultipleBatchLabel             = @MultipleBatchLabel
                                  ,@ExitOnError                    = @ExitOnError

                                  ,@SSISLinkedServer            = @SSISLinkedServer 
                                  ,@SSISProjectName             = @SSISProjectName  
                                  ,@SSISFolderName              = @SSISFolderName  
                                  ,@SSISProxyAccountName        = @SSISProxyAccountName
--#############################################################################

3a. REPLACE the following lines

--# Default 0 = Off, 1 = Exit on Error
    SET @ExitOnError        = 0

3b. WITH these lines:

--# Default NULL |--> 0 = Off, 1 = Exit on Error
    SET @ExitOnError        = NULL

9. Edit values in script \library\env\ExMeX Init Config Variable.sql if needed as higlighted below.

EDIT values

  • MaxSqlCommandInChunk: How many LPS will run in parallel.
  • ExitOnError: If an error occurs in the batch, the batch is terminated or the error is ignored and the batch continues; Default 0 = Off, 1 = Exit
  • HousekeepingHiZAmount:Amount of units which will be kept. All others can be archived. See also housekeeping documentation.
  • HousekeepingHiZAttributeCode: Column which is used to filter for housekeeping. See also housekeeping documentation.
INSERT INTO @NameValue VALUES ('MaxSqlCommandInChunk','10','INT','Defines the ma[...];
INSERT INTO @NameValue VALUES ('ExitOnError','0','BIT','If an error occurs in th[...]; -- #Housekeeping History Zone [...]; INSERT INTO @NameValue VALUES ('HousekeepingHiZAmount','5','INT','Amount of unit[...]; INSERT INTO @NameValue VALUES ('HousekeepingHiZUnit','ProcessStepRun','NVARCHAR([...]; INSERT INTO @NameValue VALUES ('HousekeepingHiZAttributeCode','AuditTrailId','NV[...];

10. Optional step: Edit script /etc/config/ConfigLoadBatchProcessStepLogistic.sql as described below. See also /etc/config/ConfigLoadBatchProcessStepLogistic.orig.sql. You can also add the stereotype only when you create a batch for a Reference Data Stage Pipeline!

ADD line

(New Stereotype 'Reference Zone')

[...]
					  -- Assign Stereotypes to batch
					  AND iec.Stereotype IN (
                                            -- 'Volatile Zone'
                                            --,'History Zone'
                                            'Reference Zone'
                                            --,'CDC Complete'
                                            --,'CDC One Data Load Before'
                                            --,'CDC Partially'
					  					    )
[...]											
											

11. Optional step, if Batch Dependency has to be deleted: Edit script /etc/config/ConfigLoadBatchLogistic.sql as described below. See also /etc/config/ConfigLoadBatchLogistic.orig.sql.

ADD lines

(Delete Batch Dependency)

[...]
/*++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*/
-- Delete Batch Dependency                                                    --

INSERT INTO @DeleteBatchDependency (PredecessorBatchLabel,SuccessorBatchLabel)
    VALUES ('Source Interface A 2 Stage','Load Stage with SSIS')
;

/*++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*/
[...]											
											

12. Execute file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.10.x to v2.11.0.sql” in SSMS in SQLCMD Mode.

13. Reinstall Metadata!

 

Posted 1 year agoby Dirk.Lerner

Release Notes 2.10.1

We are happy to bring TEDAMOH ExMeX Release 2.10.1 to you. This release contains two main features which we believe…

We are happy to bring TEDAMOH ExMeX Release 2.10.1 to you. This release contains two main features which we believe will drive you towards leaner data pipelines and a healthy history zone. Furthermore we are introducing our brand new TEDAMOH ExMeX Companion App. Our goal is to improve user experience. The first implemented ux-feature is the new filter and housekeeping capability of the framework. And we have a lot more in the release pipeline.

Have fun with the new relase,

Dirk and his Team


Pay attention to the following installation steps!

Documentation

o Housekeeping: https://tedamoh.com/helpdesk/doc-faq/exmex-framework-book/data-logistic/how-to-do/housekeeping
o Filtering: https://tedamoh.com/helpdesk/doc-faq/exmex-framework-book/data-logistic/how-to-do/filtering
o Several topics added and updated

Companion App

o New: Introducing v0.1.0 of the TEDAMOH ExMeX Companion App to maintain filter on columns. The TEDAMOH ExMeX Companion App is located in ./opt/.

PowerDesigner

Extension TEDAMOH ExMeX.xem (> v1.6.0)

o Updated: 0000193: Delete-DML wrong column name

Framework Core

Smaller fixes due to implementation of

o Added: 0000294: Check on Batch execution start if Orphans ...
o Added: 0000278: Switch to enable return and exit on error
o Added: 0000224: Parallel chunk - add additional SQL if there is "space"
o Added: 0000309: Try catch around deactivate / activate of FK constraints
o Added: 0000198: Redesign Config-Table to Name-Value
o Added: 0000283: Concept for housekeeping HiZ-tables
o Added: 0000215: Additional filter on Entity Columns

o Updated: 0000296: Different Agent names
o Updated: 0000327: Avg rows written for decision drop/create FK
o Updated: 0000217: Hash-Diff/-Key rework (already delivered with 2.9.x)
o Updated: 0000295: better comment in WaitForDelay in usp
o Updated: 0000323: Adapt changes for SSIS package

o Fixed: 0000300: Detect error of other batch
o Fixed: 0000308: Failure of usp if stringpart '(Error' not in message from agent
o FIxed: 0000199: Column ProcessStatusMessage to short
o Fixed: 0000335: Error in CDC-identification for Full-CDC

o Removed: 0000287: Remove deprecated stereotype

Installation Notes

Download

The downloads for new installations of TEDAMOH ExMeX Framework are here.

The downloads for updating existing TEDAMOH ExMeX Frameworks are here.

New Installation

See chapter Install Framework.

Update from Version 2.9.x to 2.10.1

Follow all steps!

Before upgrading any existing TEDAMOH ExMeX Metadata framework, it is recommend to backup your Database and TEDAMOH ExMeX framework folder! If you do not backup, well it’s up to you.

1. Extract 7z-file and copy files to your root directory.

2. Change InitPath in file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.9.x to v2.10.0.sql” to your local ProjectRoot.

5. Double check if correct database for installation is set in ./usr/config/UsrConfigExMeXFramework.sql (if it is used!).

3. Double check and ensure that all tmp_*-tables in Schema MetadataZone are deleted.

4. Double check and ensure to backup custom made changes to generator files. Check with following SQL:

SELECT VersionNumber
      ,CustomerVersionNumber
      ,LoadPatternId
FROM MetadataZone.BatchExecuteLoadProcessStep
where VersionNumber <> CustomerVersionNumber
GROUP BY VersionNumber
        ,CustomerVersionNumber
        ,LoadPatternId

5. A generally valid default for housekeeping is to be made in the global settings of the ExMeX framework. Currently in the file .\library\query\LoadConfigExMeXVariables.sql, section Housekeeping History Zone. The possible values are listed in the following table.

Variable Name Value
 HousekeepingHiZUnit

Possible valid units are:

'DAY'
'WEEK'
'MONTH'
'QUARTER'
'YEAR'
'ProcessStepRun'

 HousekeepingHiZAmount

Possible values are positive integer numbers:

= 0: keep history to infinity.
> 0: keep amount of history. Remove older data.

 HousekeepingHiZAttributeCode  On which global available column the housekeeping should be done.

6. Add new paramter @ExitOnError to the script ./Manual Batch.sql as shown in ./Manual Batch.orig.sql

7. Edit script /MetadataInstall.bat by merging changes from  /MetadataInstall.orig.bat.

8. Reinstall Metadata!

 

Posted 2 years agoby Dirk.Lerner

Release Note 2.9.1

Pay attention to the following installation steps! Framework Core Smaller fixes due to implementation of o Added: 0000281: Creating Job…

Pay attention to the following installation steps!

Framework Core

Smaller fixes due to implementation of

o Added: 0000281: Creating Job Agent for SSIS (Type)

Installation Notes

Download

The downloads for new installations of TEDAMOH ExMeX Framework are here.

The downloads for updating existing TEDAMOH ExMeX Frameworks are here.

New Installation

See chapter Install Framework.

Update from Version 2.9.0 to 2.9.1

Follow all steps!

Before upgrading any existing TEDAMOH ExMeX Metadata framework, it is recommend to backup your Database and TEDAMOH ExMeX framework folder! If you do not backup, well it’s up to you.

1. Extract 7z-file and copy files to your root directory.

2. Change InitPath in file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.9.0 to v2.9.1.sql” to your local ProjectRoot.

3. Execute file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.9.0 to v2.9.1.sql” in SSMS in SQLCMD Mode.

4. Edit script /Metadata Delete And Insert CMD.sql as described below. See also /Metadata Delete And Insert CMD.orig.sql.

If used in CI/CD processes: Don't forget to edit these scripts as well!

REPLACE

-- Init Environment file for Core, e.g. Developemnt, Test, Quality or Production
:setvar CoreEnvFile   "SetEnvDevelopment.sql"
    --:setvar CoreEnvFile   "SetEnvTest.sql"
    --:setvar CoreEnvFile   "SetEnvProduction.sql"

-- Init Environment file for Incoming Interface
:setvar IncomingInterfaceEnvFile "SetEnvSourceInterface.sql"

WITH

-- Init Environment file for Core, e.g. Developemnt, Test, Quality or Production
:setvar EnvFile   "SetEnvDevelopment.sql"
    --:setvar EnvFile   "SetEnvTest.sql"
    --:setvar EnvFile   "SetEnvProduction.sql

8. Edit script /Manual Batch.sql by merging changes from  /Manual Batch.orig.bat.

9. Reinstall Metadata!

 

Posted 2 years agoby Dirk.Lerner

Release Notes 2.9.0

Pay attention to the following installation steps! Documentation o Fixed: 0000208: How to restart a batch in the case of…

Pay attention to the following installation steps!

Documentation

o Fixed: 0000208: How to restart a batch in the case of a failure?

PowerDesigner

Extension TEDAMOH ExMeX.xem (v1.6.0)

o Updated: 0000212: Performance optimization for the insert
o Updated: 0000191: Change of execution order of INSERT-DML

o Added: 0000234: More detailed Debug Mode
o Added: 0000228: PD Check on SYSNAME
o Added: 0000237: Time measurement for metadata

Script

o Updated: 0000236: Documentation FBI Automation Script

o Added: 0000248: Incorporate Index copy when copying satellites

o Fixed: 0000259: Fixing PD Copy Script (stereotype columns)

Framework Core

o Added: 0000258: Generator script to call SSIS packages. See also documentation: How to execute SSIS packackes by ExMeX Framework?
o Added: 0000251: Generator for Source 2 Stage Type SSIS
o Added: 0000258: Generator script to call SSIS packages
o Added: 0000274: Execute SSIS Packages with MAX parallel LPS
o Added: 0000281: Creating Job Agent for SSIS (Type)
o Added: 0000247: Dummy LPS for Source2Stage
o Added: 0000252: Add FK-Constraint deactivation to generators
o Added: 0000262: Check FK-Constraints if exist before set to NOCHECK
o Added: 0000194: Enable dynamic CDC in generator templates

o Fixed: 0000256: Wrong Symbol in CONVERT
o Fixed: 0000227: Some bugs in Logging
o Fixed: 0000241: If in total metadata is more than 75mb, installation slows down
o Fixed: 0000245: Within recursion of batch execution cross product exists
o Fixed: 0000195: Bug in HashDiff
o Fixed: 0000243: Start of more the one master batch failes

o Updated: 0000233: Redesign check if target/source table exists
o Updated: 0000264: Update Data Logistic Script

Framework DAPI

o Added: 0000244: Additional column
o Added: 0000277: Add table Stereotype column

o Fixed: 0000273: Issue with overlapping AK and PK in stage tables

Framework Data Quality Checks

o Updated: 0000188: Improvement of DQ-Results
o Added: 0000267: Check if Target-Object has correct stereotype, as far as possible.
o Added: 0000269: DQ-Check on AK in Hub and Link Stereotypes
o Added: 0000255: DQ check on CDC column

Tools

o Added: 0000265: Tool to generate HashDiff updates after change od Hash or Sat-Split. See also documentation: How to update HashDiff column after satellite split?

Installation Notes

Download

The downloads for new installations of TEDAMOH ExMeX Framework are here.

The downloads for updating existing TEDAMOH ExMeX Frameworks are here.

New Installation

See chapter Install Framework.

Update from Version 2.8.x to 2.9.0

Follow all steps!

Before upgrading any existing TEDAMOH ExMeX Metadata framework, it is recommend to backup your Database and TEDAMOH ExMeX framework folder! If you do not backup, well it’s up to you.

1. Extract 7z-file and copy files to your root directory.

2. Change InitPath in file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.8.x to v2.9.0.sql” to your local ProjectRoot.

3. Execute file “/install/update/Install TEDAMOH ExMeX Database Framework Release v2.8.x to v2.9.0.sql” in SSMS in SQLCMD Mode.

4. Columns in DAPIs are deprecated. Available at least until Release 2.11.0.

Columns in DAPIs are deprecated. Available at least until Release 2.11.0:

- TargetStereotype (Deprecated) --> New: TargetAttributeStereotype
- SourceStereotype (Deprecated) --> New: SourceAttributeStereotype

5. Edit script /Metadata Delete And Insert CMD.sql as described below. See also /Metadata Delete And Insert CMD.orig.sql.

If used in CI/CD processes: Don't forget to edit these scripts as well!

REPLACE

-- Init Interface
:r $(InitPath)\etc\env\SetEnvSourceInterface.sql
-- Init Env Developemnt
:r $(InitPath)\etc\env\SetEnvDevelopment.sql
-- Init Env Quality
--:r $(InitPath)\SetEnvTest.sql
-- Init Env Production
--:r $(InitPath)\SetEnvProduction.sql

WITH

-- Init Environment file for Core, e.g. Developemnt, Test, Quality or Production
:setvar CoreEnvFile   "SetEnvDevelopment.sql"
    --:setvar CoreEnvFile   "SetEnvTest.sql"
    --:setvar CoreEnvFile   "SetEnvProduction.sql"

-- Init Environment file for Incoming Interface
:setvar IncomingInterfaceEnvFile "SetEnvSourceInterface.sql"

6. Edit script /etc/config/ConfigLoadBatchProcessStepLogistic.sql as described below. See also /etc/config/ConfigLoadBatchProcessStepLogistic.orig.sql.

ADD

New Stereotype 'CDC Complete'

New Stereotype 'CDC One Data Load Before'

New Stereotype 'CDC Partially'

REMOVE

Deprecated Stereotype 'Link-Satellite CDC'

REPLACE

            INNER JOIN MetadataZone.LoadProcessStep ilps
                ON  ilps.MappingGUID       = im.MappingGUID

WITH

            INNER JOIN MetadataZone.LoadProcessStep ilps
                ON  ilps.MappingGUID       = im.MappingGUID
            INNER JOIN (
                    SELECT ProcessStepTypeId
                    FROM MetadataZone.ProcessStepType
                    WHERE ProcessStepType = 'DEF'
                    ) pst
                ON ilps.ProcessStepTypeId = pst.ProcessStepTypeId

7. Edit script /etc/env/SetEnv*.sql as described below. See also /etc/env/SetEnv*.orig.sql.

Don't remove SET @SSIS*

REMOVE (if existing)

DECLARE @SSISProjectName	NVARCHAR(256);
DECLARE @SSISFolderName 	NVARCHAR(256);
DECLARE @SSISLinkedServer 	NVARCHAR(256);

8. Edit script /MetadataInstall.bat by merging changes from  /MetadataInstall.orig.bat.

9. Reinstall Metadata!

 

Posted 2 years agoby Dirk.Lerner