Where Does a Data Product End? Sources, Boundaries, and the First Breaking Change

Reference tables show up all over a data solution, sometimes to encode something, sometimes to decode it again, especially when personal data is involved. Anyone using a star schema shouldn't notice any of that, though. By the time the data gets there, it has to be done already. So the data comes from somewhere else, and the consumer still only sees the finished result.

Part 1 of this series was about what a data product is, using a star schema for accounting with around 15 tables. What it left open is where the data flowing into that box comes from. There, I used customer and employee as examples. For accounting, the cost center fits better, so that's what I'm using here.

The cost center also illustrates two more questions: where a data product ends, and how to provide it so it survives the first breaking change.

Where the data comes from

The accounting data product with two inflows: incoming interfaces from upstream systems, and a second data product with the cost centers, owned by a master data team. The consumer only sees the integrated dimension

Two inflows, one result for the consumer.

Data flows into a data product's box in two ways. Either it comes in through the incoming interfaces of upstream systems, and on the source side, that's all the team is responsible for. Or the data product uses other data products.

The second way is the more interesting one. The accounting star schema needs cost centers as a dimension, but they're maintained somewhere else. If there's already a data product for them, because a master data team built one, I simply reuse it.

Whether master data needs to be its own data product is up for debate. I take a pragmatic view: it's worth it, because someone is responsible for it and takes care of it. Not every table is a data product, but I don't need to be dogmatic and say this one isn't.

For the consumer of the star schema, none of this changes anything. They shouldn't have to join, encode, or decode cost centers themselves. The accounting team handles that when it loads the star schema, and the cost center arrives as an integrated dimension. So one data product uses another, and whoever subscribes only sees the result.

Where a data product ends

Two data products side by side, cost centers and accounting, each with its own team and its own contract. The boundary between them follows the owner and the pace of change

Two teams, two contracts, two data products.

In part 1, I drew the boundary from the consumer's side: a data product reaches as far as the needs of its consumer group. The cost center adds a second boundary, and that one is drawn by the owner. Who is responsible for it, and when does it change?

The cost centers belong to the master data team, the cost allocation belongs to accounting. Each team changes things on its own schedule, so these are two data products, even though the cost center sits inside the star schema as a dimension.

My yardstick is what makes sense together and therefore gets delivered together. And while I'm at it, I think ahead to maintenance and to versioning once a breaking change comes along. The marketplace where data products are offered gives a good hint: if everyone who subscribes to one table also subscribes to the same three others, those four probably belong in one product. Orders and order line items from the hub article are a case like that.

 

 


About this series: This is part 2 of 3 in the series "Data Products: From Model to Value". The overview is in the hub article. Part 1 covers what a data product is. Part 3 (December 9) explores automation through information models.


Model dimensions, build modules

What SCD Type 1 and Type 2 mean for a dimension is part of the Dimensional Modeling Fundamentals Class. How modules run independently of each other is what I teach in Master the Art of Data Engine Thinking, as eLearning and in person, with current dates on our website.

→ Go to the Dimensional Modeling Fundamentals Class
→ Go to Master the Art of Data Engine Thinking

Providing it, even through a breaking change

Version 1 and version 2 of the cost center dimension side by side. Only the few affected modules are added in a new version, all other modules of the star schema stay unchanged

Only what changes is new.

The contract from part 1 guarantees the consumer that nothing gets deleted or renamed without warning. At some point, something still changes in a way that no longer fits the existing contract. That's a breaking change.

In the example, it could look like this. So far, the cost center dimension has been built as SCD Type 1, holding only the current state, and now it starts keeping history (SCD Type 2). Or it already was SCD Type 2 and gets new columns that change how history is tracked. For consumers who don't have that on their radar, the consequences can be serious. They don't need the new columns, and from their point of view, the dimension suddenly contains duplicate rows.

The hub already gives the answer: put version 2 next to version 1, and consumers move over when they're ready. That doesn't mean copying the whole data product, though. Behind a star schema, there may be thirty modules (data logistics processes), and switching to SCD Type 2 affects one, two, or three of them. Only those get added in a new version. Whatever stays the same stays as it is, with no copy needed. That works because modules in Data Engine Thinking run independently of each other. You add a new module, and nothing changes in the existing ones.

And what if the breaking change comes from outside? If the master data team starts keeping history for its cost centers, the accounting team is the consumer. That's exactly where the second boundary pays off. The cost center has its own contract and its own version, and the accounting team decides when to move over. Its own consumers only notice once it provides a new version itself.

With two teams and two contracts, one thing stands out right away: each team describes the cost center in its own contract, and not necessarily in the same words. How an information model can make sure it still means the same thing everywhere, and what can be automated from there, is the topic of part 3 on December 9. Before that, my conversation with Simon Harrer about data products comes out on November 24 as episode #005 of [antifragile data] (in German).

So long,
Dirk

 


Sources
The accounting star schema is the teaching example from the Dimensional Modeling Fundamentals Class. The principle of independent modules comes from Roelant Vos and Dirk Lerner, Data Engine Thinking.