This time we dive into one of my favorite topics - CDC - which means “Change Data Capture”.
CDC “unchains” data. This is especially important if you need to integrate with existing (legacy) systems. The first question you typically have to answer is - how do I get the data out of the legacy system?
CDC typically sits directly on the relevant tables in the legacy system. This allows us to capture all changes to rows in these tables.
I love to work with Apache Kafka and also Kafka Connect for this.
With Kafka Connect, I can just add a so called “JDBC Source” Connector that gets triggered when a row gets added or updated ( delete is not so easy ).
If we have the possibility to make small adjustments to the legacy system, using a Transactional Outbox Table often simplifies the process. Everytime time a business operation gets triggered in the legacy system, we simply store a new entry in the Outbox-Table. In fact it´s nothing but an Event as an indication that the state of the legacy system has changed.
How do you model that?
If we model a system, we want to show how and when external information enters the system. I typically treat the CDC-“Event” as an external Event that needs to be translated to something internal and easily usable by the system.
External Events are yellow for me - I want to see immediately, where in the system we are dealing with external data.
Oftentimes the CDC Event is overly generic and I try to limit the effect of this to as few places as possible.
The “Price Changed” Event is something internal and well-known to the system. From the system perspective it doesn´t even matter technically how price changes comes into the system. We might switch from CDC to an API-based integration. In this case we just delete the “Slice” for the CDC Integration and replace it with an Automation to fetch prices by API.
So the really important part in this process is the Price Translation - it acts as an Anti-Corruption-Layer and shields us from the external (legacy) System.
And another important thing - we can also show how this translation works using Given / When / Thens. Sometimes a rather generic CDC Events can result in many different “Domain”-Events in our system, depending on the changes that were captured. It´s important to visualize the logic that applies.
Hope that helped. If you have questions, I´d suggest we have a quick chat about it. Feel free to just contact me directly here on LinkedIn.
Always here to help.
Cheers
Martin
Ready to Learn More?
My book “Understanding Eventsourcing” gives you the blueprint. But reading alone will take your team too long.
I can teach your team how to build these blueprints faster and skip the whole trial-and-error phase. Let’s have a chat about how this applies to your project.
Still 2 Team-Spots left for the Event Modeling Workshop this month.
Want to learn how to apply Event Modeling and Event Sourcing in practice?
Follow the Online Course “Implementing Eventsourcing” - comes with a Lifetime Event Modeling Toolkit License.