Building an EAI Middleware Layer
The Changes Are Already in the Log
In one line
CDC (Change Data Capture) is an integration method that, instead of waiting for an application to send messages, reads changes that have already happened in the database and streams them to other systems. The most reliable method is to read the DB's transaction log, and in exchange, new operational responsibilities arise, such as log retention, schema changes and ordering.
Why it was needed
Up to now, every integration in this course has had a structure in which the sending side sends. Core banking hands the processing result to the hub, and the hub carries it to the information system and the external gateway. But in the field there are many systems that cannot be attached this way. Packages whose source cannot be changed, 20-year-old ledger batches, and legacy systems where dozens of screens edit the same table directly are like that. The demand "send us a message whenever a change occurs" means finding and modifying every write point of that system, and if even one is missed, data quietly drifts apart.
The second reason is dual writes. If an application writes to the DB and then also publishes to MQ, at the moment the process dies between the two, a change appears that is in the DB but not in MQ. A distributed transaction that ties the two resources into one transaction is heavy, and most brokers do not support it. If only the DB write is a transaction and publishing is done by reading changes the DB has already finalized, this gap disappears. CDC is that reading side.
How it works
There are roughly three ways to capture changes.
| Method | Principle | Cost |
|---|---|---|
| Query (polling) | Read periodically with updated_at > 마지막 시각 (last time) |
Cannot see deletes. Misses changes at the same time or transactions committed late. Query load on the source DB |
| Trigger | A table trigger writes one row at a time to a change history table | Trigger cost on every write. You have to plant triggers in the source DB |
| Log-based | Read the DB's transaction log (PostgreSQL WAL, MySQL binlog) | You have to operate log retention, permissions and version compatibility |
The reason log-based is the most reliable is that it reads what the DB has already recorded in commit order. No matter what path the application code wrote by, deletes and bulk updates all remain in the log.
A representative open-source implementation is Debezium. The PostgreSQL connector reads the WAL with logical decoding. As the output plugin it uses pgoutput, which is built in by default on PostgreSQL 10 and later, or decoderbufs, which Debezium manages, and to use logical decoding the source DB's wal_level must be logical (PostgreSQL documentation). The MySQL connector reads the binlog and turns row-level INSERT, UPDATE and DELETE into events (Debezium MySQL).
First a snapshot once, then streaming. The log does not remain forever, so when it first attaches, the connector reads the entire table at a consistent point in time (the snapshot) and emits it as events, and then streams changes continuing from the log position at that point. The documentation explains that because streaming starts from the log position read during the snapshot, changes in between are not missed.
Events carry both before and after. The body of a Debezium change event has before (the row before the change), after (the row after the change), source (which DB, table and log position it came from), op (the operation) and ts_ms (the processing time). op is c create, u update, d delete, r snapshot read and t truncate. What before holds in PostgreSQL is decided by the table's REPLICA IDENTITY — with the default (DEFAULT), update and delete events carry only the previous values of the primary key columns, and with FULL, the previous values of all columns. If you need "what did the balance change from and to," look at this setting first.
A replication slot holds on to the log. The PostgreSQL connector uses a replication slot to record in the DB how far it has read. Even if the connector stops, the DB does not delete the WAL the slot has not yet read — so on restart it reads on from where it stopped. Put the other way around, if the connector is dead for days, the source DB's disk fills with WAL. The moment you attach CDC, source DB operations gain one more item to monitor. On the MySQL side there is a risk in the opposite direction. The documentation notes that since binlogs are deleted after the retention period, if the connector stops longer than that, the position it was reading is gone and a new snapshot is needed.
Use it with the outbox pattern. If you stream table changes as they are, consumers are tied to the source table structure (if a column name changes, the consumer breaks). So inside the business transaction, write one row for the event to be published into an outbox table as well, and CDC reads only that outbox table and emits it. It hides the source schema and removes the dual-write problem. Debezium provides an Outbox Event Router transformation that turns rows of the outbox table into events.
What it looks like in the field
First, a batch that implemented "just give me the changes" with an updated_at query misses deletes forever. A closed account remains alive in the information system for months. Second, CDC moves changes, not meaning. The fact that three row changes in the ledger table (a withdrawal row, a deposit row and a balance row) are one transfer is not in the log. If you need business events, you have to write the meaning with an outbox. Third, delivery is at least once. When the connector restarts, events already sent can come again, so consumers must do module 8's idempotent handling as it is — the log position in source or the event key serves as the basis. Fourth, schema changes. If a column is added to the source, the shape of the event changes. This is why you put the consumer contract not on the source table but on the outbox's event format.
Let us also sort out the relationship with EAI. CDC does not replace the relay layer. Transactions that need a request and response (transfer approval, limit lookup) are still done by synchronous relay. CDC suits when other systems need to know about something already done — loading the information system, search indexing, cache invalidation, notifications.
Summary and quiz
This module summarizes concepts without a lab. You check the difference between polling, triggers and log-based, snapshot and streaming, the shape of a change event, the operational responsibilities of replication slots and log retention, and the outbox by quiz.