I'm using Docker environment to do this, however if you want to try it on host machine, you can use Confluent Platform installed on your OS.

First, we need the first database to have some data. Open 

 option will prompt us to input password to connect. Once we are in the 

To enable CDC with Postgres, we need to give replication permission to our database: 

Next, we need to establish a connector using Debezium connector to access our first database and stream it to a Kafka topic. Fortunately, it's quite easy to do that. Debezium connector can be accessed through REST API.

There are two connectors we need to create, one is for 

, the process when stream are inserted into a topic, and the other one is 

 class. Below is an example configs on how to do it.



sharingcdc-source-connector

 tells us the connection is successfully created.

sharingcdc-connector

If you want to stop our connector, it is important to 

 it before starting new connector with the same name.

For example above, we declared that the connector name is 

, to delete this connector we simply send a 


To confirm that the connector has been deleted, inquiry the connectors that we have using:

 endpoint is the root context for Debezium connectors, and sending a 

 request without any parameters returns the list of available connectors.

According to the official documentation of Debezium connector for postgres : 

, the topic name is generated automatically using the 

, we defined a transformation route which transforms the predefined generated topic into just table name with 

So in our case, to name of our topic should be `todo` instead of 



To check your topic list, Kafka Connect also provide a CLI to check our topics using the 

 directory.

Open terminal in our Kafka Connect container, and run:


 connector is actually able to stream the data changes. There are several ways to do this:

Now that we have tailed our connector, we can confirm by trying to modify data, insert new data or even delete data.

What we will get in the topic monitoring console

This REST API will show us the schema as below.

Ever wondered how to duplicate your database in real time without pain? There is Change Data Capture which we will combine with Debezium to stream the data changes into Kafka topic and make use of it. This post uses Postgres to Postgres databases for the example.

Debezium for CDC : Postgres to Postgres

Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

BLOG TAG

Kafka

Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Here are some articles tagged with "Kafka" :