Real-time Data Streaming with Kafka Connect
Why Kafka Connect? While you can always write your own Kafka connector to write data from Kafka to S3 or a database using for example confluent-kafka-python, this might be hard to maintain and error prone. Kafka Connect can help you to simplify this task. In this post we will … … set up a local Kafka cluster, S3 storage & Kafka Connect with Docker Compose … create a Kafka topic and publish messages to it … use Kafka Connect to create an S3 sink connector and write the messages to S3 Run the Example You can run the full example which is described in this post by executing the following script: It will download the necessary files, start the containers, create a Kafka topic, publish messages, and create an S3 sink connector to write the data to S3:...