47 best open source kafka connect projects.

We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.

Displaying 1 to 20 from 47 results

Kafka UI - Open-Source Web UI for Apache Kafka Management

TypeScript

Kafka UI is a UI for Apache Kafka to monitor and manage Apache Kafka clusters. It is a simple tool that makes your data flows observable, helps find and troubleshoot issues faster and deliver optimal performance. Its lightweight dashboard makes it easy to track key metrics of your Kafka clusters - Brokers, Topics, Partitions, Production, and Consumption.

kafka big-data web-ui streams kafka-connect apache-kafka-ui kafka-producer kafka-client kafka-streams hacktoberfest streaming-data kafka-manager kafka-cluster event-streaming cluster-management kafka-brokers

stream-reactor - Streaming reference architecture for ETL with Kafka and Kafka-Connect

Scala

Lenses offers SQL (for data browsing and Kafka Streams), Kafka Connect connector management, cluster monitoring and more. A collection of components to build a real time ingestion pipeline.

kafka kafka-connect connector streaming cassandra hazelcast redis elasticsearch ftp influxdb coap mqtt kudu jms hbase mongodb rethinkdb documentdb cosmosdb kubernetes

Debezium - Stream changes from your databases.

Java

Debezium is a distributed platform that turns your existing databases into event streams, so applications can see and respond immediately to each row-level change in the databases. Debezium is built on top of Apache Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, from where your application consumes them. This makes it possible for your application to easily consume all of the events correctly and completely.

change-data-capture kafka-connect apache-kafka debezium cdc database kafka kafka-producer database-migration events

strimzi-kafka-operator - Apache Kafka running on Kubernetes and OpenShift

Java

Strimzi provides a way to run an Apache Kafka cluster on Kubernetes or OpenShift in various deployment configurations. See our website for more details about the project. Documentation to the current master branch as well as all releases can be found on our website.

kafka kubernetes openshift docker messaging enmasse kafka-connect kafka-streams data-streaming data-stream data-streams kubernetes-operator kubernetes-controller

mongo-kafka - MongoDB Kafka Connector

Java

The official MongoDB Kafka Connector. The connector will be published on maven central.

kafka mongodb avro source kafka-connect bson cdc sink-connector sink source-connector confluent-hub

cp-docker-images - Docker images for Confluent Platform.

Python

Docker images for deploying and running the Confluent Platform. The images are currently available on DockerHub. They are currently only available for Confluent Platform 3.0.1 and after.Full documentation for using the images can be found here.

confluent kafka docker-images kafka-connect kafka-streams

kafka-connect-elasticsearch - Kafka Connect Elasticsearch connector

Java

Elasticsearch Sink Connector for Kafka Connect

confluent kafka kafka-connector kafka-connect elasticsearch

kafka-connect-storage-cloud - Kafka Connect suite of connectors for Cloud storage (currently including Amazon S3)

Java

Documentation for this connector can be found here.To build a development version you'll need a recent version of Kafka. You can build kafka-connect-storage-cloud with Maven using the standard lifecycle phases.

kafka kafka-connect confluent cloud s3 azure google-cloud google-cloud-storage aws aws-s3

kafka-connect-storage-common - Shared software among connectors that target distributed filesystems and cloud storage

Java

Shared software modules among Kafka Connectors that target distributed filesystems and cloud storage.To build a development version you'll need a recent version of Kafka. You can build kafka-connect-storage-common with Maven using the standard lifecycle phases.

kafka kafka-connect confluent

connect-utils - Utility project for working with Kafka Connect.

Java

A helper library for building Kafka Connect Connectors.

kafka-connect kafka conversion parsing schema

kafka-connect-solr - Kafka Connect connector for writing to Solr.

Java

A Kafka Connect connector copying data from Kafka to Solr.

kafka-connect solr

kafka-connect-splunk - Kafka Connect connector for receiving data and writing data to Splunk.

Java

This connector allows Kafka Connect to emulate a Splunk Http Event Collector. This connector support receiving data and writing data to Splunk. The Sink Connector will transform data from a Kafka topic into a batch of json messages that will be written via HTTP to a configured Splunk Http Event Collector.

kafka-connect splunk splunk-http

kafka-connect-spooldir - Kafka Connect connector for reading CSV files into Kafka.

Java

A Kafka Connect connector reading delimited files from the file system.

kafka-connect csv json

kafka-connect-twitter - Kafka Connect connector to stream data in real time from Twitter.

Java

Kafka Connect plugin for streaming data from Twitter to Kafka.

kafka-connect twitter

kafka-compose - :musical_score: Docker compose files for various kafka stacks

Python

Replicates typical Kafka stack using docker compose.

kafka spark twitter docker-compose avro kafka-connect pyspark

kafka-jdbc-connector - Simple way to copy data from relational databases into kafka.

Scala

Simple way to copy data from relational databases into kafka. To copy data between Kafka and another system, users create a Connector for the system which they want to pull data from or push data to. Connectors come in two flavors: SourceConnectors to import data from another system and SinkConnectors to export data from Kafka to other datasources. This project is an implementation of SourceConnector which allows users to copy data from relational databases into Kafka topics. It provides a flexible way to keep logic of selecting next batch of records inside database stored procedures which is invoked from the connector tasks in each poll cycle.

kafka-connect jdbc-connector

fast-data-connect-cluster - Create Kafka-Connect clusters with docker

Shell

A docker image for setting up Kafka Connect clusters. This part of fast-data-dev is targeted to more advanced users and is a special case since it doesn't set-up a Kafka cluster, instead it expects to find a Kafka Cluster with Schema Registry up and running.

kafka-connect

kafka-connect-kcql-smt - Kafka-Connect SMT (Single Message Transformations) with SQL syntax (Using Apache Calcite for the SQL parsing)

Scala

Use SQL to drive the transformation of the Kafka message(key or/and value) when using Kafka Connect. Before SMT you needed a KStream app to take the message from the source topic apply the transformation to a new topic. We have developed a KStreams library ( you can find on github) to make it easy expressing simple Kafka streams transformations. However the extra topic is not required anymore for Kafka Connect!.

kafka kafka-connect kcql

kafka-connect-tools - Kafka Connect Tooling

Scala

This is a tiny command line interface (CLI) around the Kafka Connect REST Interface to manage connectors. It is used in a git like fashion where the first program argument indicates the command: it can be one of [ps|get|rm|create|run|status|status|plugins|describe|validate|restart|pause|resume]. The CLI is meant to behave as a good unix citizen: input from stdin; output to stdout; out of band info to stderr and non-zero exit status on error. Commands dealing with configuration expect or produce data in .properties style: key=value lines and comments start with a #.

kafka kafka-connect cli