Displaying 1 to 20 from 90 results

Sarama - Go library for Apache Kafka 0.8, 0.9, and 0.10.

  •    Go

Package sarama is a pure Go client library for dealing with Apache Kafka (versions 0.8 and later). It includes a high-level API for easily producing and consuming messages, and a low-level API for controlling bytes on the wire when the high-level API is insufficient.

Gizmo - A Microservice Toolkit from The New York Times

  •    Go

Gizmo Microservice Toolkit toolkit provides packages to put together server and pubsub daemons with the following features like Standardized configuration and logging, Health check endpoints with configurable strategies, Configuration for managing pprof endpoints and log levels, Structured logging containing basic request information, Useful metrics for endpoints, Graceful shutdowns, Basic interfaces to define our expectations and vocabulary and lot more.

cilium - HTTP, gRPC, and Kafka Aware Security and Networking for Containers with BPF and XDP

  •    Go

Cilium is open source software for providing and transparently securing network connectivity and loadbalancing between application workloads such as application containers or processes. Cilium operates at Layer 3/4 to provide traditional networking and security services as well as Layer 7 to protect and secure use of modern application protocols such as HTTP, gRPC and Kafka. Cilium is integrated into common orchestration frameworks such as Kubernetes and Mesos. A new Linux kernel technology called BPF is at the foundation of Cilium. It supports dynamic insertion of BPF bytecode into the Linux kernel at various integration points such as: network IO, application sockets, and tracepoints to implement security, networking and visibility logic. BPF is highly efficient and flexible. To learn more about BPF, read more in our extensive BPF and XDP Reference Guide.




goka - Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go

  •    Go

Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go. Goka aims to reduce the complexity of building highly scalable and highly available microservices. Goka extends the concept of Kafka consumer groups by binding a state table to them and persisting them in Kafka. Goka provides sane defaults and a pluggable architecture.

kafka-go - Kafka library in Go

  •    Go

sarama, which is by far the most popular but is quite difficult to work with. It is poorly documented, the API exposes low level concepts of the Kafka protocol, and it doesn't support recent Go features like contexts. It also passes all values as pointers which causes large numbers of dynamic memory allocations, more frequent garbage collections, and higher memory usage. confluent-kafka-go is a cgo based wrapper around librdkafka, which means it introduces a dependency to a C library on all Go code that uses the package. It has much better documentation than sarama but still lacks support for Go contexts.

dnpipes - Distributed Named Pipes

  •    Go

Distributed Named Pipes (or: dnpipes) are essentially a distributed version of Unix named pipes comparable to, for example, SQS in AWS or the Service Bus in Azure. Conceptually, we're dealing with a bunch of distributed processes (dpN above). These distributed processes may be long-running (such as dp0 or dp5) or batch-oriented ones, for example dp3 or dp6. There are a number of situations where you want these distributed processes to communicate, very similar to what IPC enables you to do on a single machine. Now, dnpipes are a simple mechanism to facilitate IPC between distributed processes. What follows is an interface specification as well as a reference implementation for dnpipes.

kt - Kafka command line tool

  •    Go

You can download kt via the Releases section.


sarama-cluster - Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9

  •    Go

Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9 (and later). You need to install Ginkgo & Gomega to run tests. Please see http://onsi.github.io/ginkgo for more details.

kasper - Kasper is a lightweight library for processing Kafka topics.

  •    Go

This project is currently in Beta. The API is ~95% stable so you can expect only minor breaking changes. For an introduction to Kasper and the motivation behind it, you can read our introductory blog post.

confluent-kafka-go - Confluent's Apache Kafka Golang client

  •    Go

confluent-kafka-go is Confluent's Golang client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-go is a lightweight wrapper around librdkafka, a finely tuned C client.

kafka - Load-balancing, resuming Kafka consumer for go, backed by Zookeeper.

  •    Go

Kafka libraries, tools and example applications built on top of the sarama package.The following tools can be useful for discovery, testing, and benchmarking. They also serve as examples of how to use Sarama.

vflow - Enterprise Network Flow Collector (IPFIX, sFlow, Netflow)

  •    Go

High-performance, scalable and reliable IPFIX, sFlow and Netflow collector (written in pure Golang).You can download and install pre-built debian package as below (RPM and Linux binary are available).

kafka_exporter - Kafka exporter for Prometheus

  •    Go

Kafka exporter for Prometheus. For other metrics from Kafka, have a look at the JMX exporter. Support Apache Kafka version 0.10.1.0 (and later).

watermill - Building event-driven applications easy way in Go.

  •    Go

Watermill is a Go library for working efficiently with message streams. It is intended for building event driven applications, enabling event sourcing, RPC over messages, sagas and basically whatever else comes to your mind. You can use conventional pub/sub implementations like Kafka or RabbitMQ, but also HTTP or MySQL binlog if that fits your use case. Note: Watermill should run reliably in a production environment, but it is still under heavy development and the public API may change before the 1.0.0 release.

kafka - a simpler go kafka client

  •    Go

Read will always read exactly one message or fail. Message fetches, however, are batched behind the scene, so calls to Read do not correspond 1:1 with network requests.This API is subject to change. I'd like to eventually add support for publishers, and probably move the Offsets call off of the reader interface.

kafka - Go driver for Kafka

  •    Go

Kafka is Go client library for Apache Kafka server, released under MIT license.Kafka provides minimal abstraction over wire protocol, support for transparent failover and easy to use blocking API.

kafka-pixy - gRPC/REST proxy for Kafka

  •    Go

Kafka-Pixy is a dual API (gRPC and REST) proxy for Kafka with automatic consumer group control. It is designed to hide the complexity of the Kafka client protocol and provide a stupid simple API that is trivial to implement in any language.Kafka-Pixy supports Kafka versions form 0.8.2.x to 0.10.1.x. It uses the Kafka Offset Commit/Fetch API to keep track of consumer offsets. However Group Membership API is not yet implemented, therefore it needs to talk to Zookeeper directly to manage consumer group membership.

storagetapper - StorageTapper is a scalable realtime MySQL change data streaming and transformation service

  •    Go

StorageTapper is a scalable realtime MySQL change data streaming and transformation service.Service reads data from MySQL, transforms it into an Avro schema serialized format, and publishes these events to Kafka. Consumers can then consume these events directly from Kafka.





We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.