clickhouse_sinker - Easily load data from kafka to ClickHouse with high performance

  •        93

clickhouse_sinker is a sinker program that consumes kafka message and import them to ClickHouse.

https://github.com/housepower/clickhouse_sinker

Tags
Implementation
License
Platform

   




Related Projects

Tabix - SQL Editor & Open source simple business intelligence for Clickhouse.

  •    Javascript

Tabix is a SQL Editor & Open source simple business intelligence for Clickhouse. No need to install, it works from the browser. It provides support to Draw charts, Maps of the world, Metrics RealTime charts from system.metrics, Displays database and tables as tree and lot more.

ClickHouse - Columnar DBMS and Real Time Analytics

  •    C++

ClickHouse is an open source column-oriented database management system capable of real time generation of analytical data reports using SQL queries. It is Linearly Scalable, Blazing Fast, Highly Reliable, Fault Tolerant, Data compression, Real time query processing, Web analytics, Vectorized query execution, Local and distributed joins. It can process hundreds of millions to more than a billion rows and tens of gigabytes of data per single server per second.

generate-schema - Convert JSON Objects to MySQL Schema, JSON Schema, Mongoose Schema, Google BigQuery, or a Generic template for Swagger files, documentation, code generation, and more

  •    Javascript

Convert JSON Objects to MySQL Table Schema, JSON Schema, Mongoose Schema, ClickHouse Schema, Google BigQuery, or a Generic template for documentation, code generation, and more. Optionally, add -g to the above if you want the generate-schema command line executable.

Kafka-Message-Server - Example application based on Apache Kafka framework to show it usage as distributed message server

  •    Java

Apache kafka is yet another precious gem from Apache Software Foundation. Kafka was originally developed at Linkedin and later on became a member of Apache project. Apache Kafka is a distributed publish-subscribe messaging system. Kafka differs from traditional messaging system as it is designed as distributed system, persists messages on disk and supports multiple subscribers. Kafka-Message-Server is an sample application for demonstrating kafka usage as message-server. Please follow the below instructions for productive use of the sample application.


kafka-websocket - Websocket server interface for Kafka distributed message broker

  •    Java

kafka-websocket is a simple websocket server interface to the kafka distributed message broker. It supports clients subscribing to topics, including multiple topics at once, and sending messages to topics. Messages may be either text or binary, the format for each is described below. A client may produce and consume messages on the same connection.

Project Server 2007 Bulk Edit

  •    

The purpose of this project is to allow the bulk edit of resource data. With large numbers of resources Resource Center can take a long time to load, primarily due to the security checks required. This tool goes directly against the Project Server Reporting store to pull back t

EFCore

  •    CSharp

EntityFrameworkCore extensions: Bulk operations (Insert, Update, Delete, Read, Upsert, Sync) and Batch (Delete, Update). Library is Lightweight and very Efficient, having all mostly used CRUD operation. Was selected in top 20 EF Core Extensions recommended by Microsoft. It is targeting NetStandard 2.0 so it can be used on project targeting NetCore(2.0+) or NetFramework(4.6.1+). Current version is using EF Core 2.2 and at the moment supports Microsoft SQL Server(2008+) and SQLite. EFCore/v.Nuget: EFCore2.1/v2.4.1 EFCore2.0/v2.0.8, and for EF Core 1.x use 1.1.0 (targeting NetStandard 1.4) Under the hood uses SqlBulkCopy for Insert, for Update/Delete combines BulkInsert with raw Sql MERGE. For SQLite there is no BulkCopy, instead library uses plain SQL combined with UPSERT. Bulk Tests can not have UseInMemoryDb because InMemoryProvider does not support Relational-specific methods. If Windows Authentication is used then in ConnectionString there should be Trusted_Connection=True; because Sql credentials are required to stay in connection.

SQL Compact Bulk Insert Library

  •    

.NET Library for loading data fast (doing bulk inserts) into a SQL Server Compact database file. Attempts to mimic the SQLClient SqlBulkCopy API.

BCP2SQLAzure: Free tool to import/export data between SQL Azure and SQL Server

  •    

SQL Azure BCP makes it easier for database admins and developers, to do bulk copy loads into SQL Azure. This bcp replacement allows you to define each load using an easily XML file. It´s developed using C#.

SQL sliced bulk inserter

  •    

Alternative approach to bulk insert delimited files when extreme performance is required. Input: some large delimited file (GBs) Operation: BULK INSERT Destination: SQL Server table(s) Method: Barbaric approach Technology: SQLNCLI, Mem mapped files, Windows 2008, C++2008

Headwind MDM – Open Source Mobile Device Management System for Android

  •    Java

Headwind MDM is the software for managing corporate mobile (Android-based) devices. System administrators can track phones and tablets, perform bulk installation and update of mobile applications, prevent user from entering the device settings and running unwanted apps.

Bulk JSON HttpHandler

  •    ASPNET

The Bulk Json Handler allows in ASP.NET to accomplish in one call, execute multiple json methods on the server and push return values from one method to another based on JSON information sent to server without client interaction.

Bulk Actions for SharePoint

  •    

This project aims to provide some essential and generic bulk actions for SharePoint lists. Idea is to include any custom actions that can be applied on lists in bulk. List item selection is also provided, so not only you can do your actions in bulk but also selectively.

confluent-kafka-dotnet - Confluent's Apache Kafka .NET client

  •    CSharp

confluent-kafka-dotnet is Confluent's .NET client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-dotnet is a lightweight wrapper around librdkafka, a finely tuned C client.

confluent-kafka-python - Confluent's Apache Kafka Python client

  •    Python

confluent-kafka-python is Confluent's Python client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-python is a lightweight wrapper around librdkafka, a finely tuned C client.

kafka-rest - Confluent REST Proxy for Kafka

  •    Java

The Kafka REST Proxy provides a RESTful interface to a Kafka cluster, making it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients.

ruby-kafka - A Ruby client library for Apache Kafka

  •    Ruby

A Ruby client library for Apache Kafka, a distributed log and message bus. The focus of this library will be operational simplicity, with good logging and metrics that can make debugging issues easier.Although parts of this library work with Kafka 0.8 – specifically, the Producer API – it's being tested and developed against Kafka 0.9. The Consumer API is Kafka 0.9+ only.