Scribe - Real time log aggregation used in Facebook

  •        12979

Scribe is a server for aggregating log data that's streamed in real time from clients. It is designed to be scalable and reliable. It is developed and maintained by Facebook. It is designed to scale to a very large number of nodes and be robust to network and node failures. There is a scribe server running on every node in the system, configured to aggregate messages and send them to a central scribe server (or servers) in larger groups.

https://github.com/facebook/scribe

Tags
Implementation
License
Platform

   




Related Projects

Kafka - A high-throughput distributed messaging system

  •    Java

Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.

liftbridge - Lightweight, fault-tolerant message streams.

  •    Go

Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system. It extends NATS with a Kafka-like publish-subscribe log API that is highly available and horizontally scalable. Use Liftbridge as a simpler and lighter alternative to systems like Kafka and Pulsar or use it to add streaming semantics to an existing NATS deployment. See the introduction post on Liftbridge and this post for more context and some of the inspiration behind it.

X-Itools: Enterprise Collaboration

  •    Javascript

Enterprise Collaboration modules and strong Log Analysis modules

Luxun - A high-throughput, persistent, distributed, publish-subscribe messaging system based on memo

  •    Java

A high-throughput, persistent, distributed, publish-subscribe messaging system based on memory mapped file and Thrift RPC.


Sentry - Realtime Platform-Agnostic Error Logging and Aggregation platform

  •    Python

Sentry is a realtime event logging and aggregation platform. It specializes in monitoring errors and extracting all the information needed to do a proper post-mortem without any of the hassle of the standard user feedback loop.

jafka - a fast distributed publish-subscribe messaging system (mq)

  •    Java

a fast distributed publish-subscribe messaging system (mq)

White-elephant - Hadoop log aggregator and dashboard

  •    Java

White Elephant is a Hadoop log aggregator and dashboard which enables visualization of Hadoop cluster utilization across users. The server is a JRuby web application. In a production environment it can be deployed to tomcat and reads aggregated usage data directly from Hadoop. This data is stored in an in-memory database provided by HyperSQL. Charting is provided by Rickshaw. This project is developed by LinkedIn.

emitter - High performance, distributed and low latency publish-subscribe platform.

  •    Go

Emitter is a free open source real-time messaging service that connects all devices. This publish-subscribe messaging API is built for speed and security. Emitter is a real-time communication service for connecting online devices. Infrastructure and APIs for IoT, gaming, apps and real-time web. At its core, emitter.io is a distributed, scalable and fault-tolerant publish-subscribe messaging platform based on MQTT protocol and featuring message storage.

kids - Kids Is Data Stream

  •    C++

Kids is a log aggregation system. It aggregates messages like Scribe and its pub/sub pattern is ported from Redis.

MinPubSub - 0.1kb publish/subscribe messaging micro-framework

  •    Javascript

A micro publish/subscribe messaging framework, weighing in at only 198 bytes gzipped. Created by rewriting Peter Higgins' jQuery plugin, MinPubSub is completely self contained with no external dependencies. Keep your projects loosely coupled with this powerful design pattern. Also available on NPM and Bower. If you don't want these methods added to the window object simply modify the immediate function to pass in the JavaScript object of your choosing.

GoAccess - Real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser

  •    C

GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal on *nix systems or through your browser. It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly. It supports nearly all web log formats (Apache, Nginx, Amazon S3, Elastic Load Balancing, CloudFront, etc)

gnatsd - High-Performance server for NATS, the cloud native messaging system.

  •    Go

NATS Server is a simple, high performance open source messaging system for cloud native applications, IoT messaging, and microservices architectures. It implements a highly scalable and elegant publish-subscribe (pub/sub) distribution model. The performant nature of NATS make it an ideal base for building modern, reliable, scalable cloud native distributed systems.

Faye - Simple pub/sub messaging for the web

  •    Javascript

Faye is a set of tools for simple publish-subscribe messaging between web clients. It ships with easy-to-use message routing servers for Node.js and Rack applications, and clients that can be used on the server and in the browser. It is based on the Bayeux protocol.

scribe - Scribe is a server for aggregating log data streamed in real time from a large number of servers

  •    C++

Scribe is a server for aggregating log data streamed in real time from a large number of servers.

Kafka-Message-Server - Example application based on Apache Kafka framework to show it usage as distributed message server

  •    Java

Apache kafka is yet another precious gem from Apache Software Foundation. Kafka was originally developed at Linkedin and later on became a member of Apache project. Apache Kafka is a distributed publish-subscribe messaging system. Kafka differs from traditional messaging system as it is designed as distributed system, persists messages on disk and supports multiple subscribers. Kafka-Message-Server is an sample application for demonstrating kafka usage as message-server. Please follow the below instructions for productive use of the sample application.

Fluentd - Data collector, Log Everything in JSON

  •    Ruby

Fluentd is an event collector system. It is a generalized version of syslogd, which handles JSON objects for its log messages. It collects logs from various data sources and writes them to files, database or other types of storages.

MBassador - Light-weight, high-performance event bus implementing the publish subscribe pattern

  •    Java

MBassador is a light-weight, high-performance event bus implementing the publish subscribe pattern. It is designed for ease of use and aims to be feature rich and extensible while preserving resource efficiency and performance. The core of MBassador is built around a custom data structure that provides non-blocking reads and minimized lock contention for writes such that performance degradation of concurrent read/write access is minimal.

ruby-mqtt - Pure Ruby gem that implements the MQTT protocol, a lightweight protocol for publish/subscribe messaging

  •    Ruby

Pure Ruby gem that implements the MQTT protocol, a lightweight protocol for publish/subscribe messaging. Also includes a class for parsing and generating MQTT-SN packets.

js-signals - Custom Event/Messaging system for JavaScript inspired by AS3-Signals

  •    Javascript

Custom event/messaging system for JavaScript inspired by AS3-Signals.For a more in-depth introduction read the JS-Signals Project Page and visit the links below.