Scribe - Real time log aggregation used in Facebook

  •        12589

Scribe is a server for aggregating log data that's streamed in real time from clients. It is designed to be scalable and reliable. It is developed and maintained by Facebook. It is designed to scale to a very large number of nodes and be robust to network and node failures. There is a scribe server running on every node in the system, configured to aggregate messages and send them to a central scribe server (or servers) in larger groups.

https://github.com/facebook/scribe

Tags
Implementation
License
Platform

   




Related Projects

Kafka - A high-throughput distributed messaging system


Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.

X-Itools: Enterprise Collaboration


Enterprise Collaboration modules and strong Log Analysis modules

Luxun - A high-throughput, persistent, distributed, publish-subscribe messaging system based on memo


A high-throughput, persistent, distributed, publish-subscribe messaging system based on memory mapped file and Thrift RPC.

Sentry - Realtime Platform-Agnostic Error Logging and Aggregation platform


Sentry is a realtime event logging and aggregation platform. It specializes in monitoring errors and extracting all the information needed to do a proper post-mortem without any of the hassle of the standard user feedback loop.


jafka - a fast distributed publish-subscribe messaging system (mq)


a fast distributed publish-subscribe messaging system (mq)

White-elephant - Hadoop log aggregator and dashboard


White Elephant is a Hadoop log aggregator and dashboard which enables visualization of Hadoop cluster utilization across users. The server is a JRuby web application. In a production environment it can be deployed to tomcat and reads aggregated usage data directly from Hadoop. This data is stored in an in-memory database provided by HyperSQL. Charting is provided by Rickshaw. This project is developed by LinkedIn.

GoAccess - Real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser


GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal on *nix systems or through your browser. It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly. It supports nearly all web log formats (Apache, Nginx, Amazon S3, Elastic Load Balancing, CloudFront, etc)

Faye - Simple pub/sub messaging for the web


Faye is a set of tools for simple publish-subscribe messaging between web clients. It ships with easy-to-use message routing servers for Node.js and Rack applications, and clients that can be used on the server and in the browser. It is based on the Bayeux protocol.

Fluentd - Data collector, Log Everything in JSON


Fluentd is an event collector system. It is a generalized version of syslogd, which handles JSON objects for its log messages. It collects logs from various data sources and writes them to files, database or other types of storages.

MBassador - Light-weight, high-performance event bus implementing the publish subscribe pattern


MBassador is a light-weight, high-performance event bus implementing the publish subscribe pattern. It is designed for ease of use and aims to be feature rich and extensible while preserving resource efficiency and performance. The core of MBassador is built around a custom data structure that provides non-blocking reads and minimized lock contention for writes such that performance degradation of concurrent read/write access is minimal.

MinPubSub - 198 byte publish/subscribe messaging micro-framework


198 byte publish/subscribe messaging micro-framework

ruby-mqtt - Pure Ruby gem that implements the MQTT protocol, a lightweight protocol for publish/subscribe messaging


Pure Ruby gem that implements the MQTT protocol, a lightweight protocol for publish/subscribe messaging. Also includes a class for parsing and generating MQTT-SN packets.

js-signals - Custom Event/Messaging system for JavaScript inspired by AS3-Signals


Custom event/messaging system for JavaScript inspired by AS3-Signals.For a more in-depth introduction read the JS-Signals Project Page and visit the links below.

Octopussy - Perl/XML Logs Analyzer, Alerter & Reporter


Octopussy is a Log analyzer tool. It analyzes the log, generates reports and alerts the admin. It has LDAP support to maintain users list. It exports report by Email, FTP & SCP. Scheduled reports could be generated. RRD tool to generate graphs.

Distributed Publish/Subscribe (Pub/Sub) Event System


The Web Solutions Platform (WSP) event system is a general-purpose distributed publish/subscribe event system designed for high performance and low latency. Publish/subscribe is a common communication pattern though there are many variations to solve different scenario require...

Webalizer - fast web server log file analysis


The Webalizer is a fast web server log file analysis program. It produces highly detailed, easily configurable usage reports in HTML format, for viewing with a standard web browser. It handles standard Common logfile format (CLF) server logs, several variations of the NCSA Combined logfile format, wu-ftpd/proftpd xferlog (FTP) format logs, Squid proxy server native format, and W3C Extended log formats.

Awstats - Advanced web, streaming, ftp and mail server statistics


AWStats is a powerful tool that generates advanced web, streaming, ftp or mail server statistics graphically. It can analyze log files from all major server tools like Apache log files, WebStar, IIS and a lot of other web, proxy, wap, streaming servers, mail servers and some ftp servers. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages.

NSQ - A realtime distributed messaging platform in Go


NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. It scales horizontally, without any centralized brokers. Built-in discovery simplifies the addition of nodes to the cluster.