Scribe - Real time log aggregation used in Facebook

  •        0

Scribe is a server for aggregating log data that's streamed in real time from clients. It is designed to be scalable and reliable. It is developed and maintained by Facebook. It is designed to scale to a very large number of nodes and be robust to network and node failures. There is a scribe server running on every node in the system, configured to aggregate messages and send them to a central scribe server (or servers) in larger groups.



comments powered by Disqus

Related Projects

Kafka - A high-throughput distributed messaging system

Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.

Sentry - Realtime Platform-Agnostic Error Logging and Aggregation platform

Sentry is a realtime event logging and aggregation platform. It specializes in monitoring errors and extracting all the information needed to do a proper post-mortem without any of the hassle of the standard user feedback loop.

Flume - Log management using HDFS

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

Fluentd - Data collector, Log Everything in JSON

Fluentd is an event collector system. It is a generalized version of syslogd, which handles JSON objects for its log messages. It collects logs from various data sources and writes them to files, database or other types of storages.

Phabricator - Continuous Release Management from Facebook

Phabricator is an open source collection of web applications which make it easier to write, review, and share source code. It has Wiki and CLI support. Phabricator was developed at Facebook.

Live Graph - Plot and explore your data in real-time

LiveGraph is a framework for real-time data visualisation, analysis and logging. It has a real time plotter that can automatically update graphs of your data while it is still being computed by your application. LiveGraph reads files in a simple CSV-style format. For applications developed in Java, LiveGraph additionally provides an API that handles all data logging and persistency issues.

Socialauth - OAuth library in Java

Socialauth provides support to authenticate users through external oAuth providers like Gmail, Hotmail, Yahoo, Twitter, Facebook, LinkedIn, Foursquare, MySpace, Salesforce, Yammer as well as through OpenID providers like It could be easily integrated.


JForum is a powerful and robust discussion board system implemented in Java. It provides an attractive interface, an efficient forum engine, an easy to use administrative panel, an advanced permission control system and much more.

Pidgin - The Universal Chat Client

Pidgin is an instant messaging program which lets you log in to accounts on multiple chat networks simultaneously. It runs on Windows, Linux, and other UNIX operating systems. Pidgin is compatible with the following chat networks out of the box: AIM, ICQ, Google Talk, Jabber/XMPP, MSN Messenger, Yahoo!, Bonjour, Gadu-Gadu, IRC, MXit, Novell GroupWise Messenger, Lotus Sametime, SILC, SIMPLE, MySpaceIM, and Zephyr.

Pligg - Social Publishing CMS

Pligg is an open source CMS (Content Management System) which provides social publishing software that encourages visitors to register on your website so that they can submit content and connect with other users. Our software creates websites where stories are created and voted on by members, not website editors. It is a user driven CMS that relies on independent authors content and participation to manage news articles.