Scribe - Real time log aggregation used in Facebook

  •        11865

Scribe is a server for aggregating log data that's streamed in real time from clients. It is designed to be scalable and reliable. It is developed and maintained by Facebook. It is designed to scale to a very large number of nodes and be robust to network and node failures. There is a scribe server running on every node in the system, configured to aggregate messages and send them to a central scribe server (or servers) in larger groups.

https://github.com/facebook/scribe

Tags
Implementation
License
Platform

   




Related Projects

Kafka - A high-throughput distributed messaging system


Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.

X-Itools: Enterprise Collaboration


Enterprise Collaboration modules and strong Log Analysis modules

Luxun - A high-throughput, persistent, distributed, publish-subscribe messaging system based on memo


A high-throughput, persistent, distributed, publish-subscribe messaging system based on memory mapped file and Thrift RPC.

Sentry - Realtime Platform-Agnostic Error Logging and Aggregation platform


Sentry is a realtime event logging and aggregation platform. It specializes in monitoring errors and extracting all the information needed to do a proper post-mortem without any of the hassle of the standard user feedback loop.



py_shipper - Multithreaded publish-subscribe based easily extensible python log shipper


Multithreaded publish-subscribe based easily extensible python log shipper

jafka - a fast distributed publish-subscribe messaging system (mq)


a fast distributed publish-subscribe messaging system (mq)

grind - A log aggregation and analysis UNIX tool.


A log aggregation and analysis UNIX tool.

DAD - Log aggregation, analysis, alerting and correlation for Windows, Syslog and text based logs.


Log aggregation, analysis, alerting and correlation for Windows, Syslog and text based logs.

Kiln - Kiln is a platform for log aggregation, visualization and analysis built in Ruby on Rails


Kiln is a platform for log aggregation, visualization and analysis built in Ruby on Rails

secondsight - Log aggregation and analysis framework for multithreaded/multiprocess HPC application


Log aggregation and analysis framework for multithreaded/multiprocess HPC application

chililog-server - Real Time Log Aggregation, Analysis and Monitoring


Real Time Log Aggregation, Analysis and Monitoring

White-elephant - Hadoop log aggregator and dashboard


White Elephant is a Hadoop log aggregator and dashboard which enables visualization of Hadoop cluster utilization across users. The server is a JRuby web application. In a production environment it can be deployed to tomcat and reads aggregated usage data directly from Hadoop. This data is stored in an in-memory database provided by HyperSQL. Charting is provided by Rickshaw. This project is developed by LinkedIn.

couchdb-lager - Mirror of Apache CouchDB


for the backend:```erlang{lager, [ {handlers, [ {lager_console_backend, [info, {lager_default_formatter, [time," [",severity,"] ", message, "\n"]}]}, {lager_file_backend, [{file, "error.log"}, {level, error}, {formatter, lager_default_formatter}, {formatter_config, [date, " ", time," [",severity,"] ",pid, " ", message, "\n"]}]}, {lager_file_backend, [{file, "console.log"}, {level, info}]} ]}]}.```Included is lager_default_formatter. This provides a generic, default formatting fo

Log-Dispatch-Scribe - Release history of Log-Dispatch-Scribe


Release history of Log-Dispatch-Scribe

mango - Global web log collection, aggregation, and anonymization ETL system


Global web log collection, aggregation, and anonymization ETL system

gozbus - nanocap based messaging system


Gozbus is a messaging system based on the nanocap transport, our term for a combination of the nanomsg[1] and Cap'n Proto[2] technologies.Nanomsg is the successor to ZeroMQ, and provides multiple patterns for in-process, in-host, and inter-host messaging. The peer-to-peer, publish-subscribe, enterprise bus, pipeline, and surveyor protcols can be leveraged for scalable messaging.

Scribe-with-HDFS-support


This is a customized version of Scribe to support writing log files to HDFS. Scribe is a server for aggregating log data streamed in real time from a large number of servers. It is designed to be scalable, extensible without client-side modification, and robust to failure of the network or any specific machine.

Scribe-log4j-Appender - Generic log4j appender that uses Scribe for sending log messages


Generic log4j appender that uses Scribe for sending log messages