Logstash - Centralized log storage, indexing, and searching

  •        0

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use. Logstash comes with a web interface for searching and drilling into all of your logs.

You can ship logs from any source, parse them, get the right timestamp, index them, and search them. You can query for more precise things like, 404 http errors, nagios critical alerts in hard state, or mail server faults - all without accidentally finding logs with the word '404' or 'critical' in the wrong place.

Logstash could scale horizontally. It runs as standalone service or in client-server mode. It also understands AMQP protocol and recommends RabbitMQ as message broker.




comments powered by Disqus

Related Projects

logstash - logstash - logs/event transport, processing, management, search.

logstash - logs/event transport, processing, management, search.

Sentry - Realtime Platform-Agnostic Error Logging and Aggregation platform

Sentry is a realtime event logging and aggregation platform. It specializes in monitoring errors and extracting all the information needed to do a proper post-mortem without any of the hassle of the standard user feedback loop.

Graylog2 - Open Source Log Management

Graylog2 is an open source log management solution that stores your logs in ElasticSearch. It consists of a server written in Java that accepts your syslog messages via TCP, UDP or AMQP and stores it in the database. The second part is a web interface that allows you to manage the log messages from your web browser. Take a look at the screenshots or the latest release info page to get a feeling of what you can do with Graylog2.

Kafka - A high-throughput distributed messaging system

Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.

Fluentd - Data collector, Log Everything in JSON

Fluentd is an event collector system. It is a generalized version of syslogd, which handles JSON objects for its log messages. It collects logs from various data sources and writes them to files, database or other types of storages.

Scribe - Real time log aggregation used in Facebook

Scribe is a server for aggregating log data that's streamed in real time from clients. It is designed to be scalable and reliable. It is developed and maintained by Facebook. It is designed to scale to a very large number of nodes and be robust to network and node failures. There is a scribe server running on every node in the system, configured to aggregate messages and send them to a central scribe server (or servers) in larger groups.

Rainbow - portal development made easy

Rainbow CMS available today in 29 languages, allows content authoring to be safely delegated to role-based team members who need little or no knowledge of HTML. Rainbow optionally supports a two-step approval-publish process. 75 plug-in modules are now included in the standard release. It is also fairly easy to build your own custom modules.

nxlog - Multi platform Log management

nxlog is a modular, multi-threaded, high-performance log management solution with multi-platform support. In concept it is similar to syslog-ng or rsyslog but is not limited to unix/syslog only. It can collect logs from files in various formats, receive logs from the network remotely over UDP, TCP or TLS/SSL . It supports platform specific sources such as the Windows Eventlog, Linux kernel logs, Android logs, local syslog etc.

Live Graph - Plot and explore your data in real-time

LiveGraph is a framework for real-time data visualisation, analysis and logging. It has a real time plotter that can automatically update graphs of your data while it is still being computed by your application. LiveGraph reads files in a simple CSV-style format. For applications developed in Java, LiveGraph additionally provides an API that handles all data logging and persistency issues.

Flume - Log management using HDFS

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.