Displaying 1 to 9 from 9 results

Scribe - Real time log aggregation used in Facebook

  •    C++

Scribe is a server for aggregating log data that's streamed in real time from clients. It is designed to be scalable and reliable. It is developed and maintained by Facebook. It is designed to scale to a very large number of nodes and be robust to network and node failures. There is a scribe server running on every node in the system, configured to aggregate messages and send them to a central scribe server (or servers) in larger groups.

OpenSearch - Open source distributed and RESTful search engine

  •    Java

OpenSearch is a community-driven, open source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2. It consists of a search engine daemon, OpenSearch, and a visualization and user interface, OpenSearch Dashboards. OpenSearch enables people to easily ingest, secure, search, aggregate, view, and analyze data. These capabilities are popular for use cases such as application search, log analytics, and more.

Loki - Multi-tenant log aggregation system inspired by Prometheus

  •    Go

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. It is especially good fit for storing Kubernetes Pod logs. Metadata such as Pod labels is automatically scraped and indexed. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.

Kafka - A high-throughput distributed messaging system

  •    Java

Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.




Tremor - An early-stage event processing system for unstructured data with rich support for structural pattern-matching, filtering and transformation

  •    Rust

Tremor is an event processing system. It was originally designed as a replacement for software such as Logstash or Telegraf. However tremor has outgrown this singular use case by supporting more complex workflows such as aggregation, rollups, an ETL language, and a query language.

White-elephant - Hadoop log aggregator and dashboard

  •    Java

White Elephant is a Hadoop log aggregator and dashboard which enables visualization of Hadoop cluster utilization across users. The server is a JRuby web application. In a production environment it can be deployed to tomcat and reads aggregated usage data directly from Hadoop. This data is stored in an in-memory database provided by HyperSQL. Charting is provided by Rickshaw. This project is developed by LinkedIn.

Hypertrace - Distributed tracing & Observability platform

  •    Java

Hypertrace is a cloud-native distributed tracing based Observability platform that gives visibility into your dev and production distributed systems. Hypertrace provides observability into your application architecture. It includes global, service and backend dashboards, allowing teams fast insight into service level objectives.

jaggr - JSON Aggregation CLI

  •    Go

Jaggr is a command line tool to aggregate in real time a series of JSON logs. The main goal of this tool is to prepare data for plotting with jplot.


harvest - :beetle: Portable log aggregation tool for middle-scale system operation/troubleshooting.

  •    Go

Portable log aggregation tool for middle-scale system operation/troubleshooting. Harvest provides the hrv command with the following features.