mtail - extract whitebox monitoring data from application logs for collection in a timeseries database

  •        282

mtail is a tool for extracting metrics from application logs to be exported into a timeseries database or timeseries calculator for alerting and dashboarding.It aims to fill a niche between applications that do not export their own internal state, and existing monitoring systems, without patching those applications or rewriting the same framework for custom extraction glue code.

https://github.com/google/mtail

Tags
Implementation
License
Platform

   




Related Projects

Skywalking - APM, Application Performance Monitoring System

  •    Java

SkyWalking is an APM (application performance monitor) system, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures. SkyWalking is an open source APM system, including monitoring, tracing, diagnosing capabilities for distributed system in Cloud Native architecture. It supports to collect telemetry (metrics, traces, and logs) data from multiple sources and multiple formats.

OpenTelemetry makes robust, portable telemetry a built-in feature of cloud-native software

  •    Java

OpenTelemetry provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application. You can analyze them using Prometheus, Jaeger, and other observability tools.

Signoz - Open-source Observability platform and an alternative to DataDog, NewRelic

  •    Javascript

SigNoz is an opensource observability platform. SigNoz uses distributed tracing to gain visibility into your systems and powers data using Kafka (to handle high ingestion rate and backpressure) and Apache Druid (Apache Druid is a high performance real-time analytics database), both proven in the industry to handle scale.

sloth - 🦥 Easy and simple Prometheus SLO (service level objectives) generator

  •    Go

Meet the easiest way to generate SLOs for Prometheus. Sloth generates understandable, uniform and reliable Prometheus SLOs for any kind of service. Using a simple SLO spec that results in multiple metrics and multi window multi burn alerts.

chiamon - Example Chia monitoring stack

  •    Dockerfile

This includes a docker-compose configuration to run everything, but this is primarily intended for development and testing. WARNING this is NOT a one-click install, expect to need to do some work setting everything up for your machine. PLEASE read the notes below and understand what all the services are, what they do, and how they work together.


opentelemetry-rust - OpenTelemetry API and SDK for Rust

  •    Rust

The Rust OpenTelemetry implementation. OpenTelemetry is a collection of tools, APIs, and SDKs used to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis in order to understand your software's performance and behavior. You can export and analyze them using Prometheus, Jaeger, and other observability tools.

agent - Prometheus Metrics, Loki Logs, and Tempo Traces, optimized for Grafana Cloud.

  •    Go

On top of these, the Grafana Agent enables easier sharding mechanisms that enable users to shard Agents across their cluster and lower the memory requirements per machine. A typical deployment of the Grafana Agent for Prometheus metrics can see up to a 40% reduction in memory usage with equal scrape loads.

vector - A high-performance, highly reliable, observability data pipeline.

  •    Rust

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space. To get started, follow our quickstart guide or install Vector.

vector - A high-performance observability data pipeline.

  •    Rust

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where it is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space. To get started, follow our quickstart guide or install Vector.

netdata - Real-time performance monitoring, done right! https://www.netdata.cloud

  •    C

Netdata's distributed, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices, and is perfectly safe to install on your systems mid-incident without any preparation. You can install Netdata on most Linux distributions (Ubuntu, Debian, CentOS, and more), container platforms (Kubernetes clusters, Docker), and many other operating systems (FreeBSD, macOS). No sudo required.

Cortex - A multitenant, horizontally scalable Prometheus as a Service

  •    Go

Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for Prometheus. Cortex can run across multiple machines in a cluster, exceeding the throughput and storage of a single machine. This enables you to send the metrics from multiple Prometheus servers to a single Cortex cluster and run "globally aggregated" queries across all data in a single place. Cortex makes your PromQL queries blazin' fast through aggressive parallelization and caching.

go-grpc-prometheus - Prometheus monitoring for your gRPC Go servers.

  •    Go

Prometheus monitoring for your gRPC Go servers and clients. A sister implementation for gRPC Java (same metrics, same semantics) is in grpc-ecosystem/java-grpc-prometheus.

Apache APISIX - Dynamic, real-time, high-performance API gateway

  •    Lua

Apache APISIX is a dynamic, real-time, high-performance API gateway. APISIX provides rich traffic management features such as load balancing, dynamic upstream, canary release, circuit breaking, authentication, observability, and more. You can use Apache APISIX to handle traditional north-south traffic, as well as east-west traffic between services. It can also be used as a k8s ingress controller.

Loki - Multi-tenant log aggregation system inspired by Prometheus

  •    Go

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. It is especially good fit for storing Kubernetes Pod logs. Metadata such as Pod labels is automatically scraped and indexed. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.

GCJ

  •    C

GCJ is a portable, optimizing, ahead-of-time compiler for the Java Programming Language. It can compile Java source code to Java bytecode (class files) or directly to native machine code, and Java bytecode to native machine code.

cloud-ops-sandbox - Cloud Operations Sandbox is an open source tool that helps practitioners to learn Service Reliability Engineering practices from Google and apply them on their cloud services using Cloud Operations suite of tools

  •    HTML

Cloud Operations Sandbox is an open-source tool that helps practitioners to learn Service Reliability Engineering practices from Google and apply them on their cloud services using Cloud Operations (formerly Stackdriver). It is based on Hipster Shop, a cloud-native microservices application. Google Cloud Operations Suite is a suite of tools that helps you gain full observability of your code and applications. You might want to take Cloud Operations to a "test drive" in order to answer the question, "will it work for my application needs"? The most effective way to learn is by testing the tool in "real-life" conditions, but without risking a production system. With Sandbox, we provide a tool that automatically provisions a new demo cluster, which receives traffic, simulating real users. Practitioners can experiment with various Cloud Operations tools to solve problems and accomplish standard SRE tasks in a sandboxed environment.

JPHP - An alternative to Zend PHP, like JRuby and Jython only for PHP

  •    Java

JPHP is a new implementation for PHP which uses the Java VM. It supports many features of the PHP language (7.0+).How does it work? JPHP is a compiler like javac, it compiles php sources to JVM bytecode and then can execute the result on the Java VM. JPHP is not a replacement for the Zend PHP engine or Facebook HHVM.

jphp - An alternative to Zend PHP, like JRuby and Jython only for PHP

  •    Java

JPHP is a new implementation for PHP which uses the Java VM. It supports many features of the PHP language (7.1+). How does it work? JPHP is a compiler like javac, it compiles php sources to JVM bytecode and then can execute the result on the Java VM.

Chaos Genius - ML powered analytics engine for outlier detection and root cause analysis

  •    Python

Chaos Genius is an open source ML powered analytics engine for outlier detection and root cause analysis. Chaos Genius can be used to monitor and analyse high dimensionality business, data and system metrics at scale. Using Chaos Genius, users can segment large datasets by key performance metrics (e.g. Daily Active Users, Cloud Costs, Failure Rates) and important dimensions (e.g., countryID, DeviceID, ProductID, DayofWeek) across which they want to monitor and analyse the key metrics.

SCOUTER - APM (Application Performance Management) tool.

  •    Java

SCOUTER is an open source APM like new relic and appdynamics. (APM means Application Performance Monitoring or Application Performance Management.) SCOUTER shows metrics about Users (Active user, Recently used user, Today visitor), Services ( Active service, TPS, Response time, Application profiles(method profile, sql profile, external call profile etc) and Resources (Cpu, Memory, Network and Heap usage, Connection pools etc).






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.