Tremor - An early-stage event processing system for unstructured data with rich support for structural pattern-matching, filtering and transformation

  •        192

Tremor is an event processing system. It was originally designed as a replacement for software such as Logstash or Telegraf. However tremor has outgrown this singular use case by supporting more complex workflows such as aggregation, rollups, an ETL language, and a query language.

Tremor has been successfully used to replace logstash as a Kafka to Elastic Search ingress. In this scenario, it reduced the required compute resources by about 80% (YMMV) when decoding, classify, and rate-limiting the traffic. A secondary but perhaps more important effect was that tremors dynamic backpressure and rate-limiting allowed the ElasticSearch system to stay healthy and current despite overwhelming amounts of logs during spikes.

https://www.tremor.rs
https://github.com/tremor-rs/tremor-runtime

Tags
Implementation
License
Platform

   




Related Projects

Loki - Multi-tenant log aggregation system inspired by Prometheus

  •    Go

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. It is especially good fit for storing Kubernetes Pod logs. Metadata such as Pod labels is automatically scraped and indexed. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.

Kafka - A high-throughput distributed messaging system

  •    Java

Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.

Invoke-Phant0m - Windows Event Log Killer

  •    PowerShell

This script walks thread stacks of Event Log Service process (spesific svchost.exe) and identify Event Log Threads to kill Event Log Service Threads. So the system will not be able to collect logs and at the same time the Event Log Service will appear to be running. I have made this script for two reasons. First, This script will help to Red Teams and Penetration Testers. Second, I want to learn Powershell and Low-Level things on Powershell for cyber security field.

Fluent Bit - Fast and Lightweight Logs and Metrics processor

  •    C

Fluent Bit is a fast Log Processor and Forwarder, it allows to collect log events or metrics from different sources, process them and deliver them to different backends such as Fluentd, Elasticsearch, Splunk, DataDog, Kafka, New Relic, Azure services, AWS services, Google services, NATS, InfluxDB or any custom HTTP end-point. It also comes with full SQL Stream Processing capabilities: data manipulation and analytics using SQL queries.

OpenSearch - Open source distributed and RESTful search engine

  •    Java

OpenSearch is a community-driven, open source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2. It consists of a search engine daemon, OpenSearch, and a visualization and user interface, OpenSearch Dashboards. OpenSearch enables people to easily ingest, secure, search, aggregate, view, and analyze data. These capabilities are popular for use cases such as application search, log analytics, and more.


Sentry - Realtime Platform-Agnostic Error Logging and Aggregation platform

  •    Python

Sentry is a realtime event logging and aggregation platform. It specializes in monitoring errors and extracting all the information needed to do a proper post-mortem without any of the hassle of the standard user feedback loop.

Sawbuck - Log viewer and controller for Windows Chrome logging

  •    C++

Sawbuck is a log viewer and controller for Windows Chrome logging, and for other applications or plugins that use the logging facility in Chrome base. Logging in Chrome is integrated with Event Tracing for Windows (ETW), which allows ETW controllers like Sawbuck to control log verbosity at runtime. The Chrome logging integration also captures the call stack at the logging site, which can then be resolved and displayed by log viewers such as Sawbuck.

Fluentd - Unified Logging Layer

  •    Ruby

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure. Fluentd can be used to tail access/error logs and transport them reliably to remote systems. It can "grep" for events and send out alerts. It can function as middleware to enable asynchronous, scalable logging for user action events.

LogonTracer - Investigate malicious Windows logon by visualizing and analyzing Windows event log

  •    Python

LogonTracer is a tool to investigate malicious logon by visualizing and analyzing Windows Active Directory event logs. This tool associates a host name (or an IP address) and account name found in logon-related events and displays it as a graph. This way, it is possible to see in which account login attempt occurs and which host is used. This tool can visualize the following event id related to Windows logon based on this research. LogonTracer uses PageRank, Hidden Markov model and ChangeFinder to detect malicious hosts and accounts from event log. With LogonTracer, it is also possible to display event logs in a chronological order.

Meniscus - The Python Event Logging Service

  •    Python

Meniscus is a Python based system for event collection, transit and processing in the large. It's primary use case is for large-scale Cloud logging, but can be used in many other scenarios including usage reporting and API tracing. Its components include Collection, Transport, Storage, Event Processing & Enhancement, Complex Event Processing, Analytics.

oink - Log parser to identify actions which significantly increase VM heap size

  •    Ruby

Rails plugin and log parser to help narrow down the source(s) of increased memory usage in rails applications. Oink adds memory and active record instantiation information to rails log during runtime and provides an executable to help digest the enhanced logs.

LogTrail - Log Viewer plugin for Kibana

  •    Javascript

LogTrail is a plugin for Kibana to view, analyze, search and tail log events from multiple hosts in realtime with devops friendly interface inspired by Papertrail.

Event Log Analyzer

  •    WPF

Event Log Analyzer is a simple yet powerful tool to analyze event logs in Windows. It has features to group similar events together, give graphical view of distribution of events over time and in similarity. It is developed in C# and WPF and uses MVVM framework.

alb-ingress-controller - Kubernetes Ingress Controller for AWS ALB

  •    Go

NOTE: This controller is in alpha state as we attempt to move to our first 1.0 release. The current image version is 1.0-alpha.3. Please file any issues you find and note the version used.The ALB Ingress Controller satisfies Kubernetes ingress resources by provisioning Application Load Balancers.

aws-alb-ingress-controller - AWS ALB Ingress Controller for Kubernetes

  •    Go

NOTE: This controller is in beta state as we attempt to move to our first 1.0 release. The current image version is 1.0-beta.7. Please file any issues you find and note the version used. The AWS ALB Ingress Controller satisfies Kubernetes ingress resources by provisioning Application Load Balancers.

Webalizer - fast web server log file analysis

  •    C

The Webalizer is a fast web server log file analysis program. It produces highly detailed, easily configurable usage reports in HTML format, for viewing with a standard web browser. It handles standard Common logfile format (CLF) server logs, several variations of the NCSA Combined logfile format, wu-ftpd/proftpd xferlog (FTP) format logs, Squid proxy server native format, and W3C Extended log formats.

voyager - ✈️️ Secure Ingress Controller for Kubernetes

  •    Go

Voyager is a HAProxy backed secure L7 and L4 ingress controller for Kubernetes developed by AppsCode. This can be used with any Kubernetes cloud providers including aws, gce, gke, azure, acs. This can also be used with bare metal Kubernetes clusters.Voyager provides L7 and L4 loadbalancing using a custom Kubernetes Ingress resource. This is built on top of the HAProxy to support high availability, sticky sessions, name and path-based virtual hosting. This also support configurable application ports with all the options available in a standard Kubernetes Ingress. Here is a complex ingress example that shows how various features can be used. You can find the generated HAProxy Configuration here.

kubernetes-ingress - NGINX and NGINX Plus Ingress Controllers for Kubernetes

  •    Go

This repo provides an implementation of an Ingress controller for NGINX and NGINX Plus. This implementation is different from the NGINX Ingress controller in kubernetes/ingress-nginx repo. See this doc to find out about the key differences. See the Ingress User Guide to learn more.

contour - Contour is a Kubernetes ingress controller for Lyft's Envoy proxy.

  •    Go

Contour is an Ingress controller for Kubernetes that works by deploying the Envoy proxy as a reverse proxy and load balancer. Unlike other Ingress controllers, Contour supports dynamic configuration updates out of the box while maintaining a lightweight profile. Contour also introduces a new ingress API (IngressRoute) which is implemented via a Custom Resource Definition (CRD). Its goal is to expand upon the functionality of the Ingress API to allow for a richer user experience as well as solve shortcomings in the original design.

Windows Event Log Manager

  •    

Windows Event Log Manager is a tool for creating, editing, and viewing windows event logs.






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.