Awstats - Advanced web, streaming, ftp and mail server statistics

•    CGI

---

AWStats is a powerful tool that generates advanced web, streaming, ftp or mail server statistics graphically. It can analyze log files from all major server tools like Apache log files, WebStar, IIS and a lot of other web, proxy, wap, streaming servers, mail servers and some ftp servers. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages.

web-analytics web-traffic google-analytics-alternative log log-analysis

Webalizer - fast web server log file analysis

•    C

---

The Webalizer is a fast web server log file analysis program. It produces highly detailed, easily configurable usage reports in HTML format, for viewing with a standard web browser. It handles standard Common logfile format (CLF) server logs, several variations of the NCSA Combined logfile format, wu-ftpd/proftpd xferlog (FTP) format logs, Squid proxy server native format, and W3C Extended log formats.

web-analytics web-traffic google-analytics-alternative log log-analysis

Indihiang - IIS and Apache log analyzing tool

•    CSharp

---

Indihiang Project is a web log analyzing tool. This tool analyzes IIS and Apache Web logs and generates real time reports. It has Web Log Viewer and analyzer. It is capable to analyze the trend from the logs. This tool also integrate with windows Explorer so you can attach a log file in to indihiang tool via context menu.

web-analytics web-traffic google-analytics-alternative log log-analysis

Apache Metron - Real-time Big Data Security

•    Java

---

Metron integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis. Metron provides capabilities for log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment, while applying the most current threat intelligence information to security telemetry within a single platform.

security-framework cyber-crime anomoly-detection monitoring security big-data threat-intelligence security-analytics opensoc siem threat-analytics

White-elephant - Hadoop log aggregator and dashboard

•    Java

---

White Elephant is a Hadoop log aggregator and dashboard which enables visualization of Hadoop cluster utilization across users. The server is a JRuby web application. In a production environment it can be deployed to tomcat and reads aggregated usage data directly from Hadoop. This data is stored in an in-memory database provided by HyperSQL. Charting is provided by Rickshaw. This project is developed by LinkedIn.

log-analysis log-aggregation hadoop-logs visualization log-management

X-Itools: Enterprise Collaboration

•    Javascript

---

Enterprise Collaboration modules and strong Log Analysis modules

Kong - The Microservice API Gateway

•    Lua

---

Kong is a cloud-native, fast, scalable, and distributed Microservice Abstraction Layer (also known as an API Gateway, API Middleware or in some cases Service Mesh). Backed by the battle-tested NGINX with a focus on high performance, Kong was made available as an open-source platform in 2015. Under active development, Kong is used in production at thousands of organizations from startups, Global 5000 and Government organizations.

api-gateway nginx luajit microservices api-management serverless apis iot consul docker reverse-proxy service-mesh cloud-native microservice devops-tools devops

opensoc - OpenSOC Apache Hadoop Code

•    

---

OpenSOC integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis. OpenSOC provides capabilities for log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment, while applying the most current threat intelligence information to security telemetry within a single platform. A mechanism to capture, store, and normalize any type of security telemetry at extremely high rates. Because security telemetry is constantly being generated, it requires a method for ingesting the data at high speeds and pushing it to various processing units for advanced computation and analytics.

Graylog2 - Open Source Log Management

•    Java

---

Graylog2 is an open source log management solution that stores your logs in ElasticSearch. It consists of a server written in Java that accepts your syslog messages via TCP, UDP or AMQP and stores it in the database. The second part is a web interface that allows you to manage the log messages from your web browser. Take a look at the screenshots or the latest release info page to get a feeling of what you can do with Graylog2.

log log-analysis log-search log-management syslog

angle-grinder - Slice and dice log files on the command line

•    Rust

---

Slice and dice log files on the command line. Angle-grinder allows you to parse, aggregate, sum, average, percentile, and sort your data. You can see it, live-updating, in your terminal. Angle grinder is designed for when, for whatever reason, you don't have your data in graphite/honeycomb/kibana/sumologic/splunk/etc. but still want to be able to do sophisticated analytics.

logging analytics cli-app

Zenoss - Open Source IT Management

•    Python

---

Zenoss Core is an open source IT monitoring product that delivers the functionality to effectively manage the configuration, health, performance of networks, servers and applications through a single, integrated software package.

monitoring sys-admin-tools tools system-monitoring log-analysis

Clarity - Web interface for the grep

•    Ruby

---

Clarity is a Splunk like web interface for your server log files. It supports searching (using grep) as well as trailing log files in realtime. It has been written using the event based architecture based on EventMachine and so allows real-time search of very large log files.

log log-analysis log-management log-search

Kafka - A high-throughput distributed messaging system

•    Java

---

Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.

log-analysis log-aggregation messaging-system publish-subscribe messaging log-management

Flume - Log management using HDFS

•    Java

---

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

log-collector syslog log-analysis log log-management distributed

OSSEC - Host-based Intrusion Detection System

•    C

---

OSSEC is a full platform to monitor and control your systems. It mixes together all the aspects of HIDS (host-based intrusion detection), log monitoring and SIM/SIEM together in a simple, powerful and open source solution.

intrusion-detection siem threat-intelligence security-analytics threat-analytics monitoring

Wazuh - Host and endpoint security

•    C

---

Wazuh helps you to gain deeper security visibility into your infrastructure by monitoring hosts at an operating system and application level. This solution, based on lightweight multi-platform agents, provides the capabilities like Log management and analysis, File integrity monitoring, Intrusion and anomaly detection, Policy and compliance monitoring.

ossec security loganalyzer compliance monitoring intrusion-detection policy-monitoring openscap security-hardening ids pci-dss file-integrity-management log-analysis vulnerability-detection incident-response threat-detection

Sentry - Realtime Platform-Agnostic Error Logging and Aggregation platform

•    Python

---

Sentry is a realtime event logging and aggregation platform. It specializes in monitoring errors and extracting all the information needed to do a proper post-mortem without any of the hassle of the standard user feedback loop.

log log-analysis log-search log-management aggregation event-logging

Fluentd - Data collector, Log Everything in JSON

•    Ruby

---

Fluentd is an event collector system. It is a generalized version of syslogd, which handles JSON objects for its log messages. It collects logs from various data sources and writes them to files, database or other types of storages.

log-collector syslog log-analysis log log-management

Open Web Analytics - Analytics used by Wikimedia

•    PHP

---

Open Web Analytics is open source web analytics software that you can use to track and analyze how people use your web sites and applications. OWA provides web site owners and developers with easy ways to add web analytics to their sites using simple Javascript, PHP, or REST based APIs. OWA also comes with built-in support for tracking web sites made with popular content management frameworks such as WordPress and MediaWiki.

web-analytics web-traffic google-analytics-alternative

hadoop-ansible - Ansible playbook that installs a Hadoop cluster, with HBase, Hive, Presto for analytics, and Ganglia, Smokeping, Fluentd, Elasticsearch and Kibana for monitoring and centralized log indexing

•    Shell

---

Ansible playbook that installs a CDH 4.6.0 Hadoop cluster (running on Java 7, supported from CDH 4.4), with HBase, Hive, Presto for analytics, and Ganglia, Smokeping, Fluentd, Elasticsearch and Kibana for monitoring and centralized log indexing. Follow @analytically. Browse the CI build screenshots.