skyline - Anomaly detection suite

  •        223

Skyline is a near real time anomaly detection system, built to enable passive monitoring of hundreds of thousands of metrics, without the need to configure a model/thresholds for each one, as you might do with Nagios. It is designed to be used wherever there are a large quantity of high-resolution time series which need constant monitoring. Once a metrics stream is set up (from StatsD or Graphite or another source), additional metrics are automatically added to Skyline for analysis. Skyline's easily extended algorithms attempt to automatically detect what it means for each metric to be anomalous. The documentation for your version is also viewable in a clone locally in your browser at file:///docs/_build/html/index.html and via the the Skyline Webapp frontend.

http://earthgecko-skyline.readthedocs.io/en/latest/
https://github.com/earthgecko/skyline

Tags
Implementation
License
Platform

   




Related Projects

NAB - The Numenta Anomaly Benchmark

  •    Python

Welcome. This repository contains the data and scripts comprising the Numenta Anomaly Benchmark (NAB). NAB is a novel benchmark for evaluating algorithms for anomaly detection in streaming, real-time applications. It is comprised of over 50 labeled real-world and artificial timeseries data files plus a novel scoring mechanism designed for real-time applications. Included are the tools to allow you to easily run NAB on your own anomaly detection algorithms; see the NAB entry points info. Competitive results tied to open source code will be posted in the wiki on the Scoreboard. Let us know about your work by emailing us at nab@numenta.org or submitting a pull request.

Pyod - A Python Toolkit for Scalable Outlier Detection (Anomaly Detection)

  •    Python

Important Notes: PyOD contains some neural network based models, e.g., AutoEncoders, which are implemented in keras. However, PyOD would NOT install keras and tensorflow automatically. This would reduce the risk of damaging your local installations. You are responsible for installing keras and tensorflow if you want to use neural net based models. An instruction is provided here. Anomaly detection resources, e.g., courses, books, papers and videos.

Wazuh - Host and endpoint security

  •    C

Wazuh helps you to gain deeper security visibility into your infrastructure by monitoring hosts at an operating system and application level. This solution, based on lightweight multi-platform agents, provides the capabilities like Log management and analysis, File integrity monitoring, Intrusion and anomaly detection, Policy and compliance monitoring.

AnomalyDetection - Anomaly Detection with R

  •    R

AnomalyDetection is an open-source R package to detect anomalies which is robust, from a statistical standpoint, in the presence of seasonality and an underlying trend. The AnomalyDetection package can be used in wide variety of contexts. For example, detecting anomalies in system metrics after a new software release, user engagement post an A/B test, or for problems in econometrics, financial engineering, political and social sciences.

morgoth - Metric anomaly detection

  •    Go

Morgoth provides a framework for implementing the smaller pieces of an anomaly detection problem. The basic framework is that Morgoth maintains a dictionary of normal behaviors and compares new windows of data to the normal dictionary. If the new window of data is not found in the dictionary then it is considered anomalous.


Apache Metron - Real-time Big Data Security

  •    Java

Metron integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis. Metron provides capabilities for log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment, while applying the most current threat intelligence information to security telemetry within a single platform.

wdbgark - WinDBG Anti-RootKit Extension

  •    C++

WDBGARK is an extension (dynamic library) for the Microsoft Debugging Tools for Windows. It main purpose is to view and analyze anomalies in Windows kernel using kernel debugger. It is possible to view various system callbacks, system tables, object types and so on. For more user-friendly view extension uses DML. For the most of commands kernel-mode connection is required. Feel free to use extension with live kernel-mode debugging or with kernel-mode crash dump analysis (some commands will not work). Public symbols are required, so use them, force to reload them, ignore checksum problems, prepare them before analysis and you'll be happy. Windows BETA/RC is supported by design, but read a few notes. First, i don't care about checked builds. Second, i don't care if you don't have symbols (public or private). IA64/ARM is unsupported (and will not).

vSentinel

  •    C

vSentinel is a customizable 3D mapping of your network monitoring or security data for real-time or trend-based attack and anomaly detection and analysis.

ourmon

  •    C

Ourmon is a network monitoring and anomaly detection system and displays the data for multiple BPF expressions via RRDTOOL-based graphs. It also helps the user identify various kinds of network anomalies using various flow analysis tools and logging.

netcap - A framework for secure and scalable network traffic analysis

  •    Go

The Netcap (NETwork CAPture) framework efficiently converts a stream of network packets into highly accessible type-safe structured data that represent specific protocols or custom abstractions. These audit records can be stored on disk or exchanged over the network, and are well suited as a data source for machine learning algorithms. Since parsing of untrusted input can be dangerous and network data is potentially malicious, implementation was performed in a programming language that provides a garbage collected memory safe runtime. It was developed for a series of experiments in my bachelor thesis: Implementation and evaluation of secure and scalable anomaly-based network intrusion detection. Currently, the thesis serves as documentation until the wiki is ready, it is included at the root of this repository (file: mied18.pdf). Slides from my presentation at the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities are available on researchgate.

Marketstore - DataFrame Server for Financial Timeseries Data

  •    Go

MarketStore is a database server optimized for financial timeseries data. You can think of it as an extensible DataFrame service that is accessible from anywhere in your system, at higher scalability. It is designed from the ground up to address scalability issues around handling large amounts of financial market data used in algorithmic trading backtesting, charting, and analyzing price history with data spanning many years, including tick-level for the all US equities or the exploding crypto currencies space. If you are struggling with managing lots of HDF5 files, this is perfect solution to your problem.

EventQL - The database for large-scale event analytics

  •    C++

EventQL is a distributed, column-oriented database built for large-scale event collection and analytics. It runs super-fast SQL and MapReduce queries. Its features include Automatic partitioning, Columnar storage, Standard SQL support, Scales to petabytes, Timeseries and relational data, Fast range scans and lot more.

Deeplearning4J - Neural Net Platform in Java and Scala

  •    Java

Deeplearning4J is an open source, distributed neural net library written in Java and Scala. It integrates with Hadoop and Spark and runs on several backends that enable use of CPUs and GPUs. It provides versatile n-dimensional array class for Java and Scala.

devialog

  •    Perl

devialog is a behavior/anomaly-based syslog intrusion detection system which detects unknown attacks via anomalies in syslog. It can generate signatures for ease of management, act upon anomalies in a predefined fashion or perform as a standard log parser

Jubatus - Framework and Library for Distributed Online Machine Learning

  •    C++

Jubatus is a distributed processing framework and streaming machine learning library. Jubatus includes these functionalities: Online Machine Learning Library: Classification, Regression, Recommendation (Nearest Neighbor Search), Graph Mining, Anomaly Detection, Clustering, Feature Vector Converter (fv_converter): Data Preprocess and Feature Extraction, Framework for Distributed Online Machine Learning with Fault Tolerance.

egads - Extendible Generic Anomaly Detection System

  •    Java

EGADS system, consumes time series and outputs anomalies.

bell.js - No longer maintained. Use https://github.com/eleme/banshee instead please.

  •    Javascript

Bell.js is a real-time anomalies(outliers) detection system for periodic time series, built to be able to monitor a large quantity of metrics. It collects metrics form statsd, analyzes them with the 3-sigma, once enough anomalies were found in a short time it alerts us via sms/email etc.We eleme use it to monitor our website/rpc interfaces, including api called frequency, api response time(time cost per call) and exceptions count. Our services send these statistics to statsd, statsd aggregates them every 10 seconds and broadcasts the results to its backends including bell, bell analyzes current stats with history data, calculates the trending, and alerts us if the trending behaves anomalous.

nupic - Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex

  •    Python

The Numenta Platform for Intelligent Computing (NuPIC) is a machine intelligence platform that implements the HTM learning algorithms. HTM is a detailed computational theory of the neocortex. At the core of HTM are time-based continuous learning algorithms that store and recall spatial and temporal patterns. NuPIC is suited to a variety of problems, particularly anomaly detection and prediction of streaming data sources. For more information, see numenta.org or the NuPIC Forum. For usage guides, quick starts, and API documentation, see http://nupic.docs.numenta.org/.

macrobase - MacroBase: A Search Engine for Fast Data

  •    Java

MacroBase is an anomaly detection engine designed to prioritize human attention in large-scale datasets and data streams.