Akumuli - Time-series database

  •        265

Akumuli is a time-series database for modern hardware. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from Esperanto as "accumulate".

  • True column-oriented format (not PAX).
  • Based on novel LSM and B+tree hybrid datastructure with multiversion concurrency control (no concurrency bugs, parallel writes, optimized for SSD and NVMe).
  • Crash safety and recovery.
  • Fast aggregation without pre-configured rollups or materialized views.
  • Queries can be executed without decompressing the data.
  • Fast compression algorithm (dictionary + entropy) with small memory overhead (about 2.5 bytes per element on appropriate data).
  • Compressed in-memory storage for recent data.
  • Can be used as a server application or an embedded library.
  • Simple query language based on JSON and HTTP.
  • Fast range scans and joins, read speed doesn't depend on database cardinality.
  • Fast data ingestion over the network:
    • 4.5M data points per second on 8-core Intel Xeon E5-2670 v2 (m3.2xlarge EC2 instance).
    • 16.1M data points per second on 32-core Intel Xeon E5-2680 v2 (c3.8xlarge EC2 instance).
  • Query results are streamed to client using the chunked transfer encoding of the HTTP protocol.
  • Decompression algorithm and input parsers were fuzz-tested.
  • Grafana datasource plugin.
  • Fast and compact inverted index for series lookup.




Related Projects

Lindb - Distributed Time Series Database

  •    Go

LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability. LinDB takes a lot of best practice of TSDB and implements some optimizations based on the characteristics of time series data. Unlike writing a lot of Continuous-Query for InfluxDB, LinDB supports rollup in specific interval automatically after creating the database. Moreover, LinDB is extremely fast for parallel querying and computing of distributed time series data.

awesome-time-series-database - :clock7: A curated list of awesome time series databases, benchmarks and papers

  •    Javascript

OpenTSDB is a Classical time series database on top of HBase. Now support Cassandra and Bigtable. BTrDB (Berkeley Tree Database) is a High performance time series database designed to support high density data storage applications.

OpenTSDB - A scalable, distributed Time Series Database.

  •    Java

OpenTSDB is a distributed, scalable Time Series Database (TSDB) written on top of HBase. OpenTSDB was written to address a common need: store, index and serve metrics collected from computer systems (network gear, operating systems, applications) at a large scale, and make this data easily accessible and graphable.

ClickHouse - Columnar DBMS and Real Time Analytics

  •    C++

ClickHouse is an open source column-oriented database management system capable of real time generation of analytical data reports using SQL queries. It is Linearly Scalable, Blazing Fast, Highly Reliable, Fault Tolerant, Data compression, Real time query processing, Web analytics, Vectorized query execution, Local and distributed joins. It can process hundreds of millions to more than a billion rows and tens of gigabytes of data per single server per second.

Timescaledb - An open-source time-series database optimized for fast ingest and complex queries

  •    PLpgSQL

TimescaleDB is an open-source database designed to make SQL scalable for time-series data. It is engineered up from PostgreSQL, providing automatic partitioning across time and space (partitioning key), as well as full SQL support. TimescaleDB is packaged as a PostgreSQL extension and released under the Apache 2 open-source license.

Heroic - The Time Series Database

  •    Java

Heroic is a scalable time series database based on Bigtable, Cassandra, and Elasticsearch. It is an open-source monitoring system originally built at Spotify to address the problems that were facing with large scale gathering and near real-time analysis of metrics.

Atlas - In-memory dimensional time series database

  •    Scala

Atlas was developed by Netflix to manage dimensional time series data for near real-time operational insight. Atlas features in-memory data storage, allowing it to gather and report very large numbers of metrics, very quickly. Atlas captures operational intelligence. Whereas business intelligence is data gathered for analyzing trends over time, operational intelligence provides a picture of what is currently happening within a system.

InfluxDB - Distributed Time Series Database

  •    Go

InfluxDB is an open-source, distributed, time series database with no external dependencies. It's useful for recording metrics, events, and performing analytics. Everything in InfluxDB is a time series that you can perform standard functions on like min, max, sum, count, mean, median, percentiles, and more. Collect your data on any interval and compute rollups on the fly later.

DalmatinerDB - Fast distributed metrics database in Erlang

  •    Erlang

DalmatinerDB is a metric database written in pure Erlang. It takes advantage of some special properties of metrics to make some tradeoffs. Its goal is to make a store for metric data (time, value of a metric) that is fast, has a low overhead, and is easy to query and manage. DalmatinerDB allows for metric input in second or even sub-second precision. It will interpolate the missing values to the best of it’s abilities. This is usually acceptable for aggregated data.

Hawkular metrics - Time Series Metrics Engine based on Cassandra

  •    Java

Hawkular Metrics is a metrics collection, aggregation, visualization framework. Hawkular is a set of Open Source projects designed to be a generic solution for common monitoring problems. The Hawkular projects provide REST services that can be used for all kinds of monitoring needs. The aim is to provide a generic solution that can be used for common monitoring problems.

Gnocchi - Time series database

  •    Python

Gnocchi is an open-source |time series| database. The problem that Gnocchi solves is the storage and indexing of |time series| data and resources at a large scale. This is useful in modern cloud platforms which are not only huge but also are dynamic and potentially multi-tenant. Gnocchi takes all of that into account. Gnocchi has been designed to handle large amounts of aggregates being stored while being performant, scalable and fault-tolerant. While doing this, the goal was to be sure to not build any hard dependency on any complex storage system.

SiriDB - Highly-scalable, robust and super fast time series database

  •    C

SiriDB is a highly-scalable, robust and super fast time series database. Build from the ground up SiriDB uses a unique mechanism to operate without indexes and allows server resources to be added on the fly. SiriDB's unique query language includes dynamic grouping of time series for easy and super fast analysis over large amount's of time series.

ceres - Distributable time-series database (not actively maintained)

  •    Python

Ceres is not actively maintained.Ceres is a time-series database format intended to replace Whisper as the default storage format for Graphite. In contrast with Whisper, Ceres is not a fixed-size database and is designed to better support sparse data of arbitrary fixed-size resolutions. This allows Graphite to distribute individual time-series across multiple servers or mounts.

InfluxDB - Scalable datastore for metrics, events, and real-time analytics

  •    Go

InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or monitoring and alerting purposes, user dashboards, and visualizing and exploring the data and more. The master branch on this repo now represents the latest InfluxDB, which now includes functionality for Kapacitor (background processing) and Chronograf (the UI) all in a single binary. It is designed to handle high write and query loads and provides a SQL-like query language called InfluxQL for interacting with data.

Beringei - High performance, in-memory storage engine for time series data.

  •    C++

In the fall of 2015, we published the paper “Gorilla: A Fast, Scalable, In-Memory Time Series Database” at VLDB 2015. Beringei is the open source representation of the ideas presented in this paper. Beringei is a high performance time series storage engine. Time series are commonly used as a representation of statistics, gauges, and counters for monitoring performance and health of a system.

Druid IO - Real Time Exploratory Analytics on Large Datasets

  •    Java

Druid is a distributed, column-oriented, real-time analytics data store that is commonly used to power exploratory dashboards in multi-tenant environments. Druid excels as a data warehousing solution for fast aggregate queries on petabyte sized data sets. Druid supports a variety of flexible filters, exact calculations, approximate algorithms, and other useful calculations. Druid can load both streaming and batch data.

Prometheus - Service Monitoring System and Time Series Database

  •    Go

Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

QuestDB - Fast relational time-series database

  •    Java

QuestDB is an open-source NewSQL relational database designed to process time-series data, faster. QuestDB ingests data via HTTP, PostgresSQL wire protocol, Influx line protocol or directly from Java. Reading data is done using SQL via HTTP, PostgreSQL wire protocol or via Java API. The whole database and console fits in a 3.5Mb package.

Pinot - A realtime distributed OLAP datastore

  •    Java

Pinot is a realtime distributed OLAP datastore, which is used at LinkedIn to deliver scalable real time analytics with low latency. It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). Pinot is designed to scale horizontally, so that it can scale to larger data sets and higher query rates as needed.

EventQL - The database for large-scale event analytics

  •    C++

EventQL is a distributed, column-oriented database built for large-scale event collection and analytics. It runs super-fast SQL and MapReduce queries. Its features include Automatic partitioning, Columnar storage, Standard SQL support, Scales to petabytes, Timeseries and relational data, Fast range scans and lot more.