Apache Mnemonic - Non-volatile hybrid memory storage oriented library

  •        4196

Apache Mnemonic is a non-volatile hybrid memory storage oriented library, it proposed a non-volatile/durable Java object model and durable computing service that bring several advantages to significantly improve the performance of massive real-time data processing/analytics. developers are able to use this library to design their cache-less and SerDe-less high performance applications.

  • In-place data storage on local non-volatile memory
  • Durable Object Model (DOM)
  • Durable Native Computing Model (DNCM)
  • Object graphs lazy loading & sharing
  • Auto-reclaim memory resources and Mnemonic objects
  • Hierarchical cache pool for massive data caching
  • Extensible memory services for new device adoption and allocation optimization
  • Durable data structure collection(WIP)
  • Durable computing service
  • Minimize memory footprint of on-heap
  • Reduce GC Overheads as the following chart shown (collected from Apache Spark experiments)
  • Drop-in Hadoop MapReduce support
  • Drop-in Hadoop Spark support




Related Projects

pmem - persistant memory programming

  •    C

Persistent memory (or pmem for short) is accessed like volatile memory, using processor load and store instructions, but it retains its contents across power loss like storage. This project focuses specifically on how persistent memory is exposed to server-class applications which will explicitly manage the placement of data among the three tiers (volatile memory, persistent memory, and storage).

daos - DAOS Storage Engine

  •    C

The Distributed Asynchronous Object Storage (DAOS) is an open-source software-defined object store designed from the ground up for massively distributed Non Volatile Memory (NVM). DAOS takes advantage of next generation NVM technology like Storage Class Memory (SCM) and NVM express (NVMe) while presenting a key-value storage interface and providing features such as transactional non-blocking I/O, advanced data protection with self-healing on top of commodity hardware, end-to-end data integrity, fine-grained data control and elastic storage to optimize performance and cost. DAOS is licensed under the BSD-2-Clause Plus Patent License. Please see the LICENSE & NOTICE files for more information.

c-blosc - A blocking, shuffling and loss-less compression library that can be faster than `memcpy()`

  •    C

Blosc is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc is the first compressor (that I'm aware of) that is meant not only to reduce the size of large datasets on-disk or in-memory, but also to accelerate memory-bound computations. It uses the blocking technique so as to reduce activity in the memory bus as much as possible. In short, this technique works by dividing datasets in blocks that are small enough to fit in caches of modern processors and perform compression / decompression there. It also leverages, if available, SIMD instructions (SSE2, AVX2) and multi-threading capabilities of CPUs, in order to accelerate the compression / decompression process to a maximum.

Redisson - Redis based In-Memory Data Grid for Java

  •    Java

Redisson - distributed Java objects and services (Set, Multimap, SortedSet, Map, List, Queue, BlockingQueue, Deque, BlockingDeque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Executor service, Tomcat Session Manager, Scheduler service, JCache API) on top of Redis server. Rich Redis client.

Hazelcast Jet - A general purpose distributed data processing engine, built on top of Hazelcast.

  •    Java

Hazelcast Jet is a distributed computing platform built for high-performance stream processing and fast batch processing. It embeds Hazelcast In-Memory Data Grid (IMDG) to provide a lightweight, simple-to-deploy package that includes scalable in-memory storage. Hazelcast Jet performs parallel execution to enable data-intensive applications to operate in near real-time.

MapDB - Embedded Database Engine

  •    Java

MapDB is an embedded database engine. It provides Maps and other collections backed by disk or memory storage. It offers excellent performance comparable to java collections, but is not limited by GC overhead. It is also a full database engine with storage backends, transactions, cache algorithms, expiration and many other options. MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap memory.

DiskCache - Python disk backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python

  •    Python

DiskCache is a disk and file backed cache library, written in pure-Python, and compatible with Django. The cloud-based computing of 2021 puts a premium on memory. Gigabytes of empty space is left on disks as processes vie for memory. Among these processes is Memcached (and sometimes Redis) which is used as a cache. It is good to use empty disk space for caching.

OHC - Java large off heap cache

  •    Java

Off-Heap concurrent hash map intended to store GBs of serialized data. It has optional per entry or default TTL/expireAt, Entry eviction and expiration without a separate thread, Capable of maintaining huge amounts of cache memory, Suitable for tiny/small entries with low overhead using the chunked implementation.

nitro - A high performance in-memory index storage engine

  •    Go

A high performance in-memory key-value item storage engine written in golang. The storage engine is based lock-free data structures and scales well with multicore CPUs.

PINCache - Fast, non-deadlocking parallel object cache for iOS, tvOS and OS X

  •    Objective-C

PINCache is a fork of TMCache re-architected to fix issues with deadlocking caused by heavy use. It is a key/value store designed for persisting temporary objects that are expensive to reproduce, such as downloaded data or the results of slow processing. It is comprised of two self-similar stores, one in memory (PINMemoryCache) and one on disk (PINDiskCache), all backed by GCD and safe to access from multiple threads simultaneously. On iOS, PINMemoryCache will clear itself when the app receives a memory warning or goes into the background. Objects stored in PINDiskCache remain until you trim the cache yourself, either manually or by setting a byte or age limit. Both PINMemoryCache and PINDiskCache use locks to protect reads and writes. PINCache coordinates them so that objects added to memory are available immediately to other threads while being written to disk safely in the background. Both caches are public properties of PINCache, so it's easy to manipulate one or the other separately if necessary.

LiME - LiME (formerly DMD) is a Loadable Kernel Module (LKM), which allows the acquisition of volatile memory from Linux and Linux-based devices, such as those powered by Android

  •    C

A Loadable Kernel Module (LKM) which allows for volatile memory acquisition from Linux and Linux-based devices, such as Android. This makes LiME unique as it is the first tool that allows for full memory captures on Android devices. It also minimizes its interaction between user and kernel space processes during acquisition, which allows it to produce memory captures that are more forensically sound than those of other tools designed for Linux memory acquisition. Detailed documentation on LiME's usage and internals can be found in the "doc" directory of the project.

Apache Ignite - High performance in-memory data grid

  •    Java

Apache Ignite In-Memory Data Fabric is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies.

fatcache - Memcache on SSD

  •    C

fatcache is memcache on SSD. Think of fatcache as a cache for your big data.There are two ways to think of SSDs in system design. One is to think of SSD as an extension of disk, where it plays the role of making disks fast and the other is to think of them as an extension of memory, where it plays the role of making memory fat. The latter makes sense when persistence (non-volatility) is unnecessary and data is accessed over the network. Even though memory is thousand times faster than SSD, network connected SSD-backed memory makes sense, if we design the system in a way that network latencies dominate over the SSD latencies by a large factor.

vaex - Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualize and explore big tabular data at a billion rows per second 🚀

  •    Python

Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. It calculates statistics such as mean, sum, count, standard deviation etc, on an N-dimensional grid for more than a billion (10^9) samples/rows per second. Visualization is done using histograms, density plots and 3d volume rendering, allowing interactive exploration of big data. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted). HDF5 and Apache Arrow supported.

LeakCanary - A memory leak detection library for Android and Java

  •    Kotlin

LeakCanary is a memory leak detection library for Android. LeakCanary’s knowledge of the internals of the Android Framework gives it a unique ability to narrow down the cause of each leak, helping developers dramatically reduce OutOfMemoryError crashes.

Hazelcast - In-Memory Data Grid for Java

  •    Java

Hazelcast is a clustering and highly scalable data distribution platform for Java. It supports Distributed implementations of java.util.{Queue, Set, List, Map}, java.util.concurrency.locks.Lock, java.util.concurrent.ExecutorService, Distributed Indexing and Query support, Dynamic scaling, partitioning with backups, fail-over, Web-based cluster monitoring tool and lot more.

snappydata - Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™

  •    Scala

SnappyData (aka TIBCO ComputeDB) is a distributed, in-memory optimized analytics database. SnappyData delivers high throughput, low latency, and high concurrency for unified analytics workload. By fusing an in-memory hybrid database inside Apache Spark, it provides analytic query processing, mutability/transactions, access to virtually all big data sources and stream processing all in one unified cluster. One common use case for SnappyData is to provide analytics at interactive speeds over large volumes of data with minimal or no pre-processing of the dataset. For instance, there is no need to often pre-aggregate/reduce or generate cubes over your large data sets for ad-hoc visual analytics. This is made possible by smartly managing data in-memory, dynamically generating code using vectorization optimizations and maximizing the potential of modern multi-core CPUs. SnappyData enables complex processing on large data sets in sub-second timeframes.

Zephyr - Scalable real-time operating system (RTOS) supporting multiple hardware architectures

  •    C

The Zephyr Project is a scalable real-time operating system (RTOS) supporting multiple hardware architectures, optimized for resource constrained devices, and built with security in mind. The Zephyr OS is based on a small-footprint kernel designed for use on resource-constrained systems: from simple embedded environmental sensors and LED wearables to sophisticated smart watches and IoT wireless gateways.

netflix-graph - Compact in-memory representation of directed graph data

  •    Java

NetflixGraph is a compact in-memory data structure used to represent directed graph data. You can use NetflixGraph to vastly reduce the size of your application’s memory footprint, potentially by an order of magnitude or more. If your application is I/O bound, you may be able to remove that bottleneck by holding your entire dataset in RAM. This may be possible with NetflixGraph; you’ll likely be very surprised by how little memory is actually required to represent your data.NetflixGraph provides an API to translate your data into a graph format, compress that data in memory, then serialize the compressed in-memory representation of the data so that it may be easily transported across your infrastructure.

Chronicle Map - High performance, off-heap, key-value, in memory, persisted data store

  •    Java

Chronicle Map is a high performance, off-heap, key-value, in memory, persisted data store. It works like a standard java map yet it automatically distributes data between processes, these processes can be both on the same server or across your network. In other words its a low latency, huge data key value store, which can store terabytes of data locally to your process.

We have large collection of open source products. Follow the tags from Tag Cloud >>

Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.