LArray - Large off-heap arrays for Java/Scala

  •        470

A library for managing large off-heap arrays that can hold more than 2G (2^31) entries in Java and Scala. Notably LArray is disposable by calling LArray.free or you can let GC automatically release the memory. LArray also can be used to create an mmap (memory-mapped file) whose size is more than 2GB

https://github.com/xerial/larray

Tags
Implementation
License
Platform

   




Related Projects

Agrona - Library to build high-performance applications in Java and C++

  •    Java

Agrona provides a library of data structures and utility methods that are a common need when building high-performance applications in Java and C++. It supports Buffers, Map, Sets, Cache, Queues and lot more.

Capsule - The Capsule Hash Trie Collections Library

  •    Java

Capsule aims to become a full-fledged (immutable) collections library for Java 8+ that is solely built around persistent tries. The library is designed for standalone use and for being embedded in domain-specific languages. Capsule still has to undergo some incubation before it can ship as a well-rounded collection library. Nevertheless, the code is stable and performance is solid.

Judy - General purpose dynamic array

  •    C

Judy is a general purpose dynamic array implemented as a C callable library. Judy's speed and memory usage are typically better than other data storage models and improves with very large data sets. Judy's key benefits are scalability, high performance, and memory efficiency. A Judy array is extensible and can scale up to a very large number of elements, bounded only by machine memory. Since Judy is designed as an unbounded array, the size of a Judy array is not pre-allocated but grows and shrinks dynamically with the array population.

OHC - Java large off heap cache

  •    Java

Off-Heap concurrent hash map intended to store GBs of serialized data. It has optional per entry or default TTL/expireAt, Entry eviction and expiration without a separate thread, Capable of maintaining huge amounts of cache memory, Suitable for tiny/small entries with low overhead using the chunked implementation.

MapDB - Embedded Database Engine

  •    Java

MapDB is an embedded database engine. It provides Maps and other collections backed by disk or memory storage. It offers excellent performance comparable to java collections, but is not limited by GC overhead. It is also a full database engine with storage backends, transactions, cache algorithms, expiration and many other options. MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap memory.


Redisson - Redis based In-Memory Data Grid for Java

  •    Java

Redisson - distributed Java objects and services (Set, Multimap, SortedSet, Map, List, Queue, BlockingQueue, Deque, BlockingDeque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Executor service, Tomcat Session Manager, Scheduler service, JCache API) on top of Redis server. Rich Redis client.

DiskCache - Python disk backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python

  •    Python

DiskCache is a disk and file backed cache library, written in pure-Python, and compatible with Django. The cloud-based computing of 2021 puts a premium on memory. Gigabytes of empty space is left on disks as processes vie for memory. Among these processes is Memcached (and sometimes Redis) which is used as a cache. It is good to use empty disk space for caching.

Apache Mnemonic - Non-volatile hybrid memory storage oriented library

  •    Java

Apache Mnemonic is a non-volatile hybrid memory storage oriented library, it proposed a non-volatile/durable Java object model and durable computing service that bring several advantages to significantly improve the performance of massive real-time data processing/analytics. developers are able to use this library to design their cache-less and SerDe-less high performance applications.

Chronicle Map - High performance, off-heap, key-value, in memory, persisted data store

  •    Java

Chronicle Map is a high performance, off-heap, key-value, in memory, persisted data store. It works like a standard java map yet it automatically distributes data between processes, these processes can be both on the same server or across your network. In other words its a low latency, huge data key value store, which can store terabytes of data locally to your process.

Valgrind

  •    C

Valgrind is an award-winning instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.

JCTools - Java Concurrency Tools for the JVM

  •    Java

Java Concurrency Tools for the JVM. This project aims to offer some concurrent data structures currently missing from the JDK.

DirectMemory

  •    Java

DirectMemory is a cache implementation featuring off-heap memory storage (a-la BigMemory) to enable caching of large (or large numbers of) objects without degrading jvm performance. ATTENTION PLEASE: DIRECTMEMORY IS MOVING TO APACHE - check incubation status here:

STINGER - In-memory graph store and dynamic graph analysis platform

  •    C

STINGER is a package designed to support streaming graph analytics by using in-memory parallel computation to accelerate the computation. STINGER is composed of the core data structure and the STINGER server, algorithms, and an RPC server that can be used to run queries and serve visualizations.

mio - Cross-platform C++11 header-only library for memory mapped file IO

  •    C++

An easy to use header-only cross-platform C++11 memory mapping library with an MIT license. mio has been created with the goal to be easily includable (i.e. no dependencies) in any C++ project that needs memory mapped file IO without the need to pull in Boost.

pmem - persistant memory programming

  •    C

Persistent memory (or pmem for short) is accessed like volatile memory, using processor load and store instructions, but it retains its contents across power loss like storage. This project focuses specifically on how persistent memory is exposed to server-class applications which will explicitly manage the placement of data among the three tiers (volatile memory, persistent memory, and storage).






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.