Big Queue - A big, fast and persistent queue based on memory mapped file.

  •        55

A big, fast and persistent queue based on memory mapped file.

Features include:

  1. Fast: close to the speed of direct memory access, both enqueue and dequeue are close to O(1) memory access.
  2. Big: the total size of the queue is only limited by the available disk space.
  3. Persistent: all data in the queue is persisted on disk, and is crash resistant.
  4. Reliable: OS will be responsible to presist the produced messages even your process crashes.
  5. Realtime: messages produced by producer threads will be immediately visible to consumer threads.
  6. Memory-efficient: automatic paging & swapping algorithm, only most-recently accessed data is kept in memory.
  7. Thread-safe: multiple threads can concurrently enqueue and dequeue without data corruption.
  8. Simple&Light-weight: current number of source files is 12 and the library jar is less than 30K.

http://bulldog2011.github.com/bigqueue

Tags
Implementation
License
Platform

   




Related Projects

Redisson - Redis based In-Memory Data Grid for Java


Redisson - distributed Java objects and services (Set, Multimap, SortedSet, Map, List, Queue, BlockingQueue, Deque, BlockingDeque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Executor service, Tomcat Session Manager, Scheduler service, JCache API) on top of Redis server. Rich Redis client.

Coolstorage - ORM library for .NET


The main strength of Vici CoolStorage is the ease of use. Most ORM tools still require a lot of unneeded code to accomplish basic data persistence tasks, but Vici CoolStorage is designed to relieve the programmer from these tedious and error-prone tasks, making it very intuitive to use.

Tape - A lightning fast, transactional, file-based FIFO for Android and Java.


A lightning fast, transactional, file-based FIFO for Android and Java. QueueFile is a lightning-fast, transactional, file-based FIFO. Addition and removal from an instance is an O(1) operation and is atomic. Writes are synchronous; data will be written to disk before an operation returns. The underlying file is structured to survive process and even system crashes and if an I/O exception is thrown during a mutating change, the change is aborted.

Eclipse Collections - Best Java Collection Framework


Eclipse Collections is a collections framework for Java. It has JDK-compatible List, Set and Map implementations with a rich API, additional types not found in the JDK like Bags, Multimaps, and set of utility classes that work with any JDK compatible Collections, Arrays, Maps, or Strings. The iteration protocol was inspired by the Smalltalk collection framework. It maximizes the power of Java 8 Lambda expressions and method references with rich APIs directly available on your collections.



JCTools - Java Concurrency Tools for the JVM


Java Concurrency Tools for the JVM. This project aims to offer some concurrent data structures currently missing from the JDK.

Koloboke - Java Collections till the last breadcrumb of memory and performance


Koloboke aims to replace the standard Java collections and streams with more efficient implementations. The current version of Koloboke focuses on replacing java.util.HashSet and java.util.HashMap. It provides a complete set of primitive type implementations for each collection. Its able to avoid the expensive boxing/unboxing of primitives and saves memory for boxed primitive objects. It is the fastest and the most memory efficient library implementing hash maps and sets.

Redisson - Redis based In-Memory Data Grid for Java


Redisson - Distributed and Scalable Java data structures (Set, SortedSet, Map, ConcurrentMap, List, Queue, Deque, Lock, AtomicLong, CountDownLatch, Publish / Subscribe, HyperLogLog) on top of Redis server. Advanced redis java client. It supports over 28+ data structures and services, Synchronous / asynchronous / reactive interfaces and lot more.

gs-collections - A supplement or replacement for the Java Collections Framework


GS Collections is a collections framework for Java. It has JDK-compatible List, Set and Map implementations with a rich API and set of utility classes that work with any JDK compatible Collections, Arrays, Maps or Strings. The iteration protocol was inspired by the Smalltalk collection framework. It provides replacements for ArrayList, HashSet, and HashMap optimized for performance and memory usage.

MapDB - Embedded Database Engine


MapDB is an embedded database engine. It provides Maps and other collections backed by disk or memory storage. It offers excellent performance comparable to java collections, but is not limited by GC overhead. It is also a full database engine with storage backends, transactions, cache algorithms, expiration and many other options. MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap memory.

Agrona - Library to build high-performance applications in Java and C++


Agrona provides a library of data structures and utility methods that are a common need when building high-performance applications in Java and C++. It supports Buffers, Map, Sets, Cache, Queues and lot more.

Luxun - A high-throughput, persistent, distributed, publish-subscribe messaging system based on memo


A high-throughput, persistent, distributed, publish-subscribe messaging system based on memory mapped file and Thrift RPC.

HPPC - High Performance Primitive Collections for Java


HPPC provides template-generated implementations of typical collections, such as lists, sets and maps, for all Java primitive types. The primary driving force behind HPPC is optimization for highest performance and memory efficiency.

MappedQueue - Memory Mapped Persistent Queue


Memory Mapped Persistent Queue

banana - Banana is a Java primitive collections library with its own memory management


Banana is a Java primitive collections library with its own memory management

bounded-spsc-queue - A Bounded SPSC queue for Rust


[![Nightly Build Status](https://travis-ci.org/polyfractal/bounded-spsc-queue.svg?branch=master)](https://travis-ci.org/polyfractal/bounded-spsc-queue)This crate provides a very simple bounded, Single-producer Single-consumer (SPSC)queue for Rust. It provides a data structure for two threads to communicatein one direction with minimal overhead and bounded semantics.Compared to a `sync_channel`, this queue provides a small but consistentspeedup. `sync_channel` utilizes an unbounded linked-list

Apache Commons - Alternative to Java collection


Commons-Collections seek to build upon the JDK classes by providing new interfaces, implementations and utilities. It provides collection class like Identity map, Ordered map, Reference map, Composite collections, Bag interface etc.

go-adaptive-radix-tree - Adaptive Radix Trees implemented in Go


This library provides a Go implementation of the Adaptive Radix Tree (ART).The go-adaptive-radix-tree library overperforms go-art library. The go-adaptive-radix-tree doesn't allocate any memory during search operations. It also provides prefix based iteration over the tree.

count-min-log - Go implementation of Count-Min-Log


Count-Min Sketch is a widely adopted algorithm for approximate event counting in large scale processing. However, the original version of the Count-Min-Sketch (CMS) suffers of some deficiences, especially if one is interested in the low-frequency items, such as in text- mining related tasks. Several variants of CMS have been proposed to compensate for the high relative error for low-frequency events, but the proposed solutions tend to correct the errors instead of preventing them. In this paper, we propose the Count-Min-Log sketch, which uses logarithm-based, approximate counters instead of linear counters to improve the average relative error of CMS at constant memory footprint.This version implements the 16 bit register version. Will add back the 8-bit version soon.