Big Queue - A big, fast and persistent queue based on memory mapped file.

  •        76

A big, fast and persistent queue based on memory mapped file.

Features include:

  1. Fast: close to the speed of direct memory access, both enqueue and dequeue are close to O(1) memory access.
  2. Big: the total size of the queue is only limited by the available disk space.
  3. Persistent: all data in the queue is persisted on disk, and is crash resistant.
  4. Reliable: OS will be responsible to presist the produced messages even your process crashes.
  5. Realtime: messages produced by producer threads will be immediately visible to consumer threads.
  6. Memory-efficient: automatic paging & swapping algorithm, only most-recently accessed data is kept in memory.
  7. Thread-safe: multiple threads can concurrently enqueue and dequeue without data corruption.
  8. Simple&Light-weight: current number of source files is 12 and the library jar is less than 30K.

http://bulldog2011.github.com/bigqueue

Tags
Implementation
License
Platform

   




Related Projects

Redisson - Redis based In-Memory Data Grid for Java


Redisson - distributed Java objects and services (Set, Multimap, SortedSet, Map, List, Queue, BlockingQueue, Deque, BlockingDeque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Executor service, Tomcat Session Manager, Scheduler service, JCache API) on top of Redis server. Rich Redis client.

LArray - Large off-heap arrays for Java/Scala


A library for managing large off-heap arrays that can hold more than 2G (2^31) entries in Java and Scala. Notably LArray is disposable by calling LArray.free or you can let GC automatically release the memory. LArray also can be used to create an mmap (memory-mapped file) whose size is more than 2GB

Agrona - Library to build high-performance applications in Java and C++


Agrona provides a library of data structures and utility methods that are a common need when building high-performance applications in Java and C++. It supports Buffers, Map, Sets, Cache, Queues and lot more.

Coolstorage - ORM library for .NET


The main strength of Vici CoolStorage is the ease of use. Most ORM tools still require a lot of unneeded code to accomplish basic data persistence tasks, but Vici CoolStorage is designed to relieve the programmer from these tedious and error-prone tasks, making it very intuitive to use.

Tape - A lightning fast, transactional, file-based FIFO for Android and Java.


A lightning fast, transactional, file-based FIFO for Android and Java. QueueFile is a lightning-fast, transactional, file-based FIFO. Addition and removal from an instance is an O(1) operation and is atomic. Writes are synchronous; data will be written to disk before an operation returns. The underlying file is structured to survive process and even system crashes and if an I/O exception is thrown during a mutating change, the change is aborted.


Eclipse Collections - Best Java Collection Framework


Eclipse Collections is a collections framework for Java. It has JDK-compatible List, Set and Map implementations with a rich API, additional types not found in the JDK like Bags, Multimaps, and set of utility classes that work with any JDK compatible Collections, Arrays, Maps, or Strings. The iteration protocol was inspired by the Smalltalk collection framework. It maximizes the power of Java 8 Lambda expressions and method references with rich APIs directly available on your collections.

JCTools - Java Concurrency Tools for the JVM


Java Concurrency Tools for the JVM. This project aims to offer some concurrent data structures currently missing from the JDK.

Koloboke - Java Collections till the last breadcrumb of memory and performance


Koloboke aims to replace the standard Java collections and streams with more efficient implementations. The current version of Koloboke focuses on replacing java.util.HashSet and java.util.HashMap. It provides a complete set of primitive type implementations for each collection. Its able to avoid the expensive boxing/unboxing of primitives and saves memory for boxed primitive objects. It is the fastest and the most memory efficient library implementing hash maps and sets.

gs-collections - A supplement or replacement for the Java Collections Framework


GS Collections is a collections framework for Java. It has JDK-compatible List, Set and Map implementations with a rich API and set of utility classes that work with any JDK compatible Collections, Arrays, Maps or Strings. The iteration protocol was inspired by the Smalltalk collection framework. It provides replacements for ArrayList, HashSet, and HashMap optimized for performance and memory usage.

Redisson - Redis based In-Memory Data Grid for Java


Redisson - Distributed and Scalable Java data structures (Set, SortedSet, Map, ConcurrentMap, List, Queue, Deque, Lock, AtomicLong, CountDownLatch, Publish / Subscribe, HyperLogLog) on top of Redis server. Advanced redis java client. It supports over 28+ data structures and services, Synchronous / asynchronous / reactive interfaces and lot more.

Koloboke - Java Collections till the last breadcrumb of memory and performance


A family of projects around collections in Java (so far). A carefully designed extension of the Java Collections Framework with primitive specializations and more. Java 6+. Apache 2.0 license.

MapDB - Embedded Database Engine


MapDB is an embedded database engine. It provides Maps and other collections backed by disk or memory storage. It offers excellent performance comparable to java collections, but is not limited by GC overhead. It is also a full database engine with storage backends, transactions, cache algorithms, expiration and many other options. MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap memory.

Judy - General purpose dynamic array


Judy is a general purpose dynamic array implemented as a C callable library. Judy's speed and memory usage are typically better than other data storage models and improves with very large data sets. Judy's key benefits are scalability, high performance, and memory efficiency. A Judy array is extensible and can scale up to a very large number of elements, bounded only by machine memory. Since Judy is designed as an unbounded array, the size of a Judy array is not pre-allocated but grows and shrinks dynamically with the array population.

Luxun - A high-throughput, persistent, distributed, publish-subscribe messaging system based on memo


A high-throughput, persistent, distributed, publish-subscribe messaging system based on memory mapped file and Thrift RPC.

HPPC - High Performance Primitive Collections for Java


HPPC provides template-generated implementations of typical collections, such as lists, sets and maps, for all Java primitive types. The primary driving force behind HPPC is optimization for highest performance and memory efficiency.

Boon - Low Ceremony. High Productivity. A real boon to Java to developers!


The main goal of Boon is to reduce the productivity delta between Java and Python/Groovy/Ruby. The idea is to just create a lower ceremony APIs. Less Java like APIs, and more getting things done quickly APIs. If reading a file takes one line of code in Ruby/Python/Groovy, then it will also in Java. If slice notation is part of those language, then Boon will have it to.

collections - This package contains JavaScript implementations of common data structures with idiomatic interfaces


This package contains JavaScript implementations of common data structures with idiomatic iterfaces, including extensions for Array and Object. You can use these Node Packaged Modules with Node.js, Browserify, Mr, or any compatible CommonJS module loader. Using a module loader or bundler when using Collections in web browsers has the advantage of only incorporating the modules you need. However, you can just embed <script src="collections/collections.min.js"> and all of the collections will be introduced as globals. ⚠️ require("collections") is not supported.

DynamicData - Reactive collections based on Rx.Net


Dynamic Data is a portable class library which brings the power of Reactive Extensions (Rx) to collections.Rx is extremely powerful but out of the box provides nothing to assist with managing collections. In most applications there is a need to update the collections dynamically. Typically a collection is loaded and after the initial load, asynchronous updates are received. The original collection will need to reflect these changes. In simple scenarios the code is simple. However, typical applications are much more complicated and may apply a filter, transform the original dto and apply a sort. Even with these simple every day operations the complexity of the code is quickly magnified. Dynamic data has been developed to remove the tedious code of dynamically maintaining collections. It has grown to become functionally very rich with at least 60 collection based operations which amongst other things enable filtering, sorting, grouping, joining different sources, transforms, binding, pagination, data virtualisation, expiration, disposal management plus more.

Apache Commons - Alternative to Java collection


Commons-Collections seek to build upon the JDK classes by providing new interfaces, implementations and utilities. It provides collection class like Identity map, Ordered map, Reference map, Composite collections, Bag interface etc.





We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.