Displaying 1 to 9 from 9 results

TurboPFor - Fastest Integer Compression

  •    C

Generate and test (zipfian) skewed distribution (100.000.000 integers, Block size=128/256) Note: Unlike general purpose compression, a small fixed size (ex. 128 integers) is in general used in "integer compression". Large blocks involved, while processing queries (inverted index, search engines, databases, graphs, in memory computing,...) need to be entirely decoded. (*) codecs inefficient for small block sizes are tested with 64Ki integers/block.

dictionary - High-performance dictionary coding

  •    C++

Suppose you want to compress a large array of values with (relatively) few distinct values. For example, maybe you have 16 distinct 64-bit values. Only four bits are needed to store a value in the range [0,16) using binary packing, so if you have long arrays, it is possible to save 60 bits per value (compress the data by a factor of 16).We consider the following (simple) form of dictionary coding. We have a dictionary of 64-bit values (could be pointers) stored in an array. In the compression phase, we convert the values to indexes and binary pack them. In the decompression phase, we try to recover the dictionary-coded values as fast as possible.

FastDifferentialCoding - Fast differential coding functions (using SIMD instructions)

  •    C

These functions compute fast successive differences, and recover the original values from the fast successive differences (i.e., they compute a prefix sum) using fast SIMD instructions.They can be useful as part of compressed index.

FrameOfReference - C++ library to pack and unpack vectors of integers having a small range of values using a technique called Frame of Reference

  •    C

C++ library to pack and unpack vectors of integers having a small range of values using a technique called Frame of Reference (Goldstein et al. 1998). It should run fast even though it is written in simple C++.There is a similar API with turbocompress and turbouncompress with the difference that compresseddata uses an uint8_t pointer type.

LittleIntPacker - C library to pack and unpack short arrays of integers as fast as possible

  •    C

This is a bit packing library. Bit packing is the process by which you take arrays of (32-bit) integers that fit in [0,2^L) and you write them out using as close to L bits per integer as possible.For long blocks of integers (e.g., 128 integers or more), SIMD bit packing is ideal. See for example https://github.com/lemire/simdcomp where blocks of 128 integers are packed and unpacked very fast.

SIMDCompressionAndIntersection - A C++ library to compress and intersect sorted lists of integers using SIMD instructions

  •    C++

As the name suggests, this is a C/C++ library for fast compression and intersection of lists of sorted integers using SIMD instructions. The library focuses on innovative techniques and very fast schemes, with particular attention to differential coding. It introduces new SIMD intersections schemes such as SIMD Galloping.This library can decode at least 4 billions of compressed integers per second on most desktop or laptop processors. That is, it can decompress data at a rate of 15 GB/s. This is significantly faster than generic codecs like gzip, LZO, Snappy or LZ4.

streamvbyte - Fast integer compression in C using the StreamVByte codec

  •    C

StreamVByte is a new integer compression technique that applies SIMD instructions (vectorization) to Google's Group Varint approach. The net result is faster than other byte-oriented compression techniques.The approach is patent-free, the code is available under the Apache License.

FastIntegerCompression.js - Fast integer compression library in JavaScript

  •    Javascript

This is an integer compression library in JavaScript, useful for work on indexes. Given an array of small integers, it produces an ArrayBuffer that uses far fewer bytes than the original (using VByte compression). It assumes a modern JavaScript engine with typed arrays. From the compressed data, you can later recover the original array quickly (at a rate of millions of integers per second).