Displaying 1 to 9 from 9 results

Generate and test (zipfian) skewed distribution (100.000.000 integers, Block size=128/256) Note: Unlike general purpose compression, a small fixed size (ex. 128 integers) is in general used in "integer compression". Large blocks involved, while processing queries (inverted index, search engines, databases, graphs, in memory computing,...) need to be entirely decoded. (*) codecs inefficient for small block sizes are tested with 64Ki integers/block.

integer-compression compression encoding sse2 avx2 simd time-series inverted-index intersection compressor librarySuppose you want to compress a large array of values with (relatively) few distinct values. For example, maybe you have 16 distinct 64-bit values. Only four bits are needed to store a value in the range [0,16) using binary packing, so if you have long arrays, it is possible to save 60 bits per value (compress the data by a factor of 16).We consider the following (simple) form of dictionary coding. We have a dictionary of 64-bit values (could be pointers) stored in an array. In the compression phase, we convert the values to indexes and binary pack them. In the decompression phase, we try to recover the dictionary-coded values as fast as possible.

integer-compression simd-instructions simdThese functions compute fast successive differences, and recover the original values from the fast successive differences (i.e., they compute a prefix sum) using fast SIMD instructions.They can be useful as part of compressed index.

simd-instructions simd integer-compression compressed prefix-sumC++ library to pack and unpack vectors of integers having a small range of values using a technique called Frame of Reference (Goldstein et al. 1998). It should run fast even though it is written in simple C++.There is a similar API with turbocompress and turbouncompress with the difference that compresseddata uses an uint8_t pointer type.

compression integer-compression visual-studio clang gccThis is a bit packing library. Bit packing is the process by which you take arrays of (32-bit) integers that fit in [0,2^L) and you write them out using as close to L bits per integer as possible.For long blocks of integers (e.g., 128 integers or more), SIMD bit packing is ideal. See for example https://github.com/lemire/simdcomp where blocks of 128 integers are packed and unpacked very fast.

integer integer-compression compression performanceFast decoder for VByte-compressed integers in C.It includes fast differential coding.

vbyte vbyte-compressed-integers compression integer-compression simd simd-instructionsAs the name suggests, this is a C/C++ library for fast compression and intersection of lists of sorted integers using SIMD instructions. The library focuses on innovative techniques and very fast schemes, with particular attention to differential coding. It introduces new SIMD intersections schemes such as SIMD Galloping.This library can decode at least 4 billions of compressed integers per second on most desktop or laptop processors. That is, it can decompress data at a rate of 15 GB/s. This is significantly faster than generic codecs like gzip, LZO, Snappy or LZ4.

compression integer-compression simd-instructions simd algorithms intersectionStreamVByte is a new integer compression technique that applies SIMD instructions (vectorization) to Google's Group Varint approach. The net result is faster than other byte-oriented compression techniques.The approach is patent-free, the code is available under the Apache License.

integer-compression compression simdThis is an integer compression library in JavaScript, useful for work on indexes. Given an array of small integers, it produces an ArrayBuffer that uses far fewer bytes than the original (using VByte compression). It assumes a modern JavaScript engine with typed arrays. From the compressed data, you can later recover the original array quickly (at a rate of millions of integers per second).

compression integer-compression integers performance
We have large collection of open source products. Follow the tags from
Tag Cloud >>

Open source products are scattered around the web. Please provide information
about the open source projects you own / you use.
**Add Projects.**