FiniteStateEntropy - New generation entropy codecs : Finite State Entropy and Huff0

  •        203

Huff0, a Huffman codec designed for modern CPU, featuring OoO (Out of Order) operations on multiple ALU (Arithmetic Logic Unit), achieving extremely fast compression and decompression speeds.FSE is a new kind of Entropy encoder, based on ANS theory, from Jarek Duda, achieving precise compression accuracy (like Arithmetic coding) at much higher speeds.

https://github.com/Cyan4973/FiniteStateEntropy

Tags
Implementation
License
Platform

   




Related Projects

Entropy Compression Algorithms

  •    C++

Major looseless compression algorithms library and documentation. First project: Arithmetic, Huffman, LZ77, LZ78, LZW, RLE. Second project reimplements Deflate. Documentation explains major Entropy Compression Methods.

7-Zip-zstd - 7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard

  •    C

Zstandard v1.3.7 is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder. Brotli v.1.0.7 is a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding and 2nd order context modeling, with a compression ratio comparable to the best currently available general-purpose compression methods. It is similar in speed with deflate but offers more dense compression.

Basic Compression Library

  •    C

Basic Compression Library is a set of open source implementations of RLE (Run Length Encoding), Huffman, Rice, Lempel-Ziv (LZ77) and Shannon-Fano compression algorithms.

lzfse - LZFSE compression library and command line tool

  •    C

This is a reference C implementation of the LZFSE compressor introduced in the Compression library with OS X 10.11 and iOS 9. LZFSE is a Lempel-Ziv style data compression algorithm using Finite State Entropy coding. It targets similar compression rates at higher compression and decompression speed compared to deflate using zlib.

brotli - Brotli compression format

  •    C

Brotli is a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding and 2nd order context modeling, with a compression ratio comparable to the best currently available general-purpose compression methods. It is similar in speed with deflate but offers more dense compression.The specification of the Brotli Compressed Data Format is defined in RFC 7932.


ngx_brotli - NGINX module for Brotli compression

  •    C

Brotli is a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding and 2nd order context modeling, with a compression ratio comparable to the best currently available general-purpose compression methods. It is similar in speed with deflate but offers more dense compression.Both Brotli library and nginx module are under active development.

LZF Compressor- High-performance, streaming/chunking Java LZF codec, compatible with standard C LZF package

  •    Java

LZF-compress is a Java library for encoding and decoding data in LZF format, written by Tatu Saloranta. LZF alfgorithm itself is optimized for speed, with somewhat more modest compression. Compared to the standard Deflate (algorithm gzip uses) LZF can be 5-6 times as fast to compress, and twice as fast to decompress. Compression rate is lower since no Huffman-encoding is used after lempel-ziv substring elimintation.

Deep-Compression-AlexNet - Deep Compression on AlexNet

  •    Python

This is a demo of Deep Compression compressing AlexNet from 233MB to 8.9MB without loss of accuracy. It only differs from the paper that Huffman coding is not applied. Deep Compression's video from ICLR'16 best paper award presentation is available.

FSE - FutraSiteEngine

  •    ASPNET

FSE is quot;new-typequot; website engine (CMS). It quot;skinsquot; to change the site into any style, like blogs, file sharing sites, or even personal sites, all running on FSE unchanged with no coding required. FSE is coded in ASP.NET amp; PHP.

Project-Zipline - Defines a lossless compressed data format that is independent of CPU type, operating system, file system, and character set, and is suitable for compression using the XP10 algorithm

  •    Verilog

Project Zipline is open-sourced under the MIT License, see the LICENSE file. Edit the file ./zipline.setup for VCS, SYNTH, and VERDI environment settings.

minihuff

  •    C

minihuff is a data compression library that enables the creation of a static frequency table to be stored at both ends of a connection. This allows effective compression even for very small pieces of data that maintain similar entropy characteristics.

rtl-entropy - An entropy generator using SDR peripherals, including rtl-sdr and BladeRF

  •    C

rtl-entropy is software using rtl-sdr to turn your DVB-T dongle into a high quality entropy source. It samples atmospheric noise, does Von-Neumann debiasing, runs it through the FIPS 140-2 tests, then optionally (-e) does Kaminsky debiasing if it passes the FIPS tests, then writes to the output. It can be run as a Daemon which by default writes to a FIFO, which can be read by rngd to add entropy to the system pool. If you're serious about the cryptographic security of your entropy source, you should probably short, or put a 75 Ohm load on the antenna port, and put the whole assembly in a shielded box. Then you're getting entropy from the thermal noise of the amplifiers which is much harder to interfere with than atmospheric radio.

Industrial strength BWT compression

  •    C

Help implement an industrial strength Burrows-Wheeler transform compressor! Initial release covers block-sort, MTF coding, RLE coding, Huffman coding

libjpeg-turbo - Main libjpeg-turbo repository

  •    C

libjpeg-turbo is a JPEG image codec that uses SIMD instructions (MMX, SSE2, NEON, AltiVec) to accelerate baseline JPEG compression and decompression on x86, x86-64, ARM, and PowerPC systems. On such systems, libjpeg-turbo is generally 2-6x as fast as libjpeg, all else being equal. On other types of systems, libjpeg-turbo can still outperform libjpeg by a significant amount, by virtue of its highly-optimized Huffman coding routines. In many cases, the performance of libjpeg-turbo rivals that of proprietary high-speed JPEG codecs.libjpeg-turbo implements both the traditional libjpeg API as well as the less powerful but more straightforward TurboJPEG API. libjpeg-turbo also features colorspace extensions that allow it to compress from/decompress to 32-bit and big-endian pixel buffers (RGBX, XBGR, etc.), as well as a full-featured Java interface.

Akumuli - Time-series database

  •    C++

Akumuli is a time-series database for modern hardware. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from Esperanto as "accumulate".

truffleHog - Searches through git repositories for high entropy strings and secrets, digging deep into commit history

  •    Python

Searches through git repositories for secrets, digging deep into commit history and branches. This is effective at finding secrets accidentally committed. truffleHog previously functioned by running entropy checks on git diffs. This functionality still exists, but high signal regex checks have been added, and the ability to surpress entropy checking has also been added.

truffleHog - Searches through git repositories for high entropy strings and secrets, digging deep into commit history

  •    Python

Searches through git repositories for secrets, digging deep into commit history and branches. This is effective at finding secrets accidentally committed. truffleHog previously functioned by running entropy checks on git diffs. This functionality still exists, but high signal regex checks have been added, and the ability to suppress entropy checking has also been added.

infnoise - The world's easiest TRNG to get right

  •    C

For instructions for compiling the and using Infinite Noise TRNG driver, go to the software sub-directory, and read the README file there. Both Linux and Windows are supported. The Infinite Noise TRNG is a USB key hardware true random number generator. It uses what I call a "Modular Entropy Multiplier" architecture (previously Infinite Noise Multiplier or FireBug). Besides being simple, low-cost, and fast, it is much easier to get right than other TRNGs. It naturally defends against influence from outside signals, such as radio interference and power supply noise, making it simple to build securely, without requiring an expert in analog design. Modular entropy multipliers produce a provable and easily measured level of entropy based on thermal noise, approximately equal to log2(K) per output bit, where K is a gain between 1 and 2 set by two resistors around an op-amp. A "health monitor" can track this and verify that the output entropy is within the expected range, which for the Infinite Noise TRNG described below is within 2% of log2(1.82).

Huffman Encoder Decoder

  •    Java

This project is a Huffman Encoder Decoder. The project reads a text file of any length and writes the huffman code to an output text file.

Freenet, Entropy, Samizdat Controller

  •    

Freenet, Entropy, and Samizdat Controller is a small application which allows you to control Freenet, Entropy, and Samizdat. This project is looking for a new maintainer.






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.