csharpewah - Compressed bitmaps in C#

  •        103

This is a compressed variant of the standard bitarray class. It uses a 64-bit RLE-like compression scheme. It can be used to implement bitmap indexes.The goal of word-aligned compression is not to achieve the best compression, but rather to improve query processing time. Hence, we try to save CPU cycles, maybe at the expense of storage. However, the EWAH scheme we implemented is always more efficient storage-wise than an uncompressed bitarray.

https://github.com/lemire/csharpewah

Tags
Implementation
License
Platform

   




Related Projects

javaewah - A compressed alternative to the Java BitSet class

  •    Java

The bit array data structure is implemented in Java as the BitSet class. Unfortunately, this fails to scale without compression. JavaEWAH is a word-aligned compressed variant of the Java bitset class. It uses a 64-bit run-length encoding (RLE) compression scheme. The goal of word-aligned compression is not to achieve the best compression, but rather to improve query processing time. Hence, we try to save CPU cycles, maybe at the expense of storage. However, the EWAH scheme we implemented is always more efficient storage-wise than an uncompressed bitmap (implemented in Java as the BitSet class). Unlike some alternatives, javaewah does not rely on a patented scheme.

EWAHBoolArray - A compressed bitmap class in C++.

  •    C++

The class EWAHBoolArray is a compressed bitset data structure. It supports several word sizes by a template parameter (16-bit, 32-bit, 64-bit). You should expect the 64-bit word-size to provide better performance, but higher memory usage, while a 32-bit word-size might compress a bit better, at the expense of some performance.The library also provides a basic BoolArray class which can serve as a traditional bitmap.

roaring - Roaring bitmaps in Go (golang)

  •    Go

This is a go port of the Roaring bitmap data structure.Roaring bitmaps are used by several major systems such as Apache Lucene and derivative systems such as Solr and Elasticsearch, Metamarkets' Druid, LinkedIn Pinot, Netflix Atlas, Apache Spark, OpenSearchServer, Cloud Torrent, Whoosh, Pilosa, Microsoft Visual Studio Team Services (VSTS), and eBay's Apache Kylin.

bitset - Go package implementing bitsets

  •    Go

Package bitset implements bitsets, a mapping between non-negative integers and boolean values. It should be more efficient than map[uint] bool.It provides methods for setting, clearing, flipping, and testing individual integers.

GlideBitmapPool - Glide Bitmap Pool is a memory management library for reusing the bitmap memory

  •    Java

Glide Bitmap Pool is a memory management library for reusing the bitmap memory. As it reuses bitmap memory , so no more GC calling again and again , hence smooth running application. It uses inBitmap while decoding the bitmap on the supported android versions. All the version use-cases has been handled to optimize it better. Glide Bitmap Pool can be included in any Android or Java application.


SoftwareBotany.Sunlight Word Aligned Hybrid Bit Vector Search Framework

  •    

The Software Botany Sunlight project is a search framework built using Word Aligned Hybrid Bit Vectors. Its sole purpose is to provide high performance in-memory searching of data using unknown combinations of indices. It is developed with .NET 4.0 using C#.

Pinot - A realtime distributed OLAP datastore

  •    Java

Pinot is a realtime distributed OLAP datastore, which is used at LinkedIn to deliver scalable real time analytics with low latency. It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). Pinot is designed to scale horizontally, so that it can scale to larger data sets and higher query rates as needed.

RoaringBitmap - A better compressed bitset in Java

  •    Java

Roaring bitmaps are compressed bitmaps (also called bitsets) which tend to outperform conventional compressed bitmaps such as WAH or Concise.

StyleImageView - Add styles and filters to ImageViews, Views' backgroud, Drawables and Bitmaps, with animation and bitmap output

  •    Java

This library can set the style of ImageView/View background/Drawable/Bitmap and also brightness & contrast. This library now supports 10 different styles, you can check them out below. You can enable animation, and choose to set Interpolators and Listeners. You can acquire the Bitmap of ImageView/View/Drawable after style is set. If you only need to operate ImageViews, you can use the StyleImageView class. You can specify attributes from layout file. If you are already using custom ImageViews, or you want to operate on View's background, or any Drawable or Bitmap, you can use the Styler class.

lz4-java - LZ4 compression for Java

  •    C

LZ4 compression for Java, based on Yann Collet's work available at http://code.google.com/p/lz4/. This library provides access to two compression methods that both generate a valid LZ4 stream fast scan (LZ4) and high compression (LZ4 HC). The streams produced by those 2 compression algorithms use the same compression format, are very fast to decompress and can be decompressed by the same decompressor instance.

7-Zip-zstd - 7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard

  •    C

Zstandard v1.3.7 is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder. Brotli v.1.0.7 is a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding and 2nd order context modeling, with a compression ratio comparable to the best currently available general-purpose compression methods. It is similar in speed with deflate but offers more dense compression.

Wicked Compression ASP.NET HTTP Module

  •    ASPNET

An ASP.NET HTTP Module for HTTP Compression. The primary reason for using this is if IIS Dynamic Compression is not supported in your environment. This HTTP Compression Module also supports WebResource.axd compression!

lz4 - Extremely Fast Compression algorithm

  •    C

LZ4 is lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.Speed can be tuned dynamically, selecting an "acceleration" factor which trades compression ratio for more speed up. On the other end, a high compression derivative, LZ4_HC, is also provided, trading CPU time for improved compression ratio. All versions feature the same decompression speed.

Zstandard - Fast real-time compression algorithm

  •    C

Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder. It also offers a special mode for small data, called dictionary compression, and can create dictionaries from any sample set.

zopfli - Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression

  •    C++

Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

adept-jpg-compressor - A Bash script to automate adaptive JPEG compression using common CLI tools

  •    Shell

When compressing JPEG images, the same compression level is used on the entire image. However, most JPEG images contain homogeneous and heterogeneous areas, which are varyingly well-suited for compression. Compressing heterogeneous areas in JPEGs to reduce filesize causes compression artefacts due to the lossy nature of JPEG compression. This script adaptively alters the compression level for areas within JPEGs (per-block) to achieve optimized file size while maintaining a decent visual quality. This script achieves a significantly reduced file size compared to standard tools such as cjpeg while maintaining good visual quality, as can be measured via SSIM. This is good news for the Web Performance and thus Web Developer community to achieve a great user experience on websites.

WavPack - Hybrid Lossless Wavefile Compressor

  •    C

WavPack is a completely open audio compression format providing lossless, high-quality lossy, and a unique hybrid compression mode. For version 5.0.0, several new file formats and lossless DSD audio compression were added, making WavPack a universal audio archiving solution.

lzham_codec - Lossless data compression codec with LZMA-like ratios but 1

  •    C++

LZHAM is a lossless data compression codec written in C/C++ (specifically C++03), with a compression ratio similar to LZMA but with 1.5x-8x faster decompression speed. It officially supports Linux x86/x64, Windows x86/x64, OSX, and iOS, with Android support on the way. LZHAM is a lossless (LZ based) data compression codec optimized for particularly fast decompression at very high compression ratios with a zlib compatible API. It's been developed over a period of 3 years and alpha versions have already shipped in many products. (The alpha is here: https://code.google.com/p/lzham/) LZHAM's decompressor is slower than zlib's, but generally much faster than LZMA's, with a compression ratio that is typically within a few percent of LZMA's and sometimes better.

RxFile - Rx methods to get a File and Image or Video thumbnails from a Document Provider on Android (Drive, Dropbox etc)

  •    Java

It is really easy to use it, depending on your needs it returns the appropriate file or Bitmap. You can fetch a Bitmap when you are selecting an Image or a Video. The fetched Bitmap is a thumbnail of the selected Image or Video. When you choose to select a file from Dropbox or Drive, the file is being downloaded and copied to the Library's cache folder. It then returns the File object and you have complete access over it in case you need to upload it to a server or in a similar use case.

Pilosa - Distributed bitmap index that dramatically accelerates queries across multiple, massive data sets

  •    Go

Pilosa is an open source, distributed bitmap index that dramatically accelerates continuous analysis across multiple, massive data sets. Pilosa collapses the speed and batch layer by abstracting the index from data storage, optimizing it for massive scale, and making your data instantly queryable and continuously analyzable.