go-erasure - Erasure coding (Reed–Solomon coding) in Go

  •        11

Go bindings for erasure coding (Reed-Solomon coding). Erasure coding is similar to RAID based parity encoding, but is more generalized and powerful. When defining an erasure code, you specify a k and m variable. m is the number of shards you wish to encode and k is the number shards it takes to recreate your original data. Hence k must be less than m and usually not equal (as that would be a pointless encoding). The real magic with erasure coding is that fact that ANY k of the m shards can recreate the original data. For example, a erasure coding scheme of k=8 and m=12 means any four of the encoded shards can be lost while the original data can still be constructed from the valid remaining eight shards.




Related Projects

reedsolomon - Reed-Solomon Erasure Coding in Go

  •    Go

Reed-Solomon Erasure Coding in Go, with speeds exceeding 1GB/s/cpu core implemented in pure Go. This is a Go port of the JavaReedSolomon library released by Backblaze, with some additional optimizations.

Schifra Reed Solomon Error Correcting Code Library


Schifra is a very robust, highly optimized and extremely configurable Reed-Solomon error correcting code library for both software and IP core based applications with implementations in C++ and VHDL.

tinyfecVPN - A VPN Designed for Lossy Links, with Build-in Forward Error Correction(FEC) Support

  •    C++

A Lightweight VPN with Build-in Forward Error Correction Support(or A Network Improving Tool which works at VPN mode). Improves your Network Quality on a High-latency Lossy Link. tinyfecVPN uses Forward Error Correction(Reed-Solomon code) to reduce packet loss rate, at the cost of additional bandwidth usage.


  •    Java

OceanStore is an architecture for a global-scale file system which uses erasure coding, cryptography, and Byzantine agreement to provide available, highly durable storage in a peer-to-peer environment.

Sia - Your decentralized private cloud

  •    Go

Sia is a new decentralized cloud storage platform that radically alters the landscape of cloud storage. By leveraging smart contracts, client-side encryption, and sophisticated redundancy (via Reed-Solomon codes), Sia allows users to safely store their data with hosts that they do not know or trust. The result is a cloud storage marketplace where hosts compete to offer the best service at the lowest price. And since there is no barrier to entry for hosts, anyone with spare storage capacity can join the network and start making money.

kcptun - A Secure Tunnel Based On KCP with N:M Multiplexing

  •    Go

kcptun maintains a single website — github.com/xtaci/kcptun. Any websites other than github.com/xtaci/kcptun are not endorsed by xtaci. kcptun won't publish anything on any social media.Download precompiled Releases.

enumeratum - A type-safe, reflection-free, powerful enumeration implementation for Scala with exhaustive pattern match warnings and helpful integrations

  •    Scala

Enumeratum is a type-safe and powerful enumeration implementation for Scala that offers exhaustive pattern match warnings, integrations with popular Scala libraries, and idiomatic usage that won't break your IDE. It aims to be similar enough to Scala's built in Enumeration to be easy-to-use and understand while offering more flexibility, type-safety (see this blog post describing erasure on Scala's Enumeration), and richer enum values without having to maintain your own collection of values. Enumeratum is published for Scala 2.10.x, 2.11.x, and 2.12.x as well as ScalaJS.

dyno - Runtime polymorphism done right

  •    C++

At this point, this library is experimental and it is a pure curiosity. No stability of interface or quality of implementation is guaranteed. Use at your own risks. Dyno solves the problem of runtime polymorphism better than vanilla C++ does. It provides a way to define interfaces that can be fulfilled non-intrusively, and it provides a fully customizable way of storing polymorphic objects and dispaching to virtual methods. It does not require inheritance, heap allocation or leaving the comfortable world of value semantics, and it can do so while outperforming vanilla C++.


  •    C

A Reed-Solomon error-correcting encoder/decoder library written in C. Useful for forward error correction applications such as streaming audio, magnetic and optical storage, or just plain amusement value. GPL license with commercial license available.

BaconQrCode - QR Code Generator for PHP

  •    PHP

BaconQrCode is a port of QR code portion of the ZXing library. It currently only features the encoder part, but could later receive the decoder part as well. As the Reed Solomon codec implementation of the ZXing library performs quite slow in PHP, it was exchanged with the implementation by Phil Karn.

UDPspeeder - A Tunnel which Improves your Network Quality on a High-latency Lossy Link by using Forward Error Correction,for All Traffics(TCP/UDP/ICMP)

  •    C++

A Tunnel which Improves your Network Quality on a High-latency Lossy Link by using Forward Error Correction. Assume your local network to your server is lossy. Just establish a VPN connection to your server with UDPspeeder + any UDP-based VPN, access your server via this VPN connection, then your connection quality will be significantly improved. With well-tuned parameters , you can easily reduce IP or UDP/ICMP packet-loss-rate to less than 0.01% . Besides reducing packet-loss-rate, UDPspeeder can also significantly improve your TCP latency and TCP single-thread download speed.

gse - Go efficient text segmentation; support english, chinese, japanese and other. Go 语言高性能分词

  •    Go

Go efficient text segmentation; support english, chinese, japanese and other. Dictionary with double array trie (Double-Array Trie) to achieve, Sender algorithm is the shortest path based on word frequency plus dynamic programming.

mux - A high performance and powerful trie based url path router for Go.

  •    Go

A high performance and powerful trie based url path router for Go. This router supports fixed and regex rules in routing pattern, and matches request method. It's optimized by trie structure for faster matching and large scale rules.

ac - Aho-Corasick Automaton with Double Array Trie (Multi-pattern substitute in go)

  •    Go

Aho-Corasick Automaton with Double Array Trie (Multi-pattern substitute in go)

hat-trie - C++ implementation of a fast and memory efficient HAT-trie

  •    C++

Trie implementation based on the "HAT-trie: A Cache-conscious Trie-based Data Structure for Strings." (Askitis Nikolas and Sinha Ranjan, 2007) paper. For now, only the pure HAT-trie has been implemented, the hybrid version may arrive later. Details regarding the HAT-trie data structure can be found here. The library provides an efficient and compact way to store a set or a map of strings by compressing the common prefixes. It also allows to search for keys that match a prefix. Note though that the default parameters of the structure are geared toward optimizing exact searches, if you do a lot of prefix searches you may want to reduce the burst threshold through the burst_threshold method.

go-json-rest - A quick and easy way to setup a RESTful JSON API

  •    Go

Go-Json-Rest is a thin layer on top of net/http that helps building RESTful JSON APIs easily. It provides fast and scalable request routing using a Trie based implementation, helpers to deal with JSON requests and responses, and middlewares for functionalities like CORS, Auth, Gzip, Status ...The recommended way of using this library in your project is to use the "vendoring" method, where this library code is copied in your repository at a specific revision. This page is a good summary of package management in Go.

Go-SCP - Go programming language secure coding practices guide

  •    Go

You can download this book in the following formats: PDF, Mobi and ePub. Go Language - Web Application Secure Coding Practices is a guide written for anyone who is using the Go Programming Language and aims to use it for web development.

ForestDB - A Fast Key-Value Storage Engine Based on Hierarchical B+-Tree Trie

  •    C++

ForestDB is a key-value storage engine developed by Couchbase Caching and Storage Team, and its main index structure is built from Hierarchical B+-Tree based Trie, called HB+-Trie. ForestDB paper has been published in IEEE Transactions on Computers.