kaitai_struct - Kaitai Struct: declarative language to generate binary data parsers in C++ / C# / Go / Java / JavaScript / Lua / Perl / PHP / Python / Ruby

•    Shell

---

Kaitai Struct is a declarative language used for describing various binary data structures laid out in files or in memory: i.e. binary file formats, network stream packet formats, etc. The main idea is that a particular format is described in Kaitai Struct language only once and then can be compiled with a ksc into source files in one of the supported programming languages. These modules will include a generated code for a parser that can read described data structure from a file / stream and give access to it in a nice, easy-to-comprehend API.

reverse-engineering protocol-analyser file-format declarative-language kaitai-struct c-plus-plus graphviz

bincode - A binary encoder / decoder implementation in Rust.

•    Rust

---

A compact encoder / decoder pair that uses a binary zero-fluff encoding scheme. The size of the encoded object will be the same or smaller than the size that the object takes up in memory in a running Rust program. In addition to exposing two simple functions (one that encodes to Vec<u8>, and one that decodes from &[u8]), binary-encode exposes a Reader/Writer API that makes it work perfectly with other stream-based apis such as rust files, network streams, and the flate2-rs compression library.

encoding binary serialization

bincode - A binary encoder / decoder implementation in Rust.

•    Rust

---

A compact encoder / decoder pair that uses a binary zero-fluff encoding scheme. The size of the encoded object will be the same or smaller than the size that the object takes up in memory in a running Rust program. In addition to exposing two simple functions (one that encodes to Vec<u8>, and one that decodes from &[u8]), binary-encode exposes a Reader/Writer API that makes it work perfectly with other stream-based APIs such as Rust files, network streams, and the flate2-rs compression library.

encoding binary serialization

kaitai_struct_formats - Kaitai Struct: library of binary file formats (.ksy)

•    HTML

---

This repository constitutes a library of ready-made binary file format descriptions using Kaitai Struct language (.ksy). If you want to explore the repository, please visit Kaitai Struct format gallery — that's HTML rendition of this repository, which block diagrams, all the code compiled for all possible target languages, provided with usage examples and instructions, etc, etc.

kaitai-struct file-format protocol-parser

Buckets-JS - A complete, fully tested and documented data structure library written in pure JavaScript

•    Javascript

---

Buckets is a complete, fully tested and documented data structure library written in pure JavaScript.

data-structures collections collection linked-list dictionary map multimap stack queue set bag binary-heap priority-queue binary-search binary-search-tree tree tree-structure buckets data structure linked list heap priority binary data-structure hashmap

struct - Xcode projects on steroids

•    Ruby

---

struct is a tool for iOS and Mac developers to automate the creation and management of Xcode projects. Ever lamented over your unorganised project files? struct solves that by making your filesystem be your project structure. How it is on disk is how it is in your project. Simple.

xcode ios mac projects

Simple Binary Encoding - Simple Binary Encoding (SBE) - High Performance Message Codec

•    Java

---

SBE is an OSI layer 6 presentation for encoding and decoding binary application messages for low-latency financial applications. This repository contains the reference implementations in Java, C++, Golang, and C#. The Java and C++ SBE implementations are designed with work very efficiently with the Aeron messaging system for low-latency and high-throughput communications. The Java SBE implementation has a dependency on Agrona for its buffer implementations.

codec c-plus-plus encoder-decoder serialization

kenlm - KenLM: Faster and Smaller Language Model Queries

•    C++

---

I do development in master on https://github.com/kpu/kenlm/. Normally, it works, but I do not guarantee it will compile, give correct answers, or generate non-broken binary files. For a more stable release, get http://kheafield.com/code/kenlm.tar.gz . The website http://kheafield.com/code/kenlm/ has more documentation. If you're a decoder developer, please download the latest version from there instead of copying from another decoder.

binarytree - Python Library for Studying Binary Trees

•    Python

---

Binarytree is a Python library which provides a simple API to generate, visualize, inspect and manipulate binary trees. It allows you to skip the tedious work of setting up test data, and dive straight into practising your algorithms. Heaps and BSTs (binary search trees) are also supported. You may need to use sudo depending on your environment.

python3 python2 python-3 python-2 python-library binary-trees binary-tree interview-practice interview learning practise python-3-5 algorithm data-structures data-structure heap heaps bst binary-search-tree

csvutil - csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.

•    Go

---

Package csvutil provides fast and idiomatic mapping between CSV and Go values. This package does not provide a CSV parser itself, it is based on the Reader and Writer interfaces which are implemented by eg. std csv package. This gives a possibility of choosing any other CSV writer or reader which may be more performant.

csv fast csv-parser csvutil marshal gocsv decoder unmarshal streaming std-csv

bin2xml

•    

---

Tool written in perl to convert binary files of known formats to/from free format xml. The binary specification file allows for: primative types, structs, struct pointers and translations. The first application of the tool is for DVD IFO files.

Marshal - Marshaling the typeless wild west of [String: Any]

•    Swift

---

In Swift, we all deal with JSON, plists, and various forms of [String: Any]. Marshal believes you don't need a Ph.D. in monads or magic mirrors to deal with these in an expressive and type safe way. Marshal will help you write declarative, performant, error handled code using the power of Protocol Oriented Programming™. Unmarshaling is the process of taking an intermediary data format (the marshaled object) and tranforming it into a local representation. Think of marshaling as serialization and unmarshaling as deserialization, or coding and decoding, respectively.

json ios mac apple-watch apple-tv

bolthold - BoltHold is an embeddable NoSQL store for Go types built on BoltDB

•    Go

---

BoltHold is a simple querying and indexing layer on top of a Bolt DB instance. For a similar library built on Badger see BadgerHold. The goal is to create a simple, higher level interface on top of Bolt DB that simplifies dealing with Go Types and finding data, but exposes the underlying Bolt DB for customizing as you wish. By default the encoding used is Gob, so feel free to use the GobEncoder/Decoder interface for faster serialization. Or, alternately, you can use any serialization you want by supplying encode / decode funcs to the Options struct on Open.

boltdb bucket query-criteria nosql

pbf - A low-level, lightweight protocol buffers implementation in JavaScript.

•    Javascript

---

A low-level, fast, ultra-lightweight (3KB gzipped) JavaScript library for decoding and encoding protocol buffers, a compact binary format for structured data serialization. Works both in Node and the browser. Supports lazy decoding and detailed customization of the reading/writing code.If you use webpack as your module bundler, you can use pbf-loader to load .proto files directly. It returns a compiled module ready to be used.

protocol-buffers pbf serialization encoding decoding format binary library fast protocol buffer protobuf encoder decoder

PSON - A super efficient binary serialization format for JSON.

•    Javascript

---

PSON is a super efficient binary serialization format for JSON focused on minimal encoding size. A PSON.StaticPair contains the PSON encoder and decoder for a static (or empty) dictionary and can be shared between all connections. It's recommended for production.

net buffer serialization bytebuffer websocket webrtc json protobuf thrift

File-Spector : Binary File Analyser

•    C

---

File-Spector is a small, fast and easy to use binary file analyzer and Inspector. It allows the users to format a complete binary file structure and then use it to read any binary file that matches the specified format.

container-structure-test - validate the structure of your container images

•    Go

---

The Container Structure Tests provide a powerful framework to validate the structure of a container image. These tests can be used to check the output of commands in an image, as well as verify metadata and contents of the filesystem. Tests can be run either through a standalone binary, or through a Docker image.

eXtensible Binary Universal Protocol

•    Java

---

XBUP project is attempt to design general binary communication protocol and file format. It is based on the unlimited unary-binary encoding and strongly typed block tree structure. Providing catalog service and processing infrastructure.

GoLLRB - A Left-Leaning Red-Black (LLRB) implementation of balanced binary search trees for Google Go

•    Go

---

GoLLRB is a Left-Leaning Red-Black (LLRB) implementation of 2-3 balanced binary search trees in Go Language. As of this writing and to the best of the author's knowledge, Go still does not have a balanced binary search tree (BBST) data structure. These data structures are quite useful in a variety of cases. A BBST maintains elements in sorted order under dynamic updates (inserts and deletes) and can support various order-specific queries. Furthermore, in practice one often implements other common data structures like Priority Queues, using BBST's.

skema - 🛰 Skema provides a handy & composable way to validate / transform / purify the input data.

•    Javascript

---

Supports both async and sync flows. Skema has two working modes to support either async or sync validators, setters, etc, making it capable with much more complicated challenges. NOT only type checker. Unlike TypeScript, joi, and many others, Skema is not only a JavaScript type checker, but also a good solution for your Anti-Corruption Layer (ACL) to transform and purify the input data. And Skema could also be configured as a simple schema validator too.

validation schema model structure struct skema setter