Avro

  •        0

Avro is a data serialization system. Avro provides: Rich data structures. A compact, fast, binary data format. A container file, to store persistent data. Remote procedure call (RPC). Simple integration with dynamic languages. Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages.

http://hadoop.apache.org/avro

Tags
Implementation
License
Platform

   




Related Projects

avro-rpc-quickstart - Apache Avro RPC Quick Start. Avro is a subproject of Apache Hadoop.


Apache Avro RPC Quick Start. Avro is a subproject of Apache Hadoop.

node-avro - AVRO deserialization/serialization utilities for node.js


AVRO deserialization/serialization utilities for node.js

simple-avro - Clojure wrapper for Avro schema and serialization


Clojure wrapper for Avro schema and serialization

avro-clj - A Clojure Wrapper for Apache avro serialization/deserialization


A Clojure Wrapper for Apache avro serialization/deserialization

avro-sorting - Examples of built-in and customizable sorting in Avro and Hadoop.


Examples of built-in and customizable sorting in Avro and Hadoop.

avro-utils - Utilities to use Avro files from Hadoop Map/Reduce jobs and Streaming


Utilities to use Avro files from Hadoop Map/Reduce jobs and Streaming

avro4s - Avro schema generation and serialization / deserialization


Avro4s is a schema/class generation and serializing/deserializing library for Avro written in Scala. The objective is to allow seamless use with Scala without the need to to write boilerplate conversions yourself, and without the runtime overhead of reflection. Hence, this is a macro based library and generates code for use with Avro at compile time.Avro4s allows us to generate schemas directly from classes in a totally straightforward way. Let's define some classes.

avrogo - Go library for the Avro serialization system


Go library for the Avro serialization system

cascading.avro - Cascading Scheme for the Apache Avro data serialization format


Cascading Scheme for the Apache Avro data serialization format

salvero - Scala case classes over 0mq, salat-avro for serialization, akka & lift actor integration


Scala case classes over 0mq, salat-avro for serialization, akka & lift actor integration

avro_playground - play with avro, hadoop, map reduce, hive


play with avro, hadoop, map reduce, hive

avro-hadoop-starter


Example MapReduce jobs in Java, Hadoop Streaming, Pig and Hive that read and/or write data in Avro format.

AvroDemo - Avro Demo code for Seattle Hadoop


Avro Demo code for Seattle Hadoop

avro-maven-plugin


Maven 2 Plugin for processing Apache Avro files. Avro is a subproject of Apache Hadoop.

camus-compressor - Camus Compressor merges files created by Camus and saves them in a compressed format


Camus Compressor merges files created by Camus and saves them in a compressed format.Camus is massively used at Allegro for dumping more than 200 Kafka topics onto HDFS. The script runs every 15 minutes and creates one file per Kafka partition which results in about 76800 small files per day. Most of the files do not exceed Hadoop block size. This is a clear Hadoop antipattern which leads to performance issues, for example extensive number of mappers in SQL queries’ executions.

avro-cpp


Modifications to Apache Avro serialization format C++ library that provide information about number of bytes encoded/decoded

python-avro


A mirror of the Python implementation of the Avro serialization and RPC framework. This repository removes the external dependencies (e.g. Snappy compression), so that they are not a requirement for the installation.

libserdes - Avro Serialization/Deserialization C/C++ library with Confluent schema-registry support


Copyright(C) 2015-2016 Confluent Inc.libserdes is a schema-based serializer/deserializer C/C++ library with support for Avro and the Confluent Platform Schema Registry.

avro-json


Utilities for converting to and from JSON from Avro records via Hadoop streaming or Hive.

byteable - Clojure protocol-based serialization interface for Hadoop.


Clojure protocol-based serialization interface for Hadoop.