sparksql-protobuf - Read SparkSQL parquet file as RDD[Protobuf]

  •        16

This library provides utilities to work with Protobuf objects in SparkSQL. It provides a way to read parquet file written by SparkSQL back as an RDD of compatible protobuf object. It can also converts RDD of protobuf objects into DataFrame. where we need SparkContext, parquet path and protobuf class.

http://spark-packages.org/package/saurfang/sparksql-protobuf
https://github.com/saurfang/sparksql-protobuf

Tags
Implementation
License
Platform

   




Related Projects

prototool - Your Swiss Army Knife for Protocol Buffers

  •    Go

Protobuf is one of the best interface description languages out there - it's widely adopted, and after over 15 years of use, it's practically bulletproof. However, working with Protobuf and maintaining consistency across your Protobuf files can be a pain - protoc, while being a tool that has stood the test of time, is non-trivial to use, and the Protobuf community has not developed common standards with regards to stub generation. Prototool aims to solve this by making working with Protobuf much simpler. Prototool accomplishes this by downloading and calling protoc on the fly for you, handing error messages from protoc and your plugins, and using the generated FileDescriptorSets for internal functionality, as well as wrapping a few great external libraries already in the Protobuf ecosystem.

protobuf-c - Protocol Buffers implementation in C

  •    C++

This is protobuf-c, a C implementation of the Google Protocol Buffers data serialization format. It includes libprotobuf-c, a pure C library that implements protobuf encoding and decoding, and protoc-c, a code generator that converts Protocol Buffer .proto files to C descriptor code, based on the original protoc. protobuf-c formerly included an RPC implementation; that code has been split out into the protobuf-c-rpc project. protobuf-c's mailing list is hosted on a Google Groups forum. Subscribe by sending an email to protobuf-c+subscribe@googlegroups.com.

protobuf-swift - Google ProtocolBuffers for Apple Swift

  •    Swift

An implementation of Protocol Buffers in Swift. Protocol Buffers are a way of encoding structured data in an efficient yet extensible format. This project is based on an implementation of Protocol Buffers from Google. See the Google protobuf project for more information.

protobuf - Protocol Buffers - Google's data interchange format

  •    C++

Copyright 2008 Google Inc. Protocol Buffers (a.k.a., protobuf) are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. You can find protobuf's documentation on the Google Developers site.

ProtoBufferTools

  •    

ProtobufferTools is free and open source protobuf editor, it's can simply help you to serialize your protobuf data contract class to xml, and pack and unpack th


Protobuf GUI

  •    

GUI for Google Protobuf

Protocol Buffers - Google's data interchange format

  •    C++

Protocol buffers (a.k.a., protobuf) are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

php-protobuf - PHP Protobuf - Google's Protocol Buffers for PHP

  •    PHP

Protocol Buffers are a way of encoding structured data in an efficient yet extensible format. It might be used in file formats and RPC protocols.PHP Protobuf is Google's Protocol Buffers implementation for PHP with a goal to provide high performance, including a protoc plugin to generate PHP classes from .proto files. The heavy-lifting (a parsing and a serialization) is done by a PHP extension.

protobuf-gradle-plugin - Protobuf Plugin for Gradle

  •    Groovy

❗️ Please read release notes before upgrading the plugin.For more information about the Protobuf Compiler, please refer to Google Developers Site.

protobuf - Protocol Buffers for Go with Gadgets

  •    Go

gogoprotobuf is a fork of golang/protobuf with extra code generation features.Please let us know if you are using gogoprotobuf by posting on our GoogleGroup.

swift-protobuf - Plugin and runtime library for using protobuf with Swift

  •    Swift

Apple's Swift programming language is a perfect complement to Google's Protocol Buffer ("protobuf") serialization technology. They both emphasize high performance and programmer safety. This project provides both the command-line program that adds Swift code generation to Google's protoc and the runtime library that is necessary for using the generated code. After using the protoc plugin to generate Swift code from your .proto files, you will need to add this library to your project.

rust-protobuf - Rust implementation of Google protocol buffers

  •    Rust

Protobuf implementation in Rust. 1.7+ 2.* versions should follow semver conventions. However, if you are developing critical library it's probably better to specify version as ~2.0 (>= 2.0 and <= 2.1), but not as 2.0 (>= 2.0 and <= 3.0).

pyrobuf - A Cython alternative to Google's Python Protobuf library

  •    Python

Pyrobuf is an alternative to Google's Python Protobuf library. It generates lightning-fast Cython code that's 2-4x faster than Google's Python Protobuf library using their C++ backend and 20-40x faster than Google's pure-python implementation.

ScalaBuff - the scala protocol buffers (protobuf) compiler

  •    Scala

ScalaBuff is a Scala Protocol Buffers (protobuf) compiler. It takes .proto files and outputs valid Scala classes that can be used by your code to receive or send protobuf messages. Both the ScalaBuff generator and the generated Scala classes depend on Google's Java runtime for Protocol Buffers, which is provided with ScalaBuff.

magellan - Geo Spatial Data Analytics on Spark

  •    Scala

Magellan is a distributed execution engine for geospatial analytics on big data. It is implemented on top of Apache Spark and deeply leverages modern database techniques like efficient data layout, code generation and query optimization in order to optimize geospatial queries. The application developer writes standard sql or data frame queries to evaluate geometric expressions while the execution engine takes care of efficiently laying data out in memory during query processing, picking the right query plan, optimizing the query execution with cheap and efficient spatial indices while presenting a declarative abstraction to the developer.

LearningSpark - Scala examples for learning to use Spark

  •    Scala

This project contains snippets of Scala code for illustrating various Apache Spark concepts. It is intended to help you get started with learning Apache Spark (as a Scala programmer) by providing a super easy on-ramp that doesn't involve Unix, cluster configuration, building from sources or installing Hadoop. Many of these activities will be necessary later in your learning experience, after you've used these examples to achieve basic familiarity. It is intended to accompany a number of posts on the blog A River of Bytes.

spark - .NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

  •    CSharp

.NET for Apache Spark provides high performance APIs for using Apache Spark from C# and F#. With these .NET APIs, you can access the most popular Dataframe and SparkSQL aspects of Apache Spark, for working with structured data, and Spark Structured Streaming, for working with streaming data. .NET for Apache Spark is compliant with .NET Standard - a formal specification of .NET APIs that are common across .NET implementations. This means you can use .NET for Apache Spark anywhere you write .NET code allowing you to reuse all the knowledge, skills, code, and libraries you already have as a .NET developer.

openrtb - OpenRTB model for Java and other languages via protobuf; Helper OpenRTB libraries for Java including JSON serialization

  •    Java

OpenRTB model for Java and other languages via protobuf; Helper OpenRTB libraries for Java including JSON serialization

rejoiner - Generates a unified GraphQL schema from gRPC microservices and other Protobuf sources

  •    Java

These features are actively being developed.SchemaModule is a Guice module that is used to generate parts of a GraphQL schema. It finds methods and fields that have Rejoiner annotations when it's instantiated. It then looks at the parameters and return type of these methods in order to generate the appropriate GraphQL schema. Examples of queries, mutations, and schema modifications are presented below.