Protobuf-for-node - Protocol Buffers for Node.JS

  •        4300

Protobuf for Node adds idiomatic protocol buffer handling to node.JS. It is actually two things in one: firstly, you can marshal protocol messages to and from Node byte buffers and send them over the wire in pure JS. Secondly, you can use protocol messages as a native interface from JS to C++ add-ons in the same process. It takes away the details of the V8 and the eio threadpool APIs and makes native extensions to Node much easier to implement. How to useTo read/write protocol buffers, protobuf for node needs the parsed representation of your protocol buffer schema (which is in protobuf format itself). The protobuf compiler spits out this format: $ vi feeds.protopackage feeds;message Feed { optional string title = 1; message Entry { optional string title = 1; } repeated Entry entry = 2;}:wq$ $PROTO/bin/protoc --descriptor_set_out=feeds.desc --include_imports feeds.protoUsing this file (read into a Buffer) you can create a 'Schema' object: var fs = require('fs'); var Schema = require('protobuf_for_node').Schema; var schema = new Schema(fs.readFileSync('feeds.desc'));A Schema object maps fully qualified protocol buffer names to type objects that know how to marshal JS objects to and from Buffers: var Feed = schema['feeds.Feed']; var aFeed = Feed.parse(aBuffer); var serialized = Feed.serialize(aFeed);When marshalling, the serializer accepts any JavaScript object (but only picks the properties defined in the schema): var serialized = Feed.serialize({ title: 'Title', ignored: 42 }); var aFeed = Feed.parse(serialized); => { title: 'Title' }Note that property names are the camel-cased version of the field names in the protocol buffer description, like for the Java version ( E.g. "optional string title_string = 1" will translate into a "titleString" JS property. Note how we're using uppercase for the type objects like for constructor functions. That's because they are: when parsing, objects are constructed using the respective constructor for the message type. This means that you are able to attach methods: Feed.prototype.numEntries = function() { return this.entry.length; }; var aFeed = Feed.parse(Feed.serialize({ entry: [{}, {}] })); aFeed.numEntries() => 2Native InterfaceProtocol buffers aren't only great for data interchange between processes - you can also use them to send data between code written in JS and C++ within the same process. Protobuf for node makes it simple to implement a native add-on without having to touch the V8 api at all: you implement a protobuf service instead. It's three easy steps: 1. Define the add-on service interface: // An example service to query pwd(3).package pwd;// Empty (=no arg) request message.message EntriesRequest {}message Entry { optional string name = 1; optional int32 uid = 2; optional int32 gid = 3; optional string home = 4; optional string shell = 5;}message EntriesResponse { repeated Entry entry = 1;}service Pwd { rpc GetEntries(EntriesRequest) returns (EntriesResponse);}... and generate the C++ code for it: proto/bin/protoc --cpp_out=. service.proto2. Implement and export the service in an add-on: extern "C" void init(v8::Handle target) { // Look Ma - no V8 api required! // Simple synchronous implementation. protobuf_for_node::ExportService( target, "pwd", new (class : public pwd::Pwd { virtual void GetEntries(google::protobuf::RpcController*, const pwd::EntriesRequest* request, pwd::EntriesResponse* response, google::protobuf::Closure* done) {\t struct passwd* pwd;\t while ((pwd = getpwent())) { pwd::Entry* e = response->add_entry(); e->set_name(pwd->pw_name); e->set_uid(pwd->pw_uid); e->set_gid(pwd->pw_gid); e->set_home(pwd->pw_dir); e->set_shell(pwd->pw_shell); }\t setpwent();\t done->Run(); } }));3. Use it from JS: // prints the user databaseputs(JSON.stringify(require('pwd').pwd.GetEntries({}), null, 2));If your service is CPU intensive, you should call it with an extra callback argument: the invocation is automatically placed on the eio thread pool and does not block node: // prints the user databaserequire('pwd').pwd.GetEntries({}, function(response) { puts(JSON.stringify(response), null, 2);});Your service is free to finish and call the "done" closure asynchronously. Note that marshalling between JS and the request and response protos necessarily happens on the JS thread. Passing tons of data will block just as much using asynchronous invocation. SpeedThe Protobuf for Node add-on relies on the protobuf C++ runtime but it does not require any generation, compilation or linkage of generated C++ code. It works reflectively and is thus able to deal with arbitrary schemas. This means however, that it is not as fast as with generated code. Simple measurements show that unmarshalling is about between 20% and 50% faster than V8's native JSON support. Calling C++ services is faster since it avoids marshalling to bytes but transfers data directly between the JS objects an



Related Projects


raspBerry+ is a web-based administration platform for Blackberry Enterprise Server for MS Exchange (BES). You can group-based activate/kill/delete/add and get status of users, their handhelds and services. With a little download-area and a comment-system


RASP's A Sneakernet Proxy; download using a thumbdrive.


RasmusDSP is an embeddable Audio/MIDI processor. It contains various filters and generators (including SoundFont 2.0 compatible synthesizer). Has a script interpreter which is used to describe instruments, route Audio/MIDI signal between processor units.


An acronym for cRoss-plAtform accesS control for Enterprise Applications. Rasea aims to become a reference in access control as a service based on the RBAC model.


Rascal, the Advanced Scientific CALculator, is a platform independent modular calculator. Based on modules for integer, doubles, strings, vectors and matrices it can be easily extended with existing C or C++ code.


RARS is the Robot Auto Racing Simulation, in which the drivers are robot programs. It is intended as a competition among programmers. It consists of a simulation of the physics of cars, a graphic display of the race, and a robot driver for each car.


This small program allows you to play a video directly from a RAR file and do so in real-time. Both VLC and MPlayer are supported video players.


RAReXtract is a Front-End for the UnRAR command line utility for Mac OS X 10.5 (Leopard). Its purpose is the rapid and convenient extraction of RAR archives with a double click.

RAR Expander

Rar Expander is a MacOSX program which extracts the files contained in single or multi-volume RAR archives. It uses the official unRAR library internally so it is fully compatible with archives produced by WinRAR.


This program uses a brute force algorithm to guess your encrypted compressed file\'s password. If you forget your encrypted file password, this program is the solution. This program can crack zip,7z and rar file passwords.


RArcInfo is a package for R ( to import data from binary Arc/Info V7.X coverages and E00 files . This will allow R users to used it as a primary GIS tool.

rar brute force shell script - rarbrute

This is rarbrute, a shell script to brute force encrypted rar files under unix and linux. A long wordlist and a paper about security in internet cafes is included.

Raquel Database System

The system will : 1. use RAQUEL (= Relational Algebra Query, Update and Executive Language) for programming, implementing Third Manifesto principles. 2. have a 'Lego-like' architecture of building blocks and plug-ins, for wider applicability.


RAPv4 is an engine for building web application with only a business description (in XML format). NEW 04/2006 : Stable 2006 release. Add new functions like mail, sms, web services, graph, map engine (GIS), Excel output, QBE... and also a beta release of


2d Scroller. Clone of Raptor: Call of the Shadows and Tyrian. Fun game written in c++ using allegro.


Lightweight XML based transformation tool written in C that builds upon expat, tidylib and XSLT to tranform authored web content (incl. Word processor generated HTML) into styled web content suitable for publication.


RapidSMS is an open-source internet and communications platform


RapidSmith is a research-based FPGA CAD tool framework written in Java for modern Xilinx FPGAs. Based on XDL, its objective is to serve as a rapid prototyping platform for research ideas and algorithms relating to low level FPGA CAD tools.

Rapidshare Mass Downloader

What this program does is bringing out human interaction while downloading files from rapidshare(without premium account). It downloads all the rapidshare links sequentially to the specified location.

rapido visual profiler

rapido is a visual profiler for linux-x86. It traces function call using the ptrace interface and displays the information collected in a nice visual flow chart. rapido does not require the re-compilation of the application.