Uber's open source technology stack

  •        0
  

We aggregate and tag open source projects. We have collections of more than one million projects. Check out the projects section.



Uber, a taxi aggregation platform operates around the globe. It was started as monolith application and later re-architected as bunch of microservices, which gives them scalability. Uber uses lot of open source tools and they have contributed quite a lot projects back to the community. This article is about analyzing Uber's open source technology stack.

Programming Language:

Uber initially started with Python and later switched to NodeJS. They also use Java, Go, C++.  Java and Go helps to build scalable, high performance and concurrent application. 

 

Database:

Riak and Cassandra are used to meet high-availability, low-latency.

Schemaless, a inhouse product built on top of MySQL is used to store trips. Schemaless is an append-only sparse three dimensional persistent hash map, very similar to Google’s Bigtable. The smallest data entity in Schemaless is called a cell and is immutable; once written, it cannot be overwritten or deleted. The cell is a JSON blob referenced by a row key, a column name, and a reference key called ref key. The row key is a UUID while the column name is a string and the reference key is an integer. Schemaless is NOT open source.

JanusGraph - Scalable graph database optimized for storing and querying large graphs with billions of vertices and edges distributed across a multi-machine cluster.

LevelDB - Fast key-value storage library.

 

Caching:

Redis is used for caching

Twemproxy is used as a proxy for Redis, primarily to reduce the number of connections to the caching servers on the backend. 

 

Load balancer, Service routing and discovery

When there are bunch of microservices, they should communicate with each other, they should be discoverable and provide high availability. A single API call, may internally call couple of other services. A co-ordination and high availability is required.

Nginx - Web server and also proxy's the request to backend servers.

HAProxy is used to provide high availability

TChannel - Network multiplexing and framing protocol for RPC and it has client in various programming languages. 

Ringpop - Builds cooperation and co-ordination between application. It is resilient, client-agnostic sharding and fault tolerant

Thrift and Protobuf is used as Interface definition language (IDL) and helps in serializing data between RPC client and servers.

 

Development:

Phabricator powers a lot of internal operations, from code review to documentation to process automation.  

Github is used for issue tracking and code review for open source projects

Jenkins does continuous integration

Puppet manages system configuration.

React to build user interface for data visualization

Radium set of tools to manage inline styles on React elements

Express.js - Bedrock webserver, is built on top of the this framework

D3.js is used for visualization

Mapbox - Interactive maps using mapbox

Picasso - A powerful image downloading and caching library for Android.

Gulp - Build system 

Gradle - Build system for Android

 

Deployment

Production instances run Linux with Debian Jessie.

Docker containers on Mesos to run microservices

Apache Aurora is used for long-running services and cron jobs.

Clusto - Cluster management tool. It helps keep track of inventory of the infrastructure.

 

Logging

Logtron is used to log to disk and also push the events to Kafka.

Kafka is used to store events from various services.

ELK stack  (ELK stands for Elasticsearch, Logstash and Kibana) is used to index and analyze logs.

 

Site reliability

Nagios aalerting for monitoring, tied to an alerting system for notifications.

Grafana - Metric analytics & dashboards

Apache Storm and Spark crunch data streams into useful business metrics.

 

References:

https://github.com/uber

https://github.com/uber-common

https://github.com/uber-node/

https://eng.uber.com/

 


   





Related Articles

Stackoverflow Open Source Technology Stack

  • stack software-used stackoverflow-architecture stackoverflow-stack

Stackoverflow is one of most widely used Q&A platform. It is the default choice of developers. It has over 100 million page views in a month and its Google page rank is 60. Check out its open source technology stack, the tools used to build such a popular website. Stackoverflow uses open source projects and also contributes to it.

Read More


Open Source Technology Stack used in Magnolia CMS

  • cms software-used magnolia-cms technology-stack

Magnolia CMS is one among popular java based CMS. It has support of CMS, DMS, Wiki, Forum and lot more features. This article discusses about the open source software used to build Magnolia CMS.

Read More


Open Source Technology Stack used in Opentaps ERP

  • erp stack opentaps

opentaps Open Source ERP + CRM, is a fully integrated application suite that supports eCommerce, Customer Relationship Management, Warehouse and Inventory Management, Supply Chain Management, and Financial Management to Business Intelligence and mobility integration out-of-the-box. This article discusses about the open source software used to build Opentaps ERP.

Read More


Open Source products used in Facebook

  • facebook opensource software-used technology-stack

Facebook a leading social networking website predominantly uses open source technologies to build its application. Here is the list of open source products used and contributed by Facebook.

Read More


Open source products used in Twitter

  • twitter opensource software-used technology-stack

Twitter uses many open source products and also contributes most of the code to open source. Here is the list of open source products used by Twitter. This list does not include the projects sponsored by twitter.

Read More




Open Source Technology Stack used in WhatsApp

  • whatsapp chat messaging-app

WhatsApp is a most widely used cross-platform mobile messaging app for iPhone, BlackBerry, Android, Windows Phone and Nokia. Users can send text, images videos, audio for free. It almost replaced SMS. It changed the way people interact with each other. Lets check out the technology behind WhatsApp

Read More


Git Stack for Windows. Git Server, Client, Plugin and Code review tools for Windows.

  • stack version-control scm windows

Git is a fast, scalable, distributed revision control system originally written by Linus Torvalds. Git Server is initially written for Linux and later ported to Windows, Solaris and Mac. This article provides pointers to various Git server , Git clients, plugin and code review tools for Windows. It will be a Git complete solution stack for Windows.

Read More


Open Source Products used in Tumbler

  • tumbler blog-stack software-used stack microblogging

Tumblr is a microblogging platform that allows users to effortlessly share anything. Tumblr now hosts over 70 million blogs with over 34 billion posts to date. Below is the list of open source used in Tumbler.

Read More


Open source projects used in LinkedIn

  • linkedin opensource software-used technology-stack

LinkedIn is a social network for professionals. LinkedIn handles millions of searches as well as hundreds of thousands of updates daily. They sponsored many projects to open source. Here are the list of open source products used by LinkedIn.

Read More


Hadoop + HBase + Solr = Lily

  • hadoop hbase solr content-repository data repository

Lily currently offers an open source content repository. It is the first cloud-scalable repository for social content applications. It is built from ground up using Big Data and NOSQL technology. Its technology stack includes Hadoop, HBase and Solr.

Read More


Zimbra - Enterprise Email Server Stack

  • software-used stack mail-server

Zimbra is a Enterprise messaging and collaboration software. It is a good alternative to Microsoft Exchange server. Zimbra is a Email Server but they have not written anything related to SMTP server, rather they have integrated well know open source email server in to its package. Zimbra is a system of well connected / integrated multiple open source software which delivers enterprise quality to it. This article explains most important components which helps to build the enterprise product.

Read More


Simple Mail Stack for Linux Hosted Environment

  • email email-solution mail-stack mail

Email is the primary way of communication. Small organization which hosts their website / any other application from a third party hosting service providers may require this email stack.

Read More


Open source software used in SourceForge.NET

  • software-used sourceforge python

SourceForge.NET is most popular and widely used Forge. It helps to host software projects. It has integrated support for Wiki, Forum, Tracker and Full text search. The code base named allura, is completely built on open source stack. This article explains few important one used to build SourceForge.NET.

Read More


Complete Email Solution for SMB

  • email email-solution smb email-stack

Email is most important in any organization. Lot of business are happening via Email. This article discusses about the end to end complete Email solution for an organization. What an organization requires? Email spam filtering, Email Server, Email Archiving and Email client. If all are open source then complete solution could be achieved free of cost.

Read More


Open Source Products used in Pinterest

  • opensource software-used stack

Pinterest is a tool for collecting and organizing things you love. It is a social networking site where users could pin images and write a note for that. It is now currently serving billions of pages every month. Check out the open source products used in Pinterest.

Read More


Simple and Complete Subversion solution in Windows

  • subversion scm version-control windows

Subversion version control system is most popular and widely used. Windows users always want easy to install, easy to configure and easy to use applications. This article explains about few set of tools which used in combination will bring you complete Subversion solution in windows.

Read More


Free and Open source tools to build OpenJDK in Windows

  • java openjdk build-tools

OpenJDK is the open source implementation of Java Standard Edition. Oracle (Sun) JDK implementation is going to be based on OpenJDK. OpenJDK and Oracle JDK are almost similar. There are ready made binaries available for Linux distribution but you may need to build OpenJDK on windows.

Read More


Open Source used in Barack Obama Campaign

  • obama campaign software-used

Barack Obama election fundraising campaign was hosted online and the platform was built using open source products. The campaign stayed for 6 months and it has more than 8 million page views. This article explains the open source products used to build the platform.

Read More


Open source projects used to in Wikipedia

  • wikipedia wikimedia opensource software-used

Wikipedia is a multilingual, collaboratively edited encyclopedia. It is one of the busiest site in the world. It has more than 8 million articles and accessed by millions of users around the world. This article briefly discuss about the open source software used in Wikipedia.

Read More


How to build meta search engine

  • searchengine meta-searchengine cluster

Meta Search engine is nothing but a search engine which searches more than one search engine and combines or filters the results. Each search engine has its own proprietary ranking mechanism to rank the results. When combined the search results from all leading search engines would be more informative and useful. With less page traversals we will end up our destination.

Read More