Jepsen - A framework for distributed systems verification, with fault injection

  •        86

Jepsen is a Clojure library. A test is a Clojure program which uses the Jepsen library to set up a distributed system, run a bunch of operations against that system, and verify that the history of those operations makes sense. Jepsen has been used to verify everything from eventually-consistent commutative databases to linearizable coordination systems to distributed task schedulers. It can also generate graphs of performance and availability, helping you characterize how a system responds to different faults.

A Jepsen test runs as a Clojure program on a control node. That program uses SSH to log into a bunch of db nodes, where it sets up the distributed system you're going to test using the test's pluggable os and db.

Once the system is running, the control node spins up a set of logically single-threaded processes, each with its own client for the distributed system. A generator generates new operations for each process to perform. Processes then apply those operations to the system using their clients. The start and end of each operation is recorded in a history. While performing operations, a special nemesis process introduces faults into the system--also scheduled by the generator.

Finally, the DB and OS are torn down. Jepsen uses a checker to analyze the test's history for correctness, and to generate reports, graphs, etc. The test, history, analysis, and any supplementary results are written to the filesystem under store/// for later review. Symlinks to the latest results are maintained at each level for convenience.



Related Projects

testing-distributed-systems - Curated list of resources on testing distributed systems

  •    HTML

List of resources on testing distributed systems curated by Andrey Satarin (@asatarin). Colin Scott shares his viewpoint from academia on testing distributed systems, specifically regression testing for correctness and performance bugs.

Linux Desktop Testing Project

  •    C

Linux Desktop Testing Project is aimed at producing high quality test automation framework (using GNOME / Python) and cutting-edge tools that can be used to test Linux Desktop and improve it. It uses the Accessibility libraries to poke through the application's user interface. We strive to help in building a quality desktop.

Grinder - Java Load Testing Framework

  •    Java

The Grinder is a Java load testing framework that makes it easy to run a distributed test using many load injector machines. Load test anything that has a Java API. This includes common cases such as HTTP web servers, SOAP and REST web services, and application servers (CORBA, RMI, JMS, EJBs), as well as custom protocols.

pact_broker - Enables your consumer driven contracts workflow

  •    Ruby

The Pact Broker is an application for sharing for consumer driven contracts and verification results. It is optimised for use with "pacts" (contracts created by the Pact framework), but can be used for any type of contract that can be serialized to JSON. Contract testing is an alternative approach to traditional integration testing that gives you tests that are quicker to execute, and that are more maintainable at scale. One down side of the approach is that the important information that would be available all in one place at the end of an integration test suite execution (that is, the version numbers of all the applications that were tested together, and whether the tests passed or failed) is now spread out across many different builds. The Pact Broker is a tool that brings all this information back together again, and allows you to deploy with safety.

locust - Scalable user load testing tool written in Python

  •    Python

Locust is an easy-to-use, distributed, user load testing tool. It is intended for load-testing web sites (or other systems) and figuring out how many concurrent users a system can handle. The idea is that during a test, a swarm of locusts will attack your website. The behavior of each locust (or test user if you will) is defined by you and the swarming process is monitored from a web UI in real-time. This will help you battle test and identify bottlenecks in your code before letting real users in.

Selenium - Web app testing tool

  •    Java

Selenium is a suite of tools such as Selenium IDE, Selenium Remote Control and Selenium Grid to test the web application. Selenium IDE is an integrated development environment for Selenium scripts. It is implemented as a Firefox extension, and allows you to record, edit, and debug tests. It supports record and playback.

Buildbot - Continuous Integration Framework

  •    Python

Buildbot is a continuous integration system designed to automate the build/test cycle. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. It supports arbitrary build processes, and is not limited to common build processes for particular languages. Buildbot has a variety of status-reporting tools to get information about builds in a timely manner.

rDSN - Robust Distributed System Nucleus (rDSN) is an open framework for quickly building and managing high performance and robust distributed systems

  •    C++

Robust Distributed System Nucleus (rDSN) is a framework for quickly building robust distributed systems. It has a microkernel for pluggable components, including applications, distributed frameworks, devops tools, and local runtime/resource providers, enabling their independent development and seamless integration. The project was originally developed for Microsoft Bing, and now has been adopted in production both inside and outside Microsoft.The core of rDSN is a service kernel with which we can develop (via Service API and Tool API) and plugin lots of different application, framework, tool, and local runtime modules, so that they can seamlessly benefit each other. Here is an incomplete list of the pluggable modules.

Tsung - Distributed Load Testing Tool

  •    Erlang

Tsung is an open-source multi-protocol distributed load testing tool. It can be used to stress HTTP, WebDAV, SOAP, PostgreSQL, MySQL, LDAP and Jabber/XMPP servers. The purpose of Tsung is to simulate users in order to test the scalability and performance of IP based client/server applications. You can use it to do load and stress testing of your servers. Many protocols have been implemented and tested, and it can be easily extended.

Mockito - Most popular Mocking framework for unit tests written in Java

  •    Java

Mockito is a mocking framework that tastes really good. It lets you write beautiful tests with a clean & simple API. Mockito doesn’t give you hangover because the tests are very readable and they produce clean verification errors.

muxy - Chaos engineering tool for simulating real-world distributed system failures

  •    Go

Proxy for simulating real-world distributed system failures to improve resilience in your applications.Muxy is a proxy that mucks with your system and application context, operating at Layers 4, 5 and 7, allowing you to simulate common failure scenarios from the perspective of an application under test; such as an API or a web application.

jepsen - Call Me Maybe: simulating network partitions in DBs

  •    Ruby

Call Me Maybe: simulating network partitions in DBs

Splinter - Python test framework for web applications

  •    Python

Splinter is an open source tool for testing web applications using Python. It lets you automate browser actions, such as visiting URLs and interacting with their items. Splinter is an abstraction layer on top of existing browser automation tools such as Selenium, PhantomJS and zope.testbrowser. It has a high-level API that makes it easy to write automated tests of web applications. It has drivers for Chrome and Firefox for browser-based testing, and zope.testbrowser and PhantomJS for headless testing.

vunit - VUnit is a unit testing framework for VHDL/SystemVerilog

  •    VHDL

VUnit is an open source unit testing framework for VHDL/SystemVerilog released under the terms of Mozilla Public License, v. 2.0. It features the functionality needed to realize continuous and automated testing of your HDL code. VUnit doesn't replace but rather complements traditional testing methodologies by supporting a "test early and often" approach through automation. Contributing in the form of code, feedback, ideas or bug reports are welcome. Read our contribution guide to get started.

OpenSTA - Open Systems Testing Architecture

  •    C++

OpenSTA is a distributed software testing architecture designed around CORBA. The current toolset has the capability of performing scripted HTTP and HTTPS heavy load tests with performance measurements from Win32 platforms. Results and statistics are collected during test runs by a variety of automatic and user controlled mechanisms. These can include scripted timers, SNMP data, Windows Performance Monitor stats and HTTP results & timings.

Catch2 - C++ Automated Test Cases in a Header

  •    C++

Catch2 stands for C++ Automated Test Cases in a Header and is a multi-paradigm test framework for C++. which also supports Objective-C (and maybe C). It is primarily distributed as a single header file, although certain extensions may require additional headers.

xUnit - Unit testing tool for the .NET Framework

  •    CSharp is a free, open source, community-focused unit testing tool for the .NET Framework. Written by the original inventor of NUnit v2, is the latest technology for unit testing C#, F#, VB.NET and other .NET languages. works with ReSharper, CodeRush, TestDriven.NET and Xamarin. It is part of the .NET Foundation, and operates under their code of conduct.

CUnit - A Unit Testing Framework for C

  •    C

CUnit is a lightweight system for writing, administering, and running unit tests in C. It provides C programmers a basic testing functionality with a flexible variety of user interfaces. CUnit is built as a static library which is linked with the user's testing code. It uses a simple framework for building test structures, and provides a rich set of assertions for testing common data types.

dredd - Language-agnostic HTTP API Testing Framework

  •    Javascript

Dredd is a language-agnostic command-line tool for validating API description document against backend implementation of the API. Dredd reads your API description and step by step validates whether your API implementation replies with responses as they are described in the documentation.

toxiproxy - :alarm_clock: :fire: A TCP proxy to simulate network and system conditions for chaos and resiliency testing

  •    Go

Toxiproxy is a framework for simulating network conditions. It's made specifically to work in testing, CI and development environments, supporting deterministic tampering with connections, but with support for randomized chaos and customization. Toxiproxy is the tool you need to prove with tests that your application doesn't have single points of failure. We've been successfully using it in all development and test environments at Shopify since October, 2014. See our blog post on resiliency for more information.Toxiproxy usage consists of two parts. A TCP proxy written in Go (what this repository contains) and a client communicating with the proxy over HTTP. You configure your application to make all test connections go through Toxiproxy and can then manipulate their health via HTTP. See Usage below on how to set up your project.