gush - Fast and distributed workflow runner using only Sidekiq and Redis

  •        57

Gush is a parallel workflow runner using only Redis as storage and ActiveJob for scheduling and executing jobs. Gush relies on directed acyclic graphs to store dependencies, see Parallelizing Operations With Dependencies by Stephen Toub to learn more about this method.



Related Projects

Faktory - "Takin' care of business, workin' overtime"

  •    Go

At a high level, Faktory is a work server. It is the repository for background jobs within your application. Jobs have a type and a set of arguments and are placed into queues for workers to fetch and execute.You can use this server to distribute jobs to one or hundreds of machines. Jobs can be executed with any language by clients using the Faktory API to fetch a job from a queue.

Cadence - Distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way

  •    Go

Cadence is a distributed, scalable, durable, and highly available orchestration engine we developed at Uber Engineering to execute asynchronous long-running business logic in a scalable and resilient way.

verk - A job processing system that just verks!

  •    Elixir

Verk is a job processing system backed by Redis. It uses the same job definition of Sidekiq/Resque. The goal is to be able to isolate the execution of a queue of jobs as much as possible.

sidekiq-cron - Scheduler / Cron for Sidekiq jobs

  •    Ruby

A scheduling add-on for Sidekiq. Runs a thread alongside Sidekiq workers to schedule jobs at specified times (using cron notation * * * * * parsed by Rufus-Scheduler, more about cron notation.

evQueue - Job scheduler and queueing engine

  •    C++

evQueue is an open source job scheduler and queueing engine. It features an event-driven C++ engine and a PHP / MySQL web control interface which provides tasks monitoring and creation. It provides both simple task execution and complex task chaining (workflow) using an easy to use drag & drop web interface. Workflow description includes output linking to input, conditions, loops... Queues management provides an easy way for task parallelization and resource control.

sidekiq-statistic - See statistic about your workers

  •    Ruby

Improved display of statistics for your sidekiq workers and jobs. Open Statistic tab on your sidekiq page.

sidekiq-status - an extension to the sidekiq message processing to track your jobs

  •    Ruby

An extension to Sidekiq message processing to track your jobs. Inspired by resque-status and mostly copying its features, using Sidekiq's middleware. Fully compatible with ActiveJob.

sidetiq - NO LONGER MAINTAINED Recurring jobs for Sidekiq

  •    Ruby

Recurring jobs for Sidekiq. Due to changes in Sidekiq (which would necessitate an almost complete rewrite) and the move to the ActiveJob abstraction this library is no longer maintained. Please refer to alternatives instead.

go-workers - Sidekiq compatible background workers in golang

  •    Go

Sidekiq compatible background workers in golang.

rq - Simple job queues for Python

  •    Python

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It should be integrated in your web stack easily.RQ requires Redis >= 2.7.0.

SwiftQ - Distributed Task Queue

  •    Swift

SwiftQ is a distributed task queue for server side swift applications. Task queues are used as a mechanism to distribute work across machines. SwiftQ uses messages to communicate between clients and workers. In this case the message broker is Redis. SwiftQ uses the reliable queue pattern. This ensures that all tasks get processed even in the event of networking problems or consumer crashes. SwiftQ can be used for real time operations as well as delayed execution of tasks. SwiftQ can consist of multiple producers and consumers allowing for high availability and horizontal scaling. It is recommended that a separate Redis database is used to avoid conflicting name space.

workflowdirectory - A collection of user-submitted workflows for Workflow for iOS.


This repository contains user-submitted workflows for Workflow for iOS. All workflows are saved as .wflow files and located in the workflows/ directory so that they are available without needing a link. Each workflow includes a README with information about it.

sidekiq-unique-jobs - The missing unique jobs in sidekiq

  •    Ruby

See for what is required. Starting from 5.0.0 only sidekiq >= 4 is supported and support for MRI <= 2.1 is dropped. Sidekiq consists of a client and a server. The client is responsible for pushing jobs to the queue and the worker is responsible for popping jobs from the queue. Most of the uniqueness is handled when the client is pushing jobs to the queue. The client checks if it is allowed to put a job on the queue. This is probably the most common way of locking.

rspec-sidekiq - RSpec for Sidekiq

  •    Ruby

rspec-sidekiq requires sidekiq/testing by default so there is no need to include the line require "sidekiq/testing" inside your spec_helper.rb. IMPORTANT! This has the effect of not pushing enqueued jobs to Redis but to a job array to enable testing (see the FAQ & Troubleshooting Wiki page). Thus, only include gem "rspec-sidekiq" in environments where this behaviour is required, such as the test group.

Exq - Job processing library for Elixir - compatible with Resque / Sidekiq

  •    Elixir

Exq is a job processing library compatible with Resque / Sidekiq for the Elixir language.While you may reach for Sidekiq / Resque / Celery by default when writing apps in other languages, in Elixir there are some good options to consider that are already provided by the language and platform. So before adding Exq or any Redis backed queueing library to your application, make sure to get familiar with OTP and see if that is enough for your needs. Redis backed queueing libraries do add additional infrastructure complexity and also overhead due to serialization / marshalling, so make sure to evaluate whether or it is an actual need.

Resque - Job queue in Ruby

  •    Ruby

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. Queues are picked off in order of their priority. A job from a lower priority queue will only be picked off if there are no jobs for a higher priority queue available.

mrq - Mr. Queue - A distributed worker task queue in Python using Redis & gevent

  •    Python

MRQ is a distributed task queue for python built on top of mongo, redis and gevent.MRQ was first developed at Pricing Assistant and its initial feature set matches the needs of worker queues with heterogenous jobs (IO-bound & CPU-bound, lots of small tasks & a few large ones).

Sharepoint Workflow Starter


Sharepoint Workflow Starter is a visual tool for starting/stopping workflows for multiple list items. The main features this tool offers are: 1) Start/stop workflows for list items quickly with less steps than Sharepoint's web interface. 2) Start/stop workflows for all list i...

common-workflow-language - Repository for the CWL standards

  •    Python

The Common Workflow Language (CWL) is a specification for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments, from workstations to cluster, cloud, and high performance computing (HPC) environments. CWL is designed to meet the needs of data-intensive science, such as Bioinformatics, Medical Imaging, Astronomy, Physics, and Chemistry. CWL is developed by a multi-vendor working group consisting of organizations and individuals aiming to enable scientists to share data analysis workflows. The CWL project is maintained on Github and we follow the principles for collaborative open standards development. Legally, CWL is a member project of Software Freedom Conservancy and is formally managed by the elected CWL leadership team, however every-day project decisions are made by the CWL community which is open for participation by anyone.