Bee Queue - A simple, fast, robust job/task queue for Node.js, backed by Redis

•    Javascript

---

A simple, fast, robust job/task queue for Node.js, backed by Redis.Bee-Queue is meant to power a distributed worker pool and was built with short, real-time jobs in mind. A web server can enqueue a job, wait for a worker process to complete it, and return its results within an HTTP request. Scaling is as simple as running more workers.

redis-queue job-queue worker-queue job-scheduler bee-queue redis task job queue worker distributed system message-queue task-queue messaging

Celery - Distributed Task Queue

•    Python

---

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

task-queue task queue worker distributed messaging message-queue job-queue task-manager task-scheduler task-runner sqs-queue

react-native-queue - A React Native Job Queue

•    Javascript

---

A React Native at-least-once priority job queue / task queue backed by persistent Realm storage. Jobs will persist until completed, even if user closes and re-opens app. React Native Queue is easily integrated into OS background processes (services) so you can ensure the queue will continue to process until all jobs are completed even if app isn't in focus. It also plays well with Workers so your jobs can be thrown on the queue, then processed in dedicated worker threads for greatly improved processing performance. React Native Queue is designed to be a swiss army knife for task management in React Native. It abstracts away the many annoyances related to processing complex tasks, like durability, retry-on-failure, timeouts, chaining processes, and more. Just throw your jobs onto the queue and relax - they're covered.

react react-native queue job-queue task-queue

Resque - Job queue in Ruby

•    Ruby

---

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. Queues are picked off in order of their priority. A job from a lower priority queue will only be picked off if there are no jobs for a higher priority queue available.

job-queue task job queue worker-queue background-task workflow

Fireworq - Lightweight, high-performance, language-independent job queue system

•    Go

---

Fireworq is a lightweight, high-performance job queue system with the following abilities. It is available from ANY programming language which can talk HTTP. It works with a single binary without external dependencies. It is built on top of RDBMS (MySQL), so that jobs won't be lost even if the job queue process dies. You can apply an ordinary replication scheme to the underlying DB for the reliability of the DB itself.

job queue job-queue worker-queue background-task

Faktory - "Takin' care of business, workin' overtime"

•    Go

---

At a high level, Faktory is a work server. It is the repository for background jobs within your application. Jobs have a type and a set of arguments and are placed into queues for workers to fetch and execute.You can use this server to distribute jobs to one or hundreds of machines. Jobs can be executed with any language by clients using the Faktory API to fetch a job from a queue.

message-queue job-queue task-queue background-task

Sidekiq - Simple, efficient background processing for Ruby

•    Ruby

---

Simple, efficient background processing for Ruby. Sidekiq uses threads to handle many jobs at the same time in the same process. It does not require Rails but will integrate tightly with Rails 3/4 to make background processing dead simple. Sidekiq uses multithreading so it is much more memory efficient than Resque (which forks a new process for every job).

job-queue job task queue workflow worker worker-queue background-task

goworker - Go-based background worker

•    Go

---

goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers. goworker is compatible with Resque, so you can push your jobs with Rails and Resque, and consume them with Go in the background

job-queue job task queue workflow worker worker-queue background-task

Jesque - Job queue in Java

•    Java

---

Jesque is an implementation of Resque in Java. It is fully-interoperable with the Ruby and Node.js (Coffee-Resque) implementations.

job-queue job task queue workflow worker worker-queue background-task

Bull - Premium package for handling jobs and messages in NodeJS

•    Javascript

---

The fastest, most reliable, Redis-based queue for Node. Carefully written for rock solid stability and atomicity.

priority-queue nodejs message-queue job-queue job queue task parallel worker-queue

Machinery - Asynchronous task queue/job queue based on distributed message passing

•    Go

---

Machinery is an asynchronous task queue/job queue based on distributed message passing.

task job-queue worker-queue task-scheduler queue amqp rabbitmq redis memcached mongodb message-queue

Agenda - Lightweight job scheduling for Node.js

•    Javascript

---

Agenda is a light-weight job scheduling library for Node.js. It aims to keep its code base small. It is backed by Mongo backed persistence layer. It supports scheduling with configurable priority, concurrency, repeating and persistence of job results, Scheduling via cron or human readable syntax, Event backed job queue that you can hook into.

cron scheduler job-scheduler crontab cronjob recurring job queue task runner automated frequency interval jobs delayed job-queue task-queue

Kue - Priority job queue backed by redis, built for node.js

•    Javascript

---

Kue is a priority job queue backed by redis, built for node.js.

priority-queue worker-queue job-queue job queue worker redis

huey - a little task queue for python

•    Python

---

huey is a little task queue in Python. It supports multi-process, multi-thread or greenlet task execution models. It can schedule tasks to execute at a given time, or after a given delay, schedule recurring tasks, like a crontab, retry tasks that fail automatically, task result storage.

redis task-queue queue dank job-queue worker background-task

node-resque - It's Resque... For node! Background jobs using redis.

•    Javascript

---

new queue requires only the "queue" variable to be set. You can also pass the jobs hash to it.The configuration hash passed to new NodeResque.Worker, new NodeResque.Scheduler or new NodeResque.Queue can also take a connection option.

delayed queue resque redis work worker background job task

carrierwave_backgrounder - Offload CarrierWave's image processing and storage to a background process using Delayed Job, Resque, Sidekiq, Qu, Queue Classic or Girl Friday

•    Ruby

---

I like CarrierWave. That being said, I don't like tying up app instances waiting for images to process. This gem addresses that by offloading processing or storage/processing to a background task. We currently support Delayed Job, Resque, Sidekiq, SuckerPunch, Girl Friday, Qu, and Queue Classic.

rq - Simple job queues for Python

•    Python

---

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It should be integrated in your web stack easily.RQ requires Redis >= 2.7.0.

queue message-queue task-queue messaging job-queue

firebase-queue

•    Javascript

---

There may continue to be specific use-cases for firebase-queue, however if you're looking for a general purpose, scalable queueing system for Firebase then it is likely that building on top of Google Cloud Functions for Firebase is the ideal route. A fault-tolerant, multi-worker, multi-stage job pipeline built on the Firebase Realtime Database.

job task queue worker firebase realtime pipeline

evQueue - Job scheduler and queueing engine

•    C++

---

evQueue is an open source job scheduler and queueing engine. It features an event-driven C++ engine and a PHP / MySQL web control interface which provides tasks monitoring and creation. It provides both simple task execution and complex task chaining (workflow) using an easy to use drag & drop web interface. Workflow description includes output linking to input, conditions, loops... Queues management provides an easy way for task parallelization and resource control.

scheduler job-scheduler workflow cron high-availability cluster automation event job-queue task-queue

dingo - An easy-to-use, distributed, extensible task/job queue framework for #golang

•    Go

---

One important concept I learned from Celery and inherited in Dingo is that Caller and Worker could share the same codebase. When you send a task message in Celery, that message will not contain any source code, but only the name of the task you want to execute. This works similarly to how host names work on the internet: every worker maintains a mapping of task names to their actual functions, called the task registry.

task-queue workers