Celery - Distributed Task Queue

  •        0

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

http://www.celeryproject.org/
https://github.com/celery/celery

Tags
Implementation
License
Platform

   




Related Projects

machinery - Machinery is an asynchronous task queue/job queue based on distributed message passing.


Machinery is an asynchronous task queue/job queue based on distributed message passing.First, you will need to define some tasks. Look at sample tasks in example/tasks/tasks.go to see a few examples.

tasktiger - Python task queue. Because celery is gross.


TaskTiger is a Python task queue using Redis.TaskTiger forks a subprocess for each task, This comes with several benefits: Memory leaks caused by tasks are avoided since the subprocess is terminated when the task is finished. A hard time limit can be set for each task, after which the task is killed if it hasn't completed. To ensure performance, any necessary Python modules can be preloaded in the parent process.

huey - a little task queue for python


huey is a little task queue in Python. It supports multi-process, multi-thread or greenlet task execution models. It can schedule tasks to execute at a given time, or after a given delay, schedule recurring tasks, like a crontab, retry tasks that fail automatically, task result storage.

Resque - Job queue in Ruby


Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. Queues are picked off in order of their priority. A job from a lower priority queue will only be picked off if there are no jobs for a higher priority queue available.

django-greenqueue - Asynchronous task task/job queue based on distributed message passing.


Asynchronous task task/job queue based on distributed message passing.

Jesque - Job queue in Java


Jesque is an implementation of Resque in Java. It is fully-interoperable with the Ruby and Node.js (Coffee-Resque) implementations.

goworker - Go-based background worker


goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers. goworker is compatible with Resque, so you can push your jobs with Rails and Resque, and consume them with Go in the background

delayed_job - Database backed asynchronous priority queue


Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks.

Sidekiq - Simple, efficient background processing for Ruby


Simple, efficient background processing for Ruby. Sidekiq uses threads to handle many jobs at the same time in the same process. It does not require Rails but will integrate tightly with Rails 3/4 to make background processing dead simple. Sidekiq uses multithreading so it is much more memory efficient than Resque (which forks a new process for every job).

mrq - Mr. Queue - A distributed worker task queue in Python using Redis & gevent


MRQ is a distributed task queue for python built on top of mongo, redis and gevent.MRQ was first developed at Pricing Assistant and its initial feature set matches the needs of worker queues with heterogenous jobs (IO-bound & CPU-bound, lots of small tasks & a few large ones).

Pulsar - Distributed pub-sub Messaging System from Yahoo


Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. It is horizontally scalable (Millions of independent topics and millions of messages published per second), Strong ordering and consistency guarantees, Low latency , REST API, Geo Replication and lot more.

ost - Redis based queues and workers.


Redis based queues and workers.Ost makes it easy to enqueue object ids and process them with workers.

flower - Real-time monitor and web admin for Celery distributed task queue


Real-time monitor and web admin for Celery distributed task queue

parsley - A distributed task queue for node.js, inspired by Celery


A distributed task queue for node.js, inspired by Celery

ztq - Z task queue, a redis based async task queue for python


Z task queue, a redis based async task queue for python

nomoque - Database Centric Distributed Task Queue


NoMoQue is a database-centric task queue. It's written in NodeJS and persistent data in MongoDB. It is extremely convenient when you already used these two technologies in your backend stack. You can run task producer/worker any place where you have access to your MongoDB instance.

Octobot - A distributed task queue worker designed for throughput, parallelism, and clustering.


A distributed task queue worker designed for throughput, parallelism, and clustering.

disque_jockey - A Ruby task runner for the Disque message queue


A Ruby task runner for the Disque message queue

TaskMQ - Blueprint project, message queue service with task scheduler


Blueprint project, message queue service with task scheduler

RabbitMQ - Robust messaging for applications


RabbitMQ is a messaging broker - an intermediary for messaging. It gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. It features include reliability, high availability, Clustering and Federation. RabbitMQ ships with an easy-to use management UI that allows you to monitor and control every aspect of your message broker. There are RabbitMQ clients for almost any language you can think of.