celery-singleton - Seamlessly prevent duplicate executions of celery tasks

  •        981

Duplicate tasks clogging up your message broker? Do time based rate limits make you feel icky? Look no further! This is a baseclass for celery tasks that ensures only one instance of the task can be queued or running at any given time. Uses the task's name+arguments to determine uniqueness. That's it! Your task is a singleton and calls to do_stuff.delay() will either queue a new task or return an AsyncResult for the currently queued/running instance of the task.

https://github.com/steinitzu/celery-singleton

Tags
Implementation
License
Platform

   




Related Projects

gocelery - Celery Distributed Task Queue in Go

  •    Go

Having being involved in a number of projects migrating server from python to go, I have realized Go can help improve performance of existing python web applications. Celery distributed tasks are used heavily in many python web applications and this library allows you to implement celery workers in Go as well as being able to submit celery tasks in Go. You can also use this library as pure go distributed task queue.

flower - Real-time monitor and web admin for Celery distributed task queue

  •    Python

Flower is a web based tool for monitoring and administrating Celery clusters. Flower API enables to manage the cluster via REST API, call tasks and receive task events in real-time via WebSockets.

Celery - Distributed Task Queue

  •    Python

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

celery-once - Celery Once allows you to prevent multiple execution and queuing of celery tasks.

  •    Python

Celery Once allows you to prevent multiple execution and queuing of celery tasks. To use celery_once, your tasks need to inherit from an abstract base task called QueueOnce.

django-celery-beat - Celery Periodic Tasks backed by the Django ORM

  •    Python

This extension enables you to store the periodic task schedule in the database. The periodic tasks can be managed from the Django Admin interface, where you can create, edit and delete periodic tasks and how often they should run.


node-celery - Celery client for Node.js

  •    Javascript

Note: When using AMQP as result backend with celery prior to version 3.1.7 the result queue needs to be non durable or it will fail with a: Queue.declare: (406) PRECONDITION_FAILED. For RabbitMQ backends, the entire broker options can be passed as an object that is handed off to AMQP. This allows you to specify parameters such as SSL keyfiles, vhost, and connection timeout among others.

dingo - An easy-to-use, distributed, extensible task/job queue framework for #golang

  •    Go

One important concept I learned from Celery and inherited in Dingo is that Caller and Worker could share the same codebase. When you send a task message in Celery, that message will not contain any source code, but only the name of the task you want to execute. This works similarly to how host names work on the internet: every worker maintains a mapping of task names to their actual functions, called the task registry.

tornado-celery - Non-blocking Celery client for Tornado

  •    Python

NOTE: Currently callbacks only work with AMQP and Redis backends. To use the Redis backend, you must install tornado-redis.

celery-director - Simple and rapid framework to build workflows with Celery

  •    Python

Director is a simple and rapid framework used to manage tasks and build workflows using Celery. See how to use Director with the quickstart and guides in the documentation.

django-celery-transactions - Django transaction support for Celery tasks.

  •    Python

Django transaction support for Celery tasks.

yotaq - yotaq - Your Own Task Queue for Python

  •    Python

So you need a task queue for your Python project. Sure you could check celery, and after three months trying to understand the basic configuration options you'll be good to go. Or you could use a simpler task queue like huey or rq. Pretty good. Our python code will use dill to serialize the functions to be run and redis to store the tasks.

flask-celery-example - This repository contains the example code for my blog article Using Celery with Flask

  •    Python

This repository contains the example code for my blog article Using Celery with Flask. For details on how this all works, see my article Using Celery with Flask.

metta - An information security preparedness tool to do adversarial simulation.

  •    Python

Metta is an information security preparedness tool. This project uses Redis/Celery, python, and vagrant with virtualbox to do adversarial simulation. This allows you to test (mostly) your host based instrumentation but may also allow you to test any network based detection and controls depending on how you set up your vagrants.

tasktiger - Python task queue. Because celery is gross.

  •    Python

TaskTiger is a Python task queue using Redis.TaskTiger forks a subprocess for each task, This comes with several benefits: Memory leaks caused by tasks are avoided since the subprocess is terminated when the task is finished. A hard time limit can be set for each task, after which the task is killed if it hasn't completed. To ensure performance, any necessary Python modules can be preloaded in the parent process.

django-celery - Celery integration for Django

  •    Python

Celery integration for Django

celery-haystack - An app for integrating Celery with Haystack.

  •    Python

An app for integrating Celery with Haystack.

Go Redis - Type-safe Redis client for Golang

  •    Go

Redis client for Golang. It supports Publish /Subscribe. Transactions. Pipeline and TxPipeline. Scripting. Timeouts. Redis Sentinel. Redis Cluster. Cluster of Redis Servers without using cluster mode and Redis Sentinel. Ring. Instrumentation. Cache friendly. Rate limiting. Distributed Locks.

Exq - Job processing library for Elixir - compatible with Resque / Sidekiq

  •    Elixir

Exq is a job processing library compatible with Resque / Sidekiq for the Elixir language.While you may reach for Sidekiq / Resque / Celery by default when writing apps in other languages, in Elixir there are some good options to consider that are already provided by the language and platform. So before adding Exq or any Redis backed queueing library to your application, make sure to get familiar with OTP and see if that is enough for your needs. Redis backed queueing libraries do add additional infrastructure complexity and also overhead due to serialization / marshalling, so make sure to evaluate whether or it is an actual need.






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.