workq - Job server in Go

  •        230

Workq is a job scheduling server strictly focused on simplifying job processing and streamlining coordination. It can run jobs in blocking foreground or non-blocking background mode. Workq runs as a standalone TCP server and implements a simple, text based protocol. Clients interact with Workq over a TCP socket in a request/response model with text commands. Please refer to the full protocol doc for details.

https://github.com/iamduo/workq

Tags
Implementation
License
Platform

   




Related Projects

Bee Queue - A simple, fast, robust job/task queue for Node.js, backed by Redis

  •    Javascript

A simple, fast, robust job/task queue for Node.js, backed by Redis.Bee-Queue is meant to power a distributed worker pool and was built with short, real-time jobs in mind. A web server can enqueue a job, wait for a worker process to complete it, and return its results within an HTTP request. Scaling is as simple as running more workers.

sparrow - Sparrow scheduling platform (U.C. Berkeley).

  •    Python

Sparrow is a high throughput, low latency, and fault-tolerant distributed cluster scheduler. Sparrow is designed for applications that require resource allocations frequently for very short jobs, such as analytics frameworks. Sparrow schedules from a distributed set of schedulers that maintain no shared state. Instead, to schedule a job, a scheduler obtains intantaneous load information by sending probes to a subset of worker machines. The scheduler places the job's tasks on the least loaded of the probed workers. This technique allows Sparrow to schedule in milliseconds, two orders of magnitude faster than existing approaches. Sparrow also handles failures: if a scheduler fails, a client simply directs scheduling requests to an alternate scheduler. To read more about Sparrow, check out our paper.

evQueue - Job scheduler and queueing engine

  •    C++

evQueue is an open source job scheduler and queueing engine. It features an event-driven C++ engine and a PHP / MySQL web control interface which provides tasks monitoring and creation. It provides both simple task execution and complex task chaining (workflow) using an easy to use drag & drop web interface. Workflow description includes output linking to input, conditions, loops... Queues management provides an easy way for task parallelization and resource control.

Fireworq - Lightweight, high-performance, language-independent job queue system

  •    Go

Fireworq is a lightweight, high-performance job queue system with the following abilities. It is available from ANY programming language which can talk HTTP. It works with a single binary without external dependencies. It is built on top of RDBMS (MySQL), so that jobs won't be lost even if the job queue process dies. You can apply an ordinary replication scheme to the underlying DB for the reliability of the DB itself.

Bistro - Flexible distributed scheduler, a high-performance framework supporting multiple paradigms while retaining ease of configuration, management, and monitoring

  •    C++

Bistro is a toolkit for making services that schedule and execute tasks. Bistro is a toolkit for making distributed computation systems. It can schedule and run distributed tasks, including data-parallel jobs. It enforces resource constraints for worker hosts and data-access bottlenecks. It supports remote worker pools, low-latency batch scheduling, dynamic shards, and a variety of other possibilities. It has command-line and web UIs.


chronos - Fault tolerant job scheduler for Mesos which handles dependencies and ISO8601 based schedules

  •    Scala

Chronos is a replacement for cron. It is a distributed and fault-tolerant scheduler that runs on top of Apache Mesos that can be used for job orchestration. It supports custom Mesos executors as well as the default command executor. Thus by default, Chronos executes sh (on most systems bash) scripts. Chronos can be used to interact with systems such as Hadoop (incl. EMR), even if the Mesos agents on which execution happens do not have Hadoop installed. Included wrapper scripts allow transfering files and executing them on a remote machine in the background and using asynchronous callbacks to notify Chronos of job completion or failures. Chronos is also natively able to schedule jobs that run inside Docker containers.

kq - Kafka-based Job Queue for Python

  •    Python

KQ (Kafka Queue) is a lightweight Python library which lets you queue and execute jobs asynchronously using Apache Kafka. It uses kafka-python under the hood. You may need to use sudo depending on your environment.

TaskForrest

  •    Perl

Taskforest is a simple but expressive open-source job scheduler that allows you to chain jobs/tasks and create time dependencies. It uses text config files to specify task dependencies. It has built-in RESTful web service.

qless - Queue / Pipeline Management

  •    Ruby

Qless is a powerful Redis-based job queueing system inspired by resque, but built on a collection of Lua scripts, maintained in the qless-core repo. A job is a unit of work identified by a job id or jid. A queue can contain several jobs that are scheduled to be run at a certain time, several jobs that are waiting to run, and jobs that are currently running. A worker is a process on a host, identified uniquely, that asks for jobs from the queue, performs some process associated with that job, and then marks it as complete. When it's completed, it can be put into another queue.

android-priority-jobqueue - A Job Queue specifically written for Android to easily schedule jobs (tasks) that run in the background, improving UX and application stability

  •    Java

Priority Job Queue is an implementation of a Job Queue specifically written for Android to easily schedule jobs (tasks) that run in the background, improving UX and application stability. It is written primarily with flexibility & functionality in mind. This is an ongoing project, which we will continue to add stability and performance improvements.

android-priority-jobqueue - A Job Queue specifically written for Android to easily schedule jobs (tasks) that run in the background, improving UX and application stability

  •    Java

Priority Job Queue is an implementation of a Job Queue specifically written for Android to easily schedule jobs (tasks) that run in the background, improving UX and application stability. It is written primarily with flexibility & functionality in mind. This is an ongoing project, which we will continue to add stability and performance improvements.

Faktory - "Takin' care of business, workin' overtime"

  •    Go

At a high level, Faktory is a work server. It is the repository for background jobs within your application. Jobs have a type and a set of arguments and are placed into queues for workers to fetch and execute.You can use this server to distribute jobs to one or hundreds of machines. Jobs can be executed with any language by clients using the Faktory API to fetch a job from a queue.

react-native-queue - A React Native Job Queue

  •    Javascript

A React Native at-least-once priority job queue / task queue backed by persistent Realm storage. Jobs will persist until completed, even if user closes and re-opens app. React Native Queue is easily integrated into OS background processes (services) so you can ensure the queue will continue to process until all jobs are completed even if app isn't in focus. It also plays well with Workers so your jobs can be thrown on the queue, then processed in dedicated worker threads for greatly improved processing performance. React Native Queue is designed to be a swiss army knife for task management in React Native. It abstracts away the many annoyances related to processing complex tasks, like durability, retry-on-failure, timeouts, chaining processes, and more. Just throw your jobs onto the queue and relax - they're covered.

Job Scheduler

  •    Java

The Job Scheduler is used for automation - for example, process automation using the free database systems MySQL, PostgreSQL or Firebird or using SQL Server, Oracle or DB2. You can use the Job Scheduler to launch executable files and shell scripts and to run database procedures automatically. Jobs are configurable as Web Services providing interoperability with enterprise applications.

Bull - Premium package for handling jobs and messages in NodeJS

  •    Javascript

The fastest, most reliable, Redis-based queue for Node. Carefully written for rock solid stability and atomicity.

Quartz

  •    Java

Quartz is a full-featured, open source job scheduling service that can be integrated with, or used along side virtually any Java EE or Java SE application - from the smallest stand-alone application to the largest e-commerce system.

crono - A time-based background job scheduler daemon (just like Cron) for Rails

  •    Ruby

Crono is a time-based background job scheduler daemon (just like Cron) for Ruby on Rails. Currently, there is no such thing as Ruby Cron for Rails. Well, there's Whenever but it works on top of Unix Cron, so you can't manage it from Ruby. Crono is pure Ruby. It doesn't use Unix Cron and other platform-dependent things. So you can use it on all platforms supported by Ruby. It persists job states to your database using Active Record. You have full control of jobs performing process. It's Ruby, so you can understand and modify it to fit your needs.

OddJob

  •    Java

Oddjob is a free open source Java job scheduler. Oddjob provides 'no programming required' scheduling with a business orientated approach that brings control back to the user. Oddjob Explorer allows the user to monitor and control a job any where on the network with an intuitive visual interface.