NetDist - Job (Task) distribution system for the .net world.

  •        305

NetDist (short for ".Net Distribution") is a ready-to-use task or job distribution system for .Net and C#. The main idea behind it is, that there are job-generators (handlers) which generate jobs (for example from a database) and those jobs are processed by clients which then send the result back to the server who then processes the result. The handlers and the jobs can be added / modified on runtime without restarting the server or any client. Head over to the wiki for more information or just read the simple usage guide to get a quick introduction on how to use the whole system.

https://github.com/Roemer/NetDist

Tags
Implementation
License
Platform

   




Related Projects

Gearman - Application Framework to farm out work to other Machines

  •    C

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel by doing load balancing. It also supports to call functions between languages. It is the nervous system for how distributed processing communicates. It is fault tolerant.

Superscheduler

  •    Java

Super Scheduler is a full-featured task scheduler for all system and application job scheduling. Super Scheduler is the twin software of Super Watchdog, which is event-action task scheduler for event monitoring.

chronos - Fault tolerant job scheduler for Mesos which handles dependencies and ISO8601 based schedules

  •    Scala

Chronos is a replacement for cron. It is a distributed and fault-tolerant scheduler that runs on top of Apache Mesos that can be used for job orchestration. It supports custom Mesos executors as well as the default command executor. Thus by default, Chronos executes sh (on most systems bash) scripts. Chronos can be used to interact with systems such as Hadoop (incl. EMR), even if the Mesos agents on which execution happens do not have Hadoop installed. Included wrapper scripts allow transfering files and executing them on a remote machine in the background and using asynchronous callbacks to notify Chronos of job completion or failures. Chronos is also natively able to schedule jobs that run inside Docker containers.

sparrow - Sparrow scheduling platform (U.C. Berkeley).

  •    Python

Sparrow is a high throughput, low latency, and fault-tolerant distributed cluster scheduler. Sparrow is designed for applications that require resource allocations frequently for very short jobs, such as analytics frameworks. Sparrow schedules from a distributed set of schedulers that maintain no shared state. Instead, to schedule a job, a scheduler obtains intantaneous load information by sending probes to a subset of worker machines. The scheduler places the job's tasks on the least loaded of the probed workers. This technique allows Sparrow to schedule in milliseconds, two orders of magnitude faster than existing approaches. Sparrow also handles failures: if a scheduler fails, a client simply directs scheduling requests to an alternate scheduler. To read more about Sparrow, check out our paper.


Bee Queue - A simple, fast, robust job/task queue for Node.js, backed by Redis

  •    Javascript

A simple, fast, robust job/task queue for Node.js, backed by Redis.Bee-Queue is meant to power a distributed worker pool and was built with short, real-time jobs in mind. A web server can enqueue a job, wait for a worker process to complete it, and return its results within an HTTP request. Scaling is as simple as running more workers.

Job Scheduler

  •    Java

The Job Scheduler is used for automation - for example, process automation using the free database systems MySQL, PostgreSQL or Firebird or using SQL Server, Oracle or DB2. You can use the Job Scheduler to launch executable files and shell scripts and to run database procedures automatically. Jobs are configurable as Web Services providing interoperability with enterprise applications.

cronsun - A Distributed, Fault-Tolerant Cron-Style Job System.

  •    Go

cronsun is a distributed cron-style job system. It's similar with crontab on stand-alone *nix. The goal of this project is to make it much easier to manage jobs on lots of machines and provides high availability. cronsun is different from Azkaban, Chronos, Airflow.

OBAN - Robust job processing in Elixir, backed by modern PostgreSQL

  •    Elixir

Oban's primary goals are reliability, consistency and observability. It is fundamentally different from other background job processing tools because it retains job data for historic metrics and inspection. You can leave your application running indefinitely without worrying about jobs being lost or orphaned due to crashes.

Celery - Distributed Task Queue

  •    Python

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

Quartz - A full-featured, Java-based, In-process job scheduler

  •    Java

Quartz is a full-featured, open source job scheduling service that can be integrated with, or used along side virtually any Java EE or Java SE application - from the smallest stand-alone application to the largest e-commerce system.

disco - a Map/Reduce framework for distributed computing

  •    Erlang

Disco is a distributed map-reduce and big-data framework. Like the original framework, which was publicized by Google, Disco supports parallel computations over large data sets on an unreliable cluster of computers. This makes it a perfect tool for analyzing and processing large datasets without having to bother about difficult technical questions related to distributed computing, such as communication protocols, load balancing, locking, job scheduling or fault tolerance, all of which are taken care by Disco. Note: For installing Disco, you cannot use the zip or tar.gz packages generated by github, instead you should clone this repository.

evQueue - Job scheduler and queueing engine

  •    C++

evQueue is an open source job scheduler and queueing engine. It features an event-driven C++ engine and a PHP / MySQL web control interface which provides tasks monitoring and creation. It provides both simple task execution and complex task chaining (workflow) using an easy to use drag & drop web interface. Workflow description includes output linking to input, conditions, loops... Queues management provides an easy way for task parallelization and resource control.

OddJob

  •    Java

Oddjob is a free open source Java job scheduler. Oddjob provides 'no programming required' scheduling with a business orientated approach that brings control back to the user. Oddjob Explorer allows the user to monitor and control a job any where on the network with an intuitive visual interface.

gearman-go - This package is a Gearman API for Golang

  •    Go

The client package is used for sending jobs to the Gearman job server, and getting responses from the server.The worker package will help developers in developing Gearman worker service easily.

Bistro - Flexible distributed scheduler, a high-performance framework supporting multiple paradigms while retaining ease of configuration, management, and monitoring

  •    C++

Bistro is a toolkit for making services that schedule and execute tasks. Bistro is a toolkit for making distributed computation systems. It can schedule and run distributed tasks, including data-parallel jobs. It enforces resource constraints for worker hosts and data-access bottlenecks. It supports remote worker pools, low-latency batch scheduling, dynamic shards, and a variety of other possibilities. It has command-line and web UIs.

Quartz.NET

  •    CSharp

Quartz.NET is a full-featured, open source job scheduling system that can be used from smallest apps to large scale enterprise systems. It is a port of very propular open source Java job scheduling framework, Quartz.






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.