Displaying 1 to 16 from 16 results

gush - Fast and distributed workflow runner using only Sidekiq and Redis

  •    Ruby

Gush is a parallel workflow runner using only Redis as storage and ActiveJob for scheduling and executing jobs. Gush relies on directed acyclic graphs to store dependencies, see Parallelizing Operations With Dependencies by Stephen Toub to learn more about this method.

future - :rocket: R package: future: Unified Parallel and Distributed Processing in R for Everyone

  •    R

The purpose of the future package is to provide a very simple and uniform way of evaluating R expressions asynchronously using various resources available to the user. In programming, a future is an abstraction for a value that may be available at some point in the future. The state of a future can either be unresolved or resolved. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is blocked until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved depends on what strategy is used to evaluate them. For instance, a future can be resolved using a sequential strategy, which means it is resolved in the current R session. Other strategies may be to resolve futures asynchronously, for instance, by evaluating expressions in parallel on the current machine or concurrently on a compute cluster.

parallelizer - Simplifies the parallelization of function calls

  •    Go

Running multiple function calls in parallel without a timeout.Running multiple function calls in parallel with a large enough worker pool.




cucable-plugin - Maven plugin that simplifies running Cucumber Scenarios in parallel.

  •    Java

Plugin for slicing Cucumber features into single scenario feature files for parallel test execution.

ClusterRunner - ClusterRunner makes it easy to execute test-suites across your infrastructure in the fastest and most efficient way possible

  •    Python

ClusterRunner makes it easy to execute test-suites across your infrastructure in the fastest and most efficient way possible.Give it a shot by starting at our documentation site.

parmap - Easy to use map and starmap python equivalents

  •    Python

This small python module implements four functions: map and starmap, and their async versions map_async and starmap_async. In this example, Task1 uses 5 cores, while Task2 uses 3 cores. Both tasks start to compute simultaneously, and we print a message as soon as any of the tasks finishes, retreiving the result.


django-collectfaster - Parallel file copying for Django's collectstatic.

  •    Python

This package extends Django's collectstatic management command with a --faster argument that activates the parallel file copying. The speed improvement is especially helpful for remote storage backends like S3.

corebench - corebench - run your benchmarks against high performance computing servers with many CPU cores

  •    Go

Benchmark utility that's intended to exercise benchmarks and how they scale with a large number of cores. First Provider: DigitalOcean up to 48 cores currently.

taskforce - On-demand worker pools for parallelizable tasks

  •    Erlang

Authors: Guilherme Andrade (g@gandrade.net). taskforce allows you to parallelise arbitrary tasks in a controlled way.

aloisius - A Python library to create/update/delete AWS CloudFormation stacks in parallel

  •    Python

aloisius helps you to manage the life-cycle of AWS CloudFormation stacks. It allows you to use outputs from one stack as input parameters to other stacks. There are other tools which allow you to do so, like i.e. Cumulus or Ansible, but I couldn't find one which doesn't require you to use YAML or Jinja2. It is a pure Python library and it is intended to be used in inter-play with troposphere, but you can also use it with any CloudFormation JSON templates.

parallel - PARALLEL: Stata module for parallel computing

  •    Stata

Parallel lets you run Stata faster, sometimes faster than MP itself. By organizing your job in several Stata instances, parallel allows you to work with out-of-the-box parallel computing. Using the the parallel prefix, you can get faster simulations, bootstrapping, reshaping big data, etc. without having to know a thing about parallel computing. With no need of having Stata/MP installed on your computer, parallel has showed to dramatically speedup computations up to two, four, or more times depending on how many processors your computer has. See also the HTML version of the program help file.

future

  •    R

Reproducibility is part of the core design, which means that perfect, parallel random number generation (RNG) is supported regardless of the amount of chunking, type of load balancing, and future backend being used. To enable parallel RNG, use argument future.seed = TRUE. Note that, except for the built-in parallel package, none of these higher-level APIs implement their own parallel backends, but they rather enhance existing ones. The foreach framework leverages backends such as doParallel, doMC and doFuture, and the future.apply framework leverages the future ecosystem and therefore backends such as built-in parallel, future.callr, and future.batchtools.

future.callr - :rocket: R package future.callr: A Future API for Parallel Processing using 'callr'

  •    R

The future package provides a generic API for using futures in R. A future is a simple yet powerful mechanism to evaluate an R expression and retrieve its value at some point in time. Futures can be resolved in many different ways depending on which strategy is used. There are various types of synchronous and asynchronous futures to choose from in the future package. This package, future.callr, provides a type of futures that utilizes the callr package.