pipelines - a language for scripting data flow

  •        16

Pipelines is a language and runtime for crafting massively parallel pipelines. Unlike other languages for defining data flow, the Pipeline language requires implementation of components to be defined separately in the Python scripting language. This allows the details of implementations to be separated from the structure of the pipeline, while providing access to thousands of active libraries for machine learning, data analysis and processing. Skip to Getting Started to install the Pipeline compiler. Running the Pipeline document would safely execute each component of the pipeline in parallel and output the expected result.

https://github.com/calebwin/pipelines

Tags
Implementation
License
Platform

   




Related Projects

Apache Beam - Unified model for defining both batch and streaming data-parallel processing pipelines

  •    Java

Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.

JenkinsPipelineUnit - Framework for unit testing Jenkins pipelines

  •    Groovy

Jenkins Pipeline Unit is a testing framework for unit testing Jenkins pipelines, written in Groovy Pipeline DSL. If you use Jenkins as your CI workhorse (like us @ lesfurets.com) and you enjoy writing pipeline-as-code, you already know that pipeline code is very powerful but can get pretty complex.

lambdacd - a library to define a continuous delivery pipeline in code

  •    Clojure

I'd love to hear from you! If you have a question, a bug report or feature request please reach out. LambdaCD is built in Clojure and ClojureScript with Leiningen as a build-tool. The ./go script is your main entry-point that wraps all important development tasks. Call it without arguments to see all the options.

gaia - Build powerful pipelines in any programming language.

  •    Go

Develop powerful pipelines with the help of SDKs and simply check-in your code into a git repository. Gaia automatically clones your code repository, compiles your code to a binary and executes it on-demand. All results are streamed back and formatted to a user-friendly graphical output. Automation Engineer, DevOps, SRE, Cloud Engineer, Platform Engineer - they all have one in common: The majority of tech people are not motivated to take up this work and they are hard to recruit.

Commerce Server 2009 Orders using Pipelines in a Console Application

  •    CSharp

Using Commerce Server 2009 outside of a Web Application this project show how to create dummy Orders using Pipelines in a NON WEB CONTEXT, an example is a Console Application but you could make unit/integration/load testing, and winform applications for management purpose.


datakit - Connect processes into powerful data pipelines with a simple git-like filesystem interface

  •    OCaml

DataKit is a tool to orchestrate applications using a Git-like dataflow. It revisits the UNIX pipeline concept, with a modern twist: streams of tree-structured data instead of raw text. DataKit allows you to define complex build pipelines over version-controlled data.DataKit is currently used as the coordination layer for HyperKit, the hypervisor component of Docker for Mac and Windows, and for the DataKitCI continuous integration system.

pdpipe - Easy pipelines for pandas DataFrames.

  •    Python

Easy pipelines for pandas DataFrames. Some pipeline stages require scikit-learn; they will simply not be loaded if scikit-learn is not found on the system, and pdpipe will issue a warning. To use them you must also install scikit-learn.

Nim - Nim is a compiled, garbage-collected systems programming language with a design that focuses on efficiency, expressiveness, and elegance (in the order of priority)

  •    Nim

This repository contains the Nim compiler, Nim's stdlib, tools and documentation. For more information about Nim, including downloads and documentation for the latest release, check out Nim's website.More platforms are supported, however they are not tested regularly and they may not be as stable as the above-listed platforms.

ruby-vips - Ruby extension for the libvips image processing library.

  •    Ruby

This gem provides a Ruby binding for the libvips image processing library. Programs that use ruby-vips don't manipulate images directly, instead they create pipelines of image processing operations building on a source image. When the end of the pipe is connected to a destination, the whole pipeline executes at once, streaming the image in parallel from source to destination a section at a time.

ruby-vips - Ruby extension for the libvips image processing library.

  •    Ruby

This gem provides a Ruby binding for the libvips image processing library. Programs that use ruby-vips don't manipulate images directly, instead they create pipelines of image processing operations building on a source image. When the end of the pipe is connected to a destination, the whole pipeline executes at once, streaming the image in parallel from source to destination a section at a time.

Apache Cocoon - Web development framework: separation of concerns, component-based

  •    Java

Web development framework: separation of concerns, component-based.

data-integration - A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

  •    Python

Data integration pipelines as code: pipelines, tasks and commands are created using declarative Python code. PostgreSQL as a data processing engine.

concourse-pipeline-samples - Sample code and recipes for Concourse CI pipelines and deployments.

  •    Shell

Sample code and recipes on Concourse CI pipelines and deployments.

brigade - Event-based Scripting for Kubernetes.

  •    Go

Script simple and complex workflows using JavaScript. Chain together containers, running them in parallel or serially. Fire scripts based on times, GitHub events, Docker pushes, or any other trigger. Brigade is the tool for creating pipelines for Kubernetes.Kashti is a dashboard for your Brigade pipelines.

data-pipeline-samples - This repository hosts sample pipelines

  •    Python

AWS Data Pipeline is a web service that you can use to automate the movement and transformation of data. With AWS Data Pipeline, you can define data-driven workflows, so that tasks can be dependent on the successful completion of previous tasks. You define the parameters of your data transformations and AWS Data Pipeline enforces the logic that you've set up.##Setup 1 Get the samples by cloning this repository.

pipeline - A step by step guide on creating build and deployment pipelines for Kubernetes.

  •    

The pipeline tutorial walks you through creating an end-to-end deployment pipeline using Cloud Container Builder, GitHub, and multiple Kubernetes clusters. This tutorial will demonstrate how to propagate a Kubernetes deployment through multiple environments, each backed by a dedicated Kubernetes cluster, using a collection of Kubernetes manifest files across a set of GitHub repositories representing each environment.

pipeline-plugin - Jenkins Pipeline plugin

  •    

Building continuous delivery pipelines and similarly complex tasks in Jenkins using freestyle projects and traditional plugins can be awkward. You need to mix Parameterized Trigger, Copy Artifact, Promoted Builds, Conditional Build Step, and more just to express what should be a simple script. The Pipeline plugin (formerly known as Workflow) suite attempts to make it possible to directly write that script, what people often call a pipeline, while integrating with Jenkins features like agents and publishers. See the official documentation on jenkins.io.

data-pipeline-samples - This repository hosts sample pipelines

  •    Python

AWS Data Pipeline is a web service that you can use to automate the movement and transformation of data. With AWS Data Pipeline, you can define data-driven workflows, so that tasks can be dependent on the successful completion of previous tasks. You define the parameters of your data transformations and AWS Data Pipeline enforces the logic that you've set up. ##Setup 1 Get the samples by cloning this repository.

mleap - MLeap: Deploy Spark Pipelines to Production

  •    Scala

Deploying machine learning data pipelines and algorithms should not be a time-consuming or difficult task. MLeap allows data scientists and engineers to deploy machine learning pipelines from Spark and Scikit-learn to a portable format and execution engine. Documentation is available at mleap-docs.combust.ml.

azure-pipelines-tasks - Tasks for Azure Pipelines

  •    TypeScript

Tasks for the Azure Pipelines build system