airflow-docker - Apache Airflow Docker Image.

  •        349

This repository contains Dockerfile of apache-airflow for Docker's automated build published to the public Docker Hub Registry. Pull the image from the Docker repository.

https://hub.docker.com/r/shinyorke/airflow/
https://github.com/Shinichi-Nakagawa/airflow-docker

Tags
Implementation
License
Platform

   




Related Projects

incubator-airflow - Apache Airflow (Incubating)

  •    Python

NOTE: The transition from 1.8.0 (or before) to 1.8.1 (or after) requires uninstalling Airflow before installing the new version. The package name was changed from airflow to apache-airflow as of version 1.8.1.Airflow is a platform to programmatically author, schedule, and monitor workflows.

pipeline - PipelineAI: Real-Time Enterprise AI Platform

  •    HTML

Each model is built into a separate Docker image with the appropriate Python, C++, and Java/Scala Runtime Libraries for training or prediction. Use the same Docker Image from Local Laptop to Production to avoid dependency surprises.

airflow - Apache Airflow

  •    Python

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows. When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.

airflow - AirFlow is a system to programmatically author, schedule and monitor data pipelines.

  •    Javascript

AirFlow is a system to programmatically author, schedule and monitor data pipelines.

cronsun - A Distributed, Fault-Tolerant Cron-Style Job System.

  •    Go

cronsun is a distributed cron-style job system. It's similar with crontab on stand-alone *nix. The goal of this project is to make it much easier to manage jobs on lots of machines and provides high availability. cronsun is different from Azkaban, Chronos, Airflow.


ofelia - A docker job scheduler (aka. crontab for docker)

  •    Go

Ofelia is a modern and low footprint job scheduler for docker environments, built on Go. Ofelia aims to be a replacement for the old fashioned cron. It has been a long time since cron was released, actually more than 28 years. The world has changed a lot and especially since the Docker revolution. Vixie's cron works great but it's not extensible and it's hard to debug when something goes wrong.

habitus - A Build Flow Tool for Docker

  •    Go

Habitus adds workflows to Docker build. This means you can create a chain of builds to generate your final Docker image based on a workflow. This is particularly useful if your code is in compiled languages like Java or Go or if you need to use secrets like SSH keys during the build.Habitus is a standalone build flow tool for Docker. It’s a command line tool that builds Docker images based on their Dockerfile and a build.yml.

docker-slim - DockerSlim (docker-slim): Optimize and secure your Docker containers (free and open source)

  •    Go

Creating small containers requires a lot of voodoo magic and it can be pretty painful. You shouldn't have to throw away your tools and your workflow to have skinny containers. Using Docker should be easy.docker-slim is a magic diet pill for your containers :) It will use static and dynamic analysis to create a skinny container for your app.

habitus - A Build Flow Tool for Docker

  •    Go

Habitus adds workflows to Docker build. This means you can create a chain of builds to generate your final Docker image based on a workflow. This is particularly useful if your code is in compiled languages like Java or Go or if you need to use secrets like SSH keys during the build. Habitus is a standalone build flow tool for Docker. It’s a command line tool that builds Docker images based on their Dockerfile and a build.yml.

Nomad - Easily Deploy Applications at Any Scale

  •    Go

Nomad is a cluster manager, designed for both long lived services and short lived batch processing workloads. Developers use a declarative job specification to submit work, and Nomad ensures constraints are satisfied and resource utilization is optimized by efficient task packing. Nomad supports all major operating systems and virtualized, containerized, or standalone applications.

binci - :whale: Containerize your development workflow.

  •    Javascript

Binci is a utility that allows you to easily containerize your development workflow using Docker. Simply put, it's like having a cleanroom for all of your development processes which contain services (like databases) without needing to setup and maintain these environments manually. The best way to install Binci and keep it updated is through NPM, included with Node.js, or Yarn.

source-to-image - A tool for building/building artifacts from source and injecting into docker images

  •    Go

Source-to-Image (S2I) is a toolkit and workflow for building reproducible Docker images from source code. S2I produces ready-to-run images by injecting source code into a Docker container and letting the container prepare that source code for execution. By creating self-assembling builder images, you can version and control your build environments exactly like you use Docker images to version your runtime environments.For a deep dive on S2I you can view this presentation.

ddev - DDEV-Local: a local PHP development environment system

  •    Go

ddev is an open source tool that makes it simple to get local PHP development environments up and running in minutes. It's powerful and flexible as a result of its per-project environment configurations, which can be extended, version controlled, and shared. In short, ddev aims to allow development teams to use Docker in their workflow without the complexities of bespoke configuration. Having trouble? See our support options below. You might have trouble if another local development tool is already using port 80 or 443.

captain - Captain - Convert your Git workflow to Docker :whale: containers

  •    Go

Define your workflow in the captain.yaml and use captain to your Continuous Delivery service to create containers for each commit, test them and push them to your registry only when tests passes. From the other side, you can now pull the feature branch you want to test, or create distribution channels (such as 'alpha', 'beta', 'stable') using git tags that are propagated to container tags.

chronos - Fault tolerant job scheduler for Mesos which handles dependencies and ISO8601 based schedules

  •    Scala

Chronos is a replacement for cron. It is a distributed and fault-tolerant scheduler that runs on top of Apache Mesos that can be used for job orchestration. It supports custom Mesos executors as well as the default command executor. Thus by default, Chronos executes sh (on most systems bash) scripts. Chronos can be used to interact with systems such as Hadoop (incl. EMR), even if the Mesos agents on which execution happens do not have Hadoop installed. Included wrapper scripts allow transfering files and executing them on a remote machine in the background and using asynchronous callbacks to notify Chronos of job completion or failures. Chronos is also natively able to schedule jobs that run inside Docker containers.

evQueue - Job scheduler and queueing engine

  •    C++

evQueue is an open source job scheduler and queueing engine. It features an event-driven C++ engine and a PHP / MySQL web control interface which provides tasks monitoring and creation. It provides both simple task execution and complex task chaining (workflow) using an easy to use drag & drop web interface. Workflow description includes output linking to input, conditions, loops... Queues management provides an easy way for task parallelization and resource control.

awesome-docker - :whale: A curated list of Docker resources and projects

  •    Javascript

A curated list of Docker resources and projects Inspired by @sindresorhus' awesome and improved by these amazing contributors. The creators and maintainers of this list do not receive any form of payment to accept a change made by any contributor. This page is not an official Docker product in any way. It is a list of links to projects and is maintained by volunteers. Everybody is welcome to contribute. The goal of this repo is to index open-source projects, not to advertise for profit.

data-integration - A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

  •    Python

Data integration pipelines as code: pipelines, tasks and commands are created using declarative Python code. PostgreSQL as a data processing engine.

Agile_Data_Code_2 - Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

  •    Jupyter

Like my work? I am Principal Consultant at Data Syndrome, a consultancy offering assistance and training with building full-stack analytics products, applications and systems. Find us on the web at datasyndrome.com. There is now a video course using code from chapter 8, Realtime Predictive Analytics with Kafka, PySpark, Spark MLlib and Spark Streaming. Check it out now at datasyndrome.com/video.