Nomad - Easily Deploy Applications at Any Scale

  •        1308

Nomad is a cluster manager, designed for both long lived services and short lived batch processing workloads. Developers use a declarative job specification to submit work, and Nomad ensures constraints are satisfied and resource utilization is optimized by efficient task packing. Nomad supports all major operating systems and virtualized, containerized, or standalone applications.
Jobs can specify tasks which are Docker containers. Nomad will automatically run the containers on clients which have Docker installed, scale up and down based on the number of instances requested, and automatically recover from failures. Nomad is designed to be a global-scale scheduler. Multiple datacenters can be managed as part of a larger region, and jobs can be scheduled across datacenters if requested. Multiple regions join together and federate jobs making it easy to run jobs anywhere.

https://www.nomadproject.io/
https://github.com/hashicorp/nomad

Tags
Implementation
License
Platform

   




Related Projects

hashi-ui - A modern user interface for @hashicorp Consul & Nomad


For Nomad, it was quite simple, no mobile-optimized, (somewhat) feature-complete and live-updating interface existed.Today the Consul and Nomad UI exist in the same binary, but do not "cross-talk" to each other, but long term goal is to integrate them even closer, so from Nomad Job UI you can see Consul health check status for the job tasks, and vice versa be able to cross-link between two otherwise distinct systems.

gardener - Kubernetes API server extension and controller manager managing the full lifecycle of conformant Kubernetes clusters (Shoots) as a service on AWS, Azure, GCP, and OpenStack


The Gardener implements the automated management and operation of Kubernetes clusters as a service and aims to support that service on multiple Cloud providers (AWS, GCP, Azure, OpenStack). Its main principle is to use Kubernetes itself as base for its tasks. In essence, the Gardener is an extension API server along with a bundle of Kubernetes controllers which introduces new API objects in an existing Kubernetes cluster (which is called Garden cluster) in order to use them for the management of further Kubernetes clusters (which are called Shoot clusters). To do that reliably and to offer a certain quality of service, it requires to control the main components of a Kubernetes cluster (etcd, API server, controller manager, scheduler). These so-called control plane components are hosted in Kubernetes clusters themselves (which are called Seed clusters).

Kubernetes - Container Cluster Manager


Kubernetes is an open source orchestration system for Docker containers. It handles scheduling onto nodes in a compute cluster and actively manages workloads to ensure that their state matches the users declared intentions. Using the concepts of "labels" and "pods", it groups the containers which make up an application into logical units for easy management and discovery.

evQueue - Job scheduler and queueing engine


evQueue is an open source job scheduler and queueing engine. It features an event-driven C++ engine and a PHP / MySQL web control interface which provides tasks monitoring and creation. It provides both simple task execution and complex task chaining (workflow) using an easy to use drag & drop web interface. Workflow description includes output linking to input, conditions, loops... Queues management provides an easy way for task parallelization and resource control.

Cook - Fair job scheduler on Mesos for batch workloads and Spark


Core concepts is a good place to start to learn more. Check the changelog for release info.


Helix - Cluster Management Framework


Helix is a generic cluster management framework used for the automatic management of partitioned, replicated and distributed resources hosted on a cluster of nodes. It helps to perform scheduling of maintenance tasks, such as backups, garbage collection, file consolidation, index rebuilds, repartitioning of data or resources across the cluster, informing dependent systems of changes so they can react appropriately to cluster changes, throttling system tasks and changes and so on.

PiCluster - Manage Docker Containers


PiCluster is a simple way to manage Docker containers on multiple hosts. Docker Swarm not that good and Kubernetes was too difficult to install currently on ARM. PiCluster will only build and run images from Dockerfile's on the host specified in the config file. This software will work on regular x86 hardware also and is not tied to ARM.

Redisson - Redis based In-Memory Data Grid for Java


Redisson - distributed Java objects and services (Set, Multimap, SortedSet, Map, List, Queue, BlockingQueue, Deque, BlockingDeque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Executor service, Tomcat Session Manager, Scheduler service, JCache API) on top of Redis server. Rich Redis client.

Istio - An open platform to connect, manage, and secure microservices


An open platform to connect, manage, and secure microservices. Istio is an open platform for providing a uniform way to integrate microservices, manage traffic flow across microservices, enforce policies and aggregate telemetry data. Istio's control plane provides an abstraction layer over the underlying cluster management platform, such as Kubernetes, Mesos, etc.

OpenEBS - Containerized Storage for Containers


OpenEBS is containerized block storage written in Go for cloud native and other environments w/ per container (or pod) QoS SLAs, tiering and replica policies across AZs and environments, and predictable and scalable performance.

bootkube - bootkube - Launch a self-hosted Kubernetes cluster


Bootkube is a tool for launching self-hosted Kubernetes clusters.When launched, bootkube will deploy a temporary Kubernetes control-plane (api-server, scheduler, controller-manager), which operates long enough to bootstrap a replacement self-hosted control-plane.

NSQ - A realtime distributed messaging platform in Go


NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. It scales horizontally, without any centralized brokers. Built-in discovery simplifies the addition of nodes to the cluster.

kubeadm-ha - Kubernetes high availiability deploy based on kubeadm (for v1


kube-apiserver: exposes the Kubernetes API. It is the front-end for the Kubernetes control plane. It is designed to scale horizontally – that is, it scales by deploying more instances. etcd: is used as Kubernetes’ backing store. All cluster data is stored here. Always have a backup plan for etcd’s data for your Kubernetes cluster. kube-scheduler: watches newly created pods that have no node assigned, and selects a node for them to run on. kube-controller-manager: runs controllers, which are the background threads that handle routine tasks in the cluster. Logically, each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process. kubelet: is the primary node agent. It watches for pods that have been assigned to its node (either by apiserver or via local configuration file) kube-proxy: enables the Kubernetes service abstraction by maintaining network rules on the host and performing connection forwarding. keepalived cluster config a virtual IP address (192.168.20.10), this virtual IP address point to k8s-master01, k8s-master02, k8s-master03. nginx service as the load balancer of k8s-master01, k8s-master02, k8s-master03's apiserver. The other nodes kubernetes services connect the keepalived virtual ip address (192.168.20.10) and nginx exposed port (16443) to communicate with the master cluster's apiservers.

infrakit - A toolkit for creating and managing declarative, self-healing infrastructure.


InfraKit is a toolkit for infrastructure orchestration. With an emphasis on immutable infrastructure, it breaks down infrastructure automation and management processes into small, pluggable components. These components work together to actively ensure the infrastructure state matches the user's specifications. InfraKit therefore provides infrastructure support for higher-level container orchestration systems and can make your infrastructure self-managing and self-healing. In this video, InfraKit was used to build a custom linux operating system (based on linuxkit). We then deployed a cluster of virtual machine instances on a local Mac laptop using the Mac Xhyve hypervisor (HyperKit). A cluster of 3 servers booted up in seconds. Later, after the custom OS image has been updated with a new public key, InfraKit detects the change and orchestrates a rolling update of the nodes. We then deploy the same OS image to a bare-metal ARM server running on Packet.net, where the server uses custom ipxe boot directly from the localhost. It demonstrates some of the key concepts and components in InfraKit and shows how InfraKit can be used to implement an integrated workflow from custom OS image creation to cluster deployment and Day N management. The entire demo is published as a playbook, and you can create your own playbooks too.

Flocker - Container data volume manager for your Dockerized application


Flocker is an open-source Container Data Volume Manager for your Dockerized applications. By providing tools for data migrations, Flocker gives ops teams the tools they need to run containerized stateful services like databases in production. Unlike a Docker data volume which is tied to a single server, a Flocker data volume, called a dataset, is portable and can be used with any container, no matter where that container is running.

cm_api - Cloudera Manager API Client


Cloudera Manager is the market-leading management platform for CDH. As the industry’s first end-to-end management application for Apache Hadoop, Cloudera Manager sets the standard for enterprise deployment by delivering granular visibility into and control over every part of CDH – empowering operators to improve cluster performance, enhance quality of service, increase compliance and reduce administrative costs. This project contains all the source, examples and documentation you need to easily build a Cloudera Manager client in Java or Python.

tack - Terraform module for creating Kubernetes cluster running on Container Linux by CoreOS in an AWS VPC


Opinionated Terraform module for creating a Highly Available Kubernetes cluster running on Container Linux by CoreOS (any channel) in an AWS Virtual Private Cloud VPC. With prerequisites installed make all will simply spin up a default cluster; and, since it is based on Terraform, customization is much easier than CloudFormation.The default configuration includes Kubernetes add-ons: DNS, Dashboard and UI.

Apache REEF - a stdlib for Big Data


Apache REEF (Retainable Evaluator Execution Framework) is a library for developing portable applications for cluster resource managers such as Apache Hadoop YARN or Apache Mesos. For example, Microsoft Azure Stream Analytics is built on REEF and Hadoop.

CCM - A script to easily create and destroy an Apache Cassandra cluster on localhost


A script/library to create, launch and remove an Apache Cassandra cluster on localhost. The goal of ccm and ccmlib is to make it easy to create, manage and destroy a small Cassandra cluster on a local box. It is meant for testing a Cassandra cluster.

xcpu


XCPU is a suite of tools for cluster management. It includes utilities for job starting, cluster resource management, scalable distribution of boot images as well as creation and control of virtual machines in a cluster environment.