genie - Distributed Big Data Orchestration Service

  •        98

Genie is a federated job orchestration engine developed by Netflix. Genie provides REST-ful APIs to run a variety of big data jobs like Hadoop, Pig, Hive, Spark, Presto, Sqoop and more. It also provides APIs for managing the metadata of many distributed processing clusters and the commands and applications which run on them.See the official website to find documentation about Genie and specific documentation for various releases.

https://netflix.github.io/genie
https://github.com/Netflix/genie

Tags
Implementation
License
Platform

   




Related Projects

spring-xd - Spring XD makes it easy to solve common big data problems such as data ingestion and export, real-time analytics, and batch workflow orchestration

  •    Java

While it is possible today to build such solutions using Spring (see the Spring Data Book for details and examples), Spring XD will move well beyond the framework API level by providing an out-of-the-box executable server, a pluggable module system, a high level configuration DSL, a simple model for distributing data processing instances on or off the Hadoop cluster, and more.You can fork the repository and/or monitor JIRA to see what is going on. As always, we consider the feedback from our broad and passionate community to be one of our greatest assets.

spring-boot-security-saml-sample - Sample SAML 2.0 Service Provider with Spring Boot.

  •    Java

Currently Spring Security SAML module doesn't provide a starter for Spring Boot. Moreover, its configuration is XML-based as of this writing. The aim of this project is to explain how to develop a Service Provider (SP) which uses Spring Boot and Spring Security SAML Extension, by defining an annotation-based configuration (Java Configuration). Thymeleaf is also used as template engine.

microservices-basics-spring-boot - Basic architecture framework to create complete microservices using Spring boot and Spring cloud

  •    Java

This repository is an example of how to get Microservices going using Spring Boot, Spring Cloud, Spring OAuth 2 and Netflix OSS frameworks. It also builds on distributed system concepts and tries to provide solutions for common distributed system problem using implementations for circuit breakers, consumer driven contracts etc.


mgmt - Next generation distributed, event-driven, parallel config management!

  •    Go

Mgmt is a next generation automation tool. It has similarities to other tools in the configuration management space, but has a fast, modern, distributed systems approach. The project contains an engine and a language. Please have a look at an introductory video or blog post. Mgmt is a fairly new project. It is usable today, but not yet feature complete. With your help you'll be able to influence our design and get us to 1.0 sooner! Interested developers should read the quick start guide.

spring-boot-security-oauth2-minimal - Minimal implementation of Authorization Server, Resource Server and OAuth2 Client in Spring Boot with Spring Security and JWT

  •    Java

Spring Boot can be pretty magical, especially with regard to auto-configuration. While I'm a fan of convention-over-configuration development (I've used Grails for years), that doesn't mean it always makes sense. At least not at first, to newcomers like me. I found that some of the documentation and examples tend to gloss over "minor details" which makes it hard to grasp why you need to do something, or when the framework takes care of it for you. It took me quite a few hours of trial-and-error, tracing execution paths, reading auto-configuration classes, failing with out-of-date examples, etc.

spring-petclinic - A sample Spring-based application

  •    Java

In its default configuration, Petclinic uses an in-memory database (HSQLDB) which gets populated at startup with data. A similar setup is provided for MySql in case a persistent database configuration is needed. Note that whenever the database type is changed, the data-access.properties file needs to be updated and the mysql-connector-java artifact from the pom.xml needs to be uncommented.The Spring Petclinic master branch in the main spring-projects GitHub org is the "canonical" implementation, currently based on Spring Boot and Thymeleaf. There are quite a few forks in a special GitHub org spring-petclinic. If you have a special interest in a different technology stack that could be used to implement the Pet Clinic then please join the community there.

light-4j - A fast, lightweight and more productive microservices framework

  •    Java

Light 4j is a fast, lightweight and cloud native microservices framework. Light means lightweight, lighting fast and shed light on how to program with modern Java SE. It is 44 times faster than the most popular microservices platform Spring Boot embedded Tomcat and use only 1/5 of memory.

gs-spring-boot - Building an Application with Spring Boot :: Learn how to build an application with minimal configuration

  •    Java

This guide provides a sampling of how Spring Boot helps you accelerate and facilitate application development. As you read more Spring Getting Started guides, you will see more use cases for Spring Boot. It is meant to give you a quick taste of Spring Boot. If you want to create your own Spring Boot-based project, visit Spring Initializr, fill in your project details, pick your options, and you can download either a Maven build file, or a bundled up project as a zip file. You’ll build a simple web application with Spring Boot and add some useful services to it.

tut-spring-boot-oauth2 - Spring Boot and OAuth2:: A tutorial on "social" login and single sign on with Facebook and Github

  •    Java

This guide shows you how to build a sample app doing various things with "social login" using OAuth2 and Spring Boot. It starts with a simple, single-provider single-sign on, and works up to a self-hosted OAuth2 Authorization Server with a choice of authentication providers (Facebook or Github). The samples are all single-page apps using Spring Boot and Spring OAuth on the back end. They also all use plain jQuery on the front end, but the changes needed to convert to a different JavaScript framework or to use server side rendering would be minimal. Because one of the samples is a full OAuth2 Authorization Server we have used the shim JAR which supports bridging from Spring Boot 2.0 to the old Spring Security OAuth2 library. The simpler samples could also be implemented using the native OAuth2 support in Spring Boot security features. The configuration is very similar.

QConf - Qihoo Distributed Configuration Management System

  •    C++

We have build another interesting proect pika. Pika is a nosql compatible with redis protocol with huge storage space. You can have a try.QConf is a Distributed Configuration Management System! A better replacement of the traditional configuration file. As designed, configuration items which is constantly accessed and modified should be completely separated with application code, and QConf is where it should be.

spring-fu - Spring Fu is an experimental Kotlin micro-framework based on functional configuration intended to test new ideas for future Spring Boot releases

  •    Kotlin

Spring Fu (for functional APIs) is an experimental Kotlin micro-framework based on functional configuration intended to test ideas for upcoming Spring Boot releases. It makes it easy to create fast and lightweight Spring-powered applications with functional APIs instead of annotations. See the reference documentation for more details.

spring-cloud-dataflow - Spring Cloud Data Flow is a toolkit for building data integration and real-time data processing pipelines

  •    Java

Spring Cloud Data Flow is a toolkit for building data integration and real-time data processing pipelines.Pipelines consist of Spring Boot apps, built using the Spring Cloud Stream or Spring Cloud Task microservice frameworks.

Bistro - Flexible distributed scheduler, a high-performance framework supporting multiple paradigms while retaining ease of configuration, management, and monitoring

  •    C++

Bistro is a toolkit for making services that schedule and execute tasks. Bistro is a toolkit for making distributed computation systems. It can schedule and run distributed tasks, including data-parallel jobs. It enforces resource constraints for worker hosts and data-access bottlenecks. It supports remote worker pools, low-latency batch scheduling, dynamic shards, and a variety of other possibilities. It has command-line and web UIs.

spring-petclinic-microservices - Distributed version of Spring Petclinic built with Spring Cloud

  •    Java

This microservices branch was initially derived from AngularJS version to demonstrate how to split sample Spring application into microservices. To achieve that goal we used Spring Cloud Netflix technology stack. In order to start entire infrastructure using Docker, you have to build images by executing mvn clean install -PbuildDocker from a project root. Once images are ready, you can start them with a single command docker-compose up. Containers startup order is coordinated with wait-for-it.sh script. After starting services it takes a while for API Gateway to be in sync with service registry, so don't be scared of initial Zuul timeouts. You can track services availability using Eureka dashboard available by default at http://localhost:8761.

containerpilot - A service for autodiscovery and configuration of applications running in containers

  •    Go

An init system for cloud-native distributed applications that automates the process of service discovery, configuration, and lifecycle management inside the container, so you can focus on your apps.We can write our new applications to do that, but existing apps will need some help. We can wrap each application in a shell script that registers itself with the discovery service easily enough, but watching for changes to that service and ensuring that health checks are being made is more complicated. We can put a second process in the container, but as soon as we do that we need an init system running inside the container as well.

CFILE (Configuration File Management)

  •    Perl

CFILE is a distributed configuration file management system. Unlike many systems that try to provide a nice way to configure individual machines, CFILE provides a way to keep configuration consistant across an entrie group of machines.





We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.