Apache NiFi - An easy to use, powerful, and reliable system to process and distribute data

We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.

Apache NiFi - An easy to use, powerful, and reliable system to process and distribute data

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Data flow can be tracked and modified at run time. It automates the movement of data between disparate data sources and systems, making data ingestion fast, easy and secure. The project was created by the United States National Security Agency (NSA).
It is data source agnostic, supporting disparate and distributed sources of differing formats, schemas, protocols, speeds and sizes such as machines, geo location devices, click streams, files, social feeds, log files and videos and more.

https://nifi.apache.org/
https://github.com/apache/nifi

Tags	dataflow data-management data-collector enterprise-integration-patterns log-management
Implementation	Java
License	Apache
Platform	OS-Independent

Related Projects

Fluentd - Unified Logging Layer

Ruby

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure. Fluentd can be used to tail access/error logs and transport them reliably to remote systems. It can "grep" for events and send out alerts. It can function as middleware to enable asynchronous, scalable logging for user action events.

log-collector syslog log-analysis log log-management log-aggregator log-search data-collector

Flume - Log management using HDFS

Java

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

log-collector syslog log-analysis log log-management distributed

kylo - Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on Apache Hadoop and Spark

Java

Kylo

hadoop spark nifi kylo data-lake teradata

opentaps = ERP + CRM

Java

opentaps Open Source ERP + CRM, is a fully integrated application suite that supports eCommerce, Customer Relationship Management, Warehouse and Inventory Management, Supply Chain Management, and Financial Management to Business Intelligence and mobility integration out-of-the-box.

erp enterprise-resource-planning bpm business-process

Open Distro for Elasticsearch - Elasticsearch enhanced with enterprise security, alerting, SQL, and more

Java

Open Distro for Elasticsearch is an Apache 2.0-licensed distribution of Elasticsearch enhanced with Enterprise Security, Alerting, SQL, Index Management, k-Nearest Neighbor Search, Performance Analyzer and more.

elastic-search search-engine searchengine aggregations real-time enterprise-search

Pimcore - The leading enterprise platform for CMS/CMF, MDM, PIM, DAM and E-Commerce

The Pimcore PIM module provides the world's only enterprise PIM without software cost, centralizing all your product data independently from the channel. Pimcore makes it fast and easy to manage product information. It can integrate, consolidate, and manage any type and any amount of digital data. Provide the data to any channel such as commerce, mobile apps, print, digital signage.

cms ecommerce ecommerce-platform master-data-management dam wcms digital-platform data-management product-management product-information-management cms-framework mdm online-shop online-store digital-asset-management

SiteWhere - The Open Platform for Internet of Things (IoT)

Java

SiteWhere is an open source platform for capturing, storing, integrating, and analyzing data from IoT devices. SiteWhere is a multi-tenant, application enablement platform for the Internet of Things (IoT) providing device management, complex event processing (CEP) and integration through a modern, scalable architecture. SiteWhere provides REST APIs for all system functionality.

internet-of-things device-management iot iot-platform mqtt iot-framework arduino raspberry-pi

Harbor - An enterprise-class container registry server based on Docker Distribution

Project Harbor is an enterprise-class registry server that stores and distributes Docker images. It extends the open source Docker Distribution by adding the functionalities usually required by an enterprise, such as security, identity and management. As an enterprise private registry, Harbor offers better performance and security.

docker docker-registry registry-server private-registry containers docker-distribution

OpenNebula - Data Center Management Solution

OpenNebula provides solution for building and managing virtualized enterprise data centers and cloud infrastructures to enable on-premise IaaS clouds. OpenNebula interoperability makes cloud an evolution by leveraging existing IT assets, protecting your investments, and avoiding vendor lock-in. penNebula was designed to address the requirements of business use cases from leading companies and across multiple industries, such as Hosting, Telecom, eGovernment, Utility Computing and lot more.

cloud cloud-computing datacenter iaas infrastructure-as-a-service cloud-management

choerodon - The open source PaaS for Kubernetes.

Choerodon is an open source enterprise service platform based on container orchestration and management capabilities of Kubernetes. It integrates the tool chain of DevOps, microservices, and mobile application framework to help companies achieve Scrum application delivery and automated operations management, and provide business components such as IoT, payment, data, smart insight, and enterprise application market to help companies focus on business and accelerate digital transformation. A comprehensive tool chain supporting DevOps best practices, supporting Scrum management from planning, programming, building, testing, publishing, and operations.

kubernetes paas k8s containers microservices-application enterprise choerodon

Syncope - Open Source Identity Management

Java

Apache Syncope is an Open Source system for managing digital identities in enterprise environments. Identity management (or IdM) means to manage user data on systems and applications, using the combination of business processes and IT. IdM involves considering user attributes, roles, resources and entitlements. It provides access control on Who has access to What, When, How, and Why?

identity-management access-management roles access-control identity user-management identity-access-management

PatientOS

Java

PatientOS - an Open Source (GPL) Healthcare Information System. PatientOS (pronounced Patient-oh-es where O and S stand for Open Source) has been designed from the outset to be a Healthcare Information System (HIS). Healthcare Information System (EHR EMR PMS) for small hospitals and clinics. Scheduling, Orders, Meds, Pharmacy, Clinical Doc, HL7, Billing & more. Broad commercial support. Distributed clinical system written in pure Java with toolset to customize.

healthcare healthcare-information-system

OFBiz - Apache Open For Business Project

Java

Apache OFBiz is an open source enterprise automation software project. It means ERP (Enterprise Resource Planning), CRM (Customer RelationShip Management), E-Business / E-Commerce, SCM (Supply Chain Management), MRP (Manufacturing Resources Planning), CMMS/EAM (Maintenance Management System/Enterprise Asset Management), POS (Point Of Sale), and so on.

erp crm ecommerce cms bpm business-management supply-chain-management point-of-sale asset-management

Kaa - Middleware platform for the Internet of Things

Java

Kaa is a production-ready, multi-purpose middleware platform for building complete end-to-end IoT solutions, connected applications, and smart products. The Kaa platform provides an open, feature-rich toolkit for the IoT product development and thus dramatically reduces associated cost, risks, and time-to-market. For a quick start, Kaa offers a set of out-of-the-box enterprise-grade IoT features that can be easily plugged in and used to implement a large majority of the IoT use cases.

iot iot-platform internet-of-things middleware backend-as-service baas

fractalide - Reusable Reproducible Composable Software

Racket

Fractalide is a free and open source service programming platform using dataflow graphs. Graph nodes represent computations, while graph edges represent typed data (may also describe tensors) communicated between them. This flexible architecture can be applied to many different computation problems, initially the focus will be Microservices to be expanded out into the Internet of Things. Fractalide is in the same vein as the NSA’s Niagrafiles (now known as Apache-NiFi) or Google’s TensorFlow but stripped of all Java, Python and GUI bloat. Fractalide faces big corporate players like Ab Initio, a company that charges a lot of money for dataflow solutions.

flow-based-programming nix hypercard racket

DataflowJavaSDK - Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines

Java

Google Cloud Dataflow SDK for Java is a distribution of Apache Beam designed to simplify usage of Apache Beam on Google Cloud Dataflow service. This artifact includes the parent POM for other Dataflow SDK artifacts.

google-cloud-dataflow data-science data-analysis data-mining big-data data-processing

Apache Beam - Unified model for defining both batch and streaming data-parallel processing pipelines

Java

Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.

data-processing data-streaming batch-processing stream-processing distributed big-data

Enterprise SharePoint Patterns

Enterprise SharePoint Patterns. Tools for the Professional SharePoint Architect and Engineer. SharePoint site templates designed for rapid development.

asset-management change-management configuration-mgmt erp grants-management hr-management powershell-scripts

incubator-gobblin - Gobblin is a distributed big data integration framework (ingestion, replication, compliance, retention) for batch and streaming systems

Java

Apache Gobblin is a universal data ingestion framework for extracting, transforming, and loading large volume of data from a variety of data sources, e.g., databases, rest APIs, FTP/SFTP servers, filers, etc., onto Hadoop. Apache Gobblin handles the common routine tasks required for all data ingestion ETLs, including job/task scheduling, task partitioning, error handling, state management, data quality checking, data publishing, etc. Gobblin ingests data from different data sources in the same execution framework, and manages metadata of different sources all in one place. This, combined with other features such as auto scalability, fault tolerance, data quality assurance, extensibility, and the ability of handling data model evolution, makes Gobblin an easy-to-use, self-serving, and efficient data ingestion framework.

ActiveMQ

Java

Apache ActiveMQ is the most popular and powerful open source messaging and Integration Patterns provider. Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4.

messaging message-broker integration-pattern asynchronous message-queue