Transporter - Sync data between persistence engines, like ETL only not stodgy

  •        176

Compose Transporter helps with database transformations from one store to another. It can also sync from one to another or several stores.Transporter allows the user to configure a number of data adaptors as sources or sinks. These can be databases, files or other resources. Data is read from the sources, converted into a message format, and then send down to the sink where the message is converted into a writable format for its destination. The user can also create data transformations in JavaScript which can sit between the source and sink and manipulate or filter the message flow.

https://github.com/compose/transporter

Tags
Implementation
License
Platform

   




Related Projects

Ora2Pg - Tool used to migrate an Oracle database to a PostgreSQL compatible schema

  •    Perl

Ora2Pg is a free tool used to migrate an Oracle database to a PostgreSQL compatible schema. It connects your Oracle database, scan it automatically and extracts its structure or data, it then generates SQL scripts that you can load into PostgreSQL.

phpMyFAQ - phpMyFAQ - Open Source FAQ system for PHP and MySQL, PostgreSQL and other databases

  •    PHP

phpMyFAQ is a multilingual, completely database-driven FAQ-system. It supports various databases to store all data, PHP 5.6+ is needed in order to access this data. phpMyFAQ also offers a multi-language Content Management System with a WYSIWYG editor and an Image Manager, real time search support with Elasticsearch, flexible multi-user support with user and group based permissions on categories and records, a wiki-like revision feature, a news system, user-tracking, 40+ supported languages, enhanced automatic content negotiation, HTML5/CSS3 based responsive templates, PDF-support, a backup-system, a dynamic sitemap, related FAQs, tagging, RSS feeds, built-in spam protection systems, OpenLDAP and Microsoft Active Directory support, and an easy to use installation script. phpMyFAQ is only supported on PHP 5.6.0 and up, you need a database as well. Supported databases are MySQL, Percona Server, PostgreSQL, Microsoft SQL Server, SQLite3 and MariaDB. If you want to use Elasticsearch as main search engine, you need Elasticsearch 2.x as well. Check our detailed requirements on phpmyfaq.de for more information.

SQLToNoSQLImporter - Import data from SQL to NoSQL systems

  •    Java

SQLToNoSQLImporter is a Solr like data import handler to import Sql (MySQL,Oracle,PostgreSQL) data to NoSQL Systems (Mongodb,CouchDB,Elastic Search). Migration is now completely configuration driven. SQLToNoSQLImporter reads from sql databases, converts and then batch inserts them into NoSQL datastore.

mosql - MongoDB to PostgreSQL Streaming Replication

  •    Ruby

MoSQL imports the contents of your MongoDB database cluster into a PostgreSQL instance, using an oplog tailer to keep the SQL mirror live up-to-date. This lets you run production services against a MongoDB database, and then run offline analytics or reporting using the full power of SQL.


DBeaver - Universal SQL Manager and SQL Client

  •    Java

DBeaver is a multi-platform database tool for developers, SQL programmers, database administrators and analysts. It supports all popular databases: MySQL, PostgreSQL, SQLite, Oracle, DB2, SQL Server, Sybase, Teradata, MongoDB, Cassandra, Redis, etc. Its feature include metadata editor, SQL editor, rich data editor, ERD, data export/import/migration, SQL execution plans, etc.

node-pg-migrate - Node.js database migration management for Postgresql

  •    Javascript

Node.js database migration management built exclusively for postgres. (But can also be used for other DBs conforming to SQL standard - e.g. CockroachDB.) Started by Theo Ephraim, now maintained by Salsita Software. You can specify your database connection information using config.

Mongify - Data translator system for moving your SQL data to MongoDB

  •    Ruby

Mongify allows you to map your data from a sql database and into a mongodb document database. It supports MySQL, PostgreSQL, SQLite, Oracle, SQLServer, and DB2.

Evolve - Database migration tool for .NET and .NET Core projects. Inspired by Flyway.

  •    CSharp

Database migration tool for .NET and .NET Core. Inspired by Flyway. Evolve is an easy migration tool that uses plain SQL scripts. Its purpose is to automate your database changes, and help keep those changes synchronized through all your environments and development teams. This makes it an ideal tool for continuous integration / delivery.

Adminer - Database Management in Single PHP file

  •    PHP

Adminer (formerly phpMinAdmin) is a full-featured database management tool written in PHP. Conversely to phpMyAdmin, it consist of a single file ready to deploy to the target server. Adminer is available for MySQL, PostgreSQL, SQLite, MS SQL, Oracle, SimpleDB, Elasticsearch and MongoDB

nodock - Docker Compose for Node projects with Node, MySQL, Redis, MongoDB, NGINX, Apache2, Memcached, Certbot and RabbitMQ images

  •    Shell

The docker Node.js image is very simple, you give it an entrypoint and it runs it. This is fine for very simple/small scripts but for larger projects you'll probably want something a bit more robust. The goal of NoDock is to provide a complete environment for your node project: Node.js service(s), databases, web servers, queues, etc. while doing the "wiring" for you.

PostGIS - Spatial and Geographic objects for PostgreSQL

  •    C

PostGIS is a spatial database extender for PostgreSQL object-relational database. It adds support for geographic objects allowing location queries to be run in SQL. PostGIS adds extra types (geometry, geography, raster and others) to the PostgreSQL database. It also adds functions, operators, and index enhancements that apply to these spatial types.

ToroDB Stampede - Provides better analytics on top of MongoDB and make it easier to migrate from MongoDB to SQL

  •    Java

ToroDB helps to transform your NoSQL data from a MongoDB replica set into a relational database in PostgreSQL. There are other solutions that are able to store the JSON document in a relational table using PostgreSQL JSON support, but it doesn't solve the real problem of 'how to really use that data'. ToroDB Stampede replicates the document structure in different relational tables and stores the document data in different tuples using those tables.

pgAdmin - Administration and development platform for PostgreSQL

  •    C

pgAdmin is the most popular and feature rich administration and development platform for PostgreSQL. It is designed to answer the needs of all users, from writing simple SQL queries to developing complex databases. The graphical interface supports all PostgreSQL features and makes administration easy. The application also includes a syntax highlighting SQL editor, a server-side code editor, an SQL/batch/shell job scheduling agent, support for the Slony-I replication engine and much more.

Citus - Scalable PostgreSQL for multi-tenant and real-time workloads

  •    C

Citus is a distributed database that scales across commodity servers using transparent sharding and replication. Citus extends the underlying database rather than forking it, giving developers and enterprises the power and familiarity of a relational database. As an extension, Citus supports new PostgreSQL releases, and allows you to benefit from new features while maintaining compatibility with existing PostgreSQL tools.

stream-reactor - Streaming reference architecture for ETL with Kafka and Kafka-Connect

  •    Scala

Lenses offers SQL (for data browsing and Kafka Streams), Kafka Connect connector management, cluster monitoring and more. A collection of components to build a real time ingestion pipeline.

pgModeler - PostgreSQL Database Modeler

  •    C++

pgModeler - PostgreSQL Database Modeler - is an open source data modeling tool designed for PostgreSQL. No more DDL commands written by hand let pgModeler do the job for you! This software reunites the concepts of entity-relationship diagrams and the features that PostgreSQL implements as extensions of SQL standards.

SQL Relay - Database Connection Pool library with API available in all programming languages

  •    C++

SQL Relay is a persistent database connection pooling, proxying and load balancing system for Unix and Linux supporting ODBC, and all major databases. It has APIs for C, C++, ODBC, Perl, Perl-DBI, Python, Python-DB, Zope, PHP, Ruby, Ruby-DBI, Java, TCL and Erlang, drop-in replacement libraries for MySQL and PostgreSQL clients.