lapidus - Stream your PostgreSQL, MySQL or MongoDB databases anywhere, fast.

  •        111

Currently MySQL and PostgreSQL databases are fully supported. MongoDB supports inserts and deletes, however, updates return a copy of the operation (for example, a $rename operation will return a $set for the new field and an $unset for the old field) instead of the object as it exists in the database. Redis support is on the way. Lapidus can currently be used as a daemon or Node.js module. Support for piping line-delimited JSON to other processes is a high priority. You'll need PostgreSQL 9.4 or higher with logical replication configured and the JSONCDC plugin installed and loaded. Any PostgreSQL fork that ships with pg_recvlogical should be compatible.


async : ^1.5.2
errno : ^0.1.4
jsonlint : ^1.6.2
ldjson-stream : ^1.2.1
mongodb : ^2.1.18
yargs : ^3.32.0



Related Projects

pglogical - Logical Replication extension for PostgreSQL 9

  •    C

The pglogical extension provides logical streaming replication for PostgreSQL, using a publish/subscribe model. It is based on technology developed as part of the BDR project ( To use pglogical the provider and subscriber must be running PostgreSQL 9.4 or newer.

rdb - Javascript ORM

  •    Javascript

ORM for nodejs. Supports postgres, mySql and sqlite. 1.7.1 Support for schemas (postgres only). 1.7.0 sqlite3 is now a peer dependency. Add it to your own package.json if you intend to use it. 1.6.9 Bugfix: one-to-many relation returns empty if strategy is included. 1.6.8 Bugfix: one-to-many relation returns empty if insert/update is done earlier in transaction. 1.6.7 Bugfix in relations. 1.6.6 Bugfix. 1.6.5 Improved performance on relations. 1.6.4 Bugfix. 1.6.3 Bugfix: potential incorrect timeZoneOffset when serializing date to JSON. Got timeZoneOffset from now() instead of on actual date. 1.6.2 Removed es6 syntax to ensure backwards compatability. Fixed global var leak. 1.6.1 Now supporting sqlite. 1.6.0 Bugfix: potential ambigous column error when using limit and relating to other tables. 1.5.9 Bugfix: using multipleStatements in mySql could sometimes cause an error when updates are run right before a select. Improved performance on limit when relating to other tables. Using uuid instead of node-uuid Updated all dependencies but generic-pool to latest. (Generic-pool has some breaking changes in latest. I will update it in next release.) 1.5.8 Cleanup line breaks in documentation. 1.5.7 Bugfix: getById.exclusive and tryGetById.exclusive did not lock if row was cached. Improved performance on tryGetFirst. 1.5.6 Raw sql filters can accept sql both as string and as function. E.g. var filter = {sql: function() {return 'foo > 1';}}. 1.5.5 Optional locks for getMany, tryGetFirst and tryGetById. Instead of calling getMany(params) just call getMany.exclusive(params). Same syntax goes for tryGetFirst and tryGetById. This will result in SELECT FOR UPDATE. Bugfix: bulk deletes now accepts raw sql filters too. 1.5.4 Transaction locks. Postgres only. 1.5.3 Upgraded to pg 6.0.3 1.5.2 Improved performance and reduced memory footprint. 1.5.1 Documented JSON column type. Bug fix: Insert and foreign key violation. 1.5.0 JSON column type. Postgres json type does not support rdb filters. 1.4.1 Empty filter would sometimes cause invalid filter. 1.4.0 Raw SQL query. 1.3.0 getMany() now supports limit and orderBy - same syntax as in streaming. 1.2.3 Bugfix: iEqual gave incorrect sql when parameterized. 1.2.2 Exlusive no longer returns a clone of table. It has changes current table to exclusive locking. 1.2.1 Bugfix: Exclusive row locks 1.2.0 Exclusive row locks 1.1.0 Now supporting streaming. Requires postgres or MySQL >=5.7.7 1.0.8 README fixup. 1.0.7 Better performance on insert and update. 1.0.6 Bugfix: Transaction domain should not forward rdb singleton from old domain. 1.0.5 Documentation cleanup. 1.0.4 orderBy in toDto(). 1.0.3 toDto() using next tick on every thousandth row to avoid maximum call stack size exceeded. 1.0.2 Reduced number of simultaneous promises in order to avoid maximum call stack size exceeded. 1.0.1 Bugfix: Incorrect insert/updates on timestamp without timezone. The time was converted utc instead of stripping the timezone. 1.0.0 Transaction domain forwards properties from old domain. Semantic versioning from now on. 0.5.1 Improved performance 0.5.0 Logging: rdb.log(someFunc) logs sql and parameters. Raw sql filters. 0.4.9 New method: tryGetById. New filter: iEqual, postgres only. Bugfix: rows.toJSON() without strategy did not include any children. 0.4.8 Explicit pooling with size and end(). Bugfix: mySql did not release client to pool. 0.4.7 Upgraded to pg 4.3.0 Upgraded to mysql 2.5.5 0.4.6 Upgraded pg 4.2.0. 0.4.5 Oops. Forgot to use pg.js instead of pg. 0.4.4 Upgraded all dependencies to latest. Using pg.js instead of pg. 0.4.3 Can ignore columns when serializing to dto. 0.4.2 Bugfix: update on a row crashes when a delete occurs earlier in same transaction. 0.4.1 Bugfix: more global leaks. 0.4.0 Bugfix: global leak. 0.3.9 Bugfix: eager loading joins/hasOne with non unique column names was not handled correctly. 0.3.8 Supports mySql. Bulk deletes. 0.3.7 Bugfix: eager loading manyRelation on a join/hasOne returned empty array #11 0.3.6 Fixed sql injection vulnerability. 0.3.5 Built-in fetching strategies for lazy loading. Works best in readonly scenarios. 0.3.4 Docs and examples split moved to separate file. 0.3.3 Fixed documentation layout again. 0.3.2 Fixed documentation layout. 0.3.1 Case insensitive filters: iStartsWith, iEndsWith and iContains. 0.3.0 Fix broken links in docs. 0.2.9 Support for row.delete(). Rollback only throws when error is present. 0.2.8 Guid accepts uppercase letters. Bugfix: null inserts on guid columns yielded wrong sql. 0.2.7 New method, toDto(), converts row to data transfer object. Bugfix: toJSON returned incorrect string on hasMany relations. 0.2.6 Fixed incorrect links in README. 0.2.5 Bugfix: caching on composite keys could give a crash #7. Improved sql compression on insert/update. 0.2.4 Bugfix: getMany with many-strategy and shallowFilter yields incorrect query #6. 0.2.3 Reformatted documentation. No code changes.

wal2json - JSON output plugin for changeset extraction

  •    C

wal2json is an output plugin for logical decoding. It means that the plugin have access to tuples produced by INSERT and UPDATE. Also, UPDATE/DELETE old row versions can be accessed depending on the configured replica identity. Changes can be consumed using the streaming protocol (logical replication slots) or by a special SQL API. The wal2json output plugin produces a JSON object per transaction. All of the new/old tuples are available in the JSON object. Also, there are options to include properties such as transaction timestamp, schema-qualified, data types, and transaction ids.

Barman - Backup and Recovery manager for PostgreSQL

  •    Python

Barman (Backup and Recovery Manager) is an open source administration tool for disaster recovery of PostgreSQL servers . It allows your organisation to perform remote backups of multiple servers in business critical environments and to help DBAs during the recovery phase. Its features include backup catalogues, incremental backup, retention policies, remote backup and recovery, archiving and compression of WAL files and backups.

node-pg-copy-streams - COPY FROM / COPY TO for node-postgres

  •    Javascript

COPY FROM / COPY TO for node-postgres. Stream from one database to another, and stuff. PostgreSQL supports streaming binary data directly into and out of a table. This means you can take your favorite CSV or TSV or whatever format file and pipe it directly into an existing PostgreSQL table.

kingbus - A distributed MySQL binlog storage system built on Raft

  •    Go

MySQL replication protocol compatibility, pull the binlog files from the master through gtid mode, and push the binlog file to slave through gtid mode in the same way. Geo-Replication, kingbus uses Raft to support Geo-Replication. The binlog data written to the cluster is guaranteed to be consistent between multiple nodes, and the order of binlog event is exactly the same as that on the master.

Maxwell's daemon - A mysql-to-json kafka producer

  •    Java

This is Maxwell's daemon, an application that reads MySQL binlogs and writes row updates to Kafka as JSON. Maxwell has a low operational bar and produces a consistent, easy to ingest stream of updates. It allows you to easily "bolt on" some of the benefits of stream processing systems without going through your entire code base to add (unreliable) instrumentation points.

pgrebase - Easy postgres codebase injection - aka postgres non data live reload

  •    Go

PgRebase is a tool that allows you to easily handle your postgres codebase for functions, triggers and views. If you started outsourcing data manipulation to your database through postgresql cool features, you probably realized this is painful. Postgresql's functions, triggers and views are not your usual codebase, they live in postgres, and you often have to drop them if you want to edit them, eg when you change a function signature. You could edit them directly in psql, but then it's migrating servers / other devs installation that becomes difficult.

schemats - Generate typescript interface definitions from SQL database schema

  •    TypeScript

Using Schemats, you can generate TypeScript interface definitions from (Postgres, MySQL) SQL database schema automatically. For an overview on the motivation and rational behind this project, please take a look at Statically typed PostgreSQL queries in Typescript .

mysql_fdw - PostgreSQL foreign data wrapper for MySQL

  •    C

This PostgreSQL extension implements a Foreign Data Wrapper (FDW) for MySQL. Please note that this version of mysql_fdw works with PostgreSQL and EDB Postgres Advanced Server 9.3, 9.4, 9.5, 9.6, 10 and 11.

Aura.SqlQuery - Independent query builders for MySQL, PostgreSQL, SQLite, and Microsoft SQL Server.

  •    PHP

Provides query builders for MySQL, Postgres, SQLite, and Microsoft SQL Server. These builders are independent of any particular database connection library, although PDO in general is recommended. This package is installable and PSR-4 autoloadable via Composer as aura/sqlquery.

Go-JWT-Postgres-Mysql-Restful-API - This is an API built with golang, jwt, gorm, postgresql, mysql

  •    Go

This is an application built with golang, jwt, gorm, postgresql, mysql.

MySQL-Syncer - MySQL Syncer is a project which parse mysql binlog and sync to other datases, such as redis, mongodb and any other databases

  •    C

MySQL Syncer is a project which parse mysql binlog and sync to other datases, such as redis, mongodb and any other databases..

awesome-postgres - A curated list of awesome PostgreSQL software, libraries, tools and resources, inspired by awesome-mysql


🐘 Contributions welcome. Add links through pull requests or create an issue to start a discussion. Please take a look at the contribution guidelines.

Metabase - The simplest, fastest way to get business intelligence and analytics to everyone in your company

  •    Clojure

Metabase is the easy, open source way for everyone in your company to ask questions and learn from data. Get a real-time glimpse into what your company is learning about your data. Activity helps people in your company find an answer, jump start their own exploration, or improve existing questions.

postgres-operator - Postgres operator creates and manages PostgreSQL clusters running in Kubernetes

  •    Go

The operator watches additions, updates, and deletions of PostgreSQL cluster manifests and changes the running clusters accordingly. For example, when a user submits a new manifest, the operator fetches that manifest and spawns a new Postgres cluster along with all necessary entities such as Kubernetes StatefulSets and Postgres roles. See this Postgres cluster manifest for settings that a manifest may contain. The operator also watches updates to its own configuration and alters running Postgres clusters if necessary. For instance, if a pod docker image is changed, the operator carries out the rolling update. That is, the operator re-spawns one-by-one pods of each StatefulSet it manages with the new Docker image.

integresql - IntegreSQL manages isolated PostgreSQL databases for your integration tests.

  •    Go

IntegreSQL manages isolated PostgreSQL databases for your integration tests. Do your engineers a favour by allowing them to write fast executing, parallel and deterministic integration tests utilizing real PostgreSQL test databases. Resemble your live environment in tests as close as possible.

node-postgres - PostgreSQL client for node.js

  •    Javascript

Non-blocking PostgreSQL client for Node.js. Pure JavaScript and optional native libpq bindings. It supports Connection pooling, PostgreSQL data-type coercion, Parameterized queries, Named statements with query plan caching, Async notifications with LISTEN/NOTIFY and lot more.

pgcenter - Command-line admin tool for observing and troubleshooting Postgres.

  •    Go

pgCenter is a command-line admin tool for observing and troubleshooting Postgres. Postgres provides various activity statistics that include detailed information about its behaviour: connections, statements, database operations, replication, resources usage and more. General purpose of the statistics is to help DBAs to monitor and troubleshoot Postgres. However, these statistics provided in textual form retrieved from SQL functions and views, and Postgres doesn't provide any tools for working with them. pgCenter's main goal is to help Postgres DBA manage statistics that theу have in their databases and see all the necessary data in convenient format based on builtin stats views and functions.

We have large collection of open source products. Follow the tags from Tag Cloud >>

Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.