wal2json - JSON output plugin for changeset extraction

  •        27

wal2json is an output plugin for logical decoding. It means that the plugin have access to tuples produced by INSERT and UPDATE. Also, UPDATE/DELETE old row versions can be accessed depending on the configured replica identity. Changes can be consumed using the streaming protocol (logical replication slots) or by a special SQL API. The wal2json output plugin produces a JSON object per transaction. All of the new/old tuples are available in the JSON object. Also, there are options to include properties such as transaction timestamp, schema-qualified, data types, and transaction ids.

https://github.com/eulerto/wal2json

Tags
Implementation
License
Platform

   




Related Projects

repmgr - The Most Popular Replication Manager for PostgreSQL (Postgres) - latest version 4

  •    C

repmgr is a suite of open-source tools to manage replication and failover within a cluster of PostgreSQL servers. It enhances PostgreSQL's built-in replication capabilities with utilities to set up standby servers, monitor replication, and perform administrative tasks such as failover or switchover operations. repmgr 4 is a complete rewrite of the existing repmgr codebase, allowing the use of all of the latest features in PostgreSQL replication.

rdb - Javascript ORM

  •    Javascript

ORM for nodejs. Supports postgres, mySql and sqlite. 1.7.1 Support for schemas (postgres only). 1.7.0 sqlite3 is now a peer dependency. Add it to your own package.json if you intend to use it. 1.6.9 Bugfix: one-to-many relation returns empty if strategy is included. 1.6.8 Bugfix: one-to-many relation returns empty if insert/update is done earlier in transaction. 1.6.7 Bugfix in relations. 1.6.6 Bugfix. 1.6.5 Improved performance on relations. 1.6.4 Bugfix. 1.6.3 Bugfix: potential incorrect timeZoneOffset when serializing date to JSON. Got timeZoneOffset from now() instead of on actual date. 1.6.2 Removed es6 syntax to ensure backwards compatability. Fixed global var leak. 1.6.1 Now supporting sqlite. 1.6.0 Bugfix: potential ambigous column error when using limit and relating to other tables. 1.5.9 Bugfix: using multipleStatements in mySql could sometimes cause an error when updates are run right before a select. Improved performance on limit when relating to other tables. Using uuid instead of node-uuid Updated all dependencies but generic-pool to latest. (Generic-pool has some breaking changes in latest. I will update it in next release.) 1.5.8 Cleanup line breaks in documentation. 1.5.7 Bugfix: getById.exclusive and tryGetById.exclusive did not lock if row was cached. Improved performance on tryGetFirst. 1.5.6 Raw sql filters can accept sql both as string and as function. E.g. var filter = {sql: function() {return 'foo > 1';}}. 1.5.5 Optional locks for getMany, tryGetFirst and tryGetById. Instead of calling getMany(params) just call getMany.exclusive(params). Same syntax goes for tryGetFirst and tryGetById. This will result in SELECT FOR UPDATE. Bugfix: bulk deletes now accepts raw sql filters too. 1.5.4 Transaction locks. Postgres only. 1.5.3 Upgraded to pg 6.0.3 1.5.2 Improved performance and reduced memory footprint. 1.5.1 Documented JSON column type. Bug fix: Insert and foreign key violation. 1.5.0 JSON column type. Postgres json type does not support rdb filters. 1.4.1 Empty filter would sometimes cause invalid filter. 1.4.0 Raw SQL query. 1.3.0 getMany() now supports limit and orderBy - same syntax as in streaming. 1.2.3 Bugfix: iEqual gave incorrect sql when parameterized. 1.2.2 Exlusive no longer returns a clone of table. It has changes current table to exclusive locking. 1.2.1 Bugfix: Exclusive row locks 1.2.0 Exclusive row locks 1.1.0 Now supporting streaming. Requires postgres or MySQL >=5.7.7 1.0.8 README fixup. 1.0.7 Better performance on insert and update. 1.0.6 Bugfix: Transaction domain should not forward rdb singleton from old domain. 1.0.5 Documentation cleanup. 1.0.4 orderBy in toDto(). 1.0.3 toDto() using next tick on every thousandth row to avoid maximum call stack size exceeded. 1.0.2 Reduced number of simultaneous promises in order to avoid maximum call stack size exceeded. 1.0.1 Bugfix: Incorrect insert/updates on timestamp without timezone. The time was converted utc instead of stripping the timezone. 1.0.0 Transaction domain forwards properties from old domain. Semantic versioning from now on. 0.5.1 Improved performance 0.5.0 Logging: rdb.log(someFunc) logs sql and parameters. Raw sql filters. 0.4.9 New method: tryGetById. New filter: iEqual, postgres only. Bugfix: rows.toJSON() without strategy did not include any children. 0.4.8 Explicit pooling with size and end(). Bugfix: mySql did not release client to pool. 0.4.7 Upgraded to pg 4.3.0 Upgraded to mysql 2.5.5 0.4.6 Upgraded pg 4.2.0. 0.4.5 Oops. Forgot to use pg.js instead of pg. 0.4.4 Upgraded all dependencies to latest. Using pg.js instead of pg. 0.4.3 Can ignore columns when serializing to dto. 0.4.2 Bugfix: update on a row crashes when a delete occurs earlier in same transaction. 0.4.1 Bugfix: more global leaks. 0.4.0 Bugfix: global leak. 0.3.9 Bugfix: eager loading joins/hasOne with non unique column names was not handled correctly. 0.3.8 Supports mySql. Bulk deletes. 0.3.7 Bugfix: eager loading manyRelation on a join/hasOne returned empty array #11 0.3.6 Fixed sql injection vulnerability. 0.3.5 Built-in fetching strategies for lazy loading. Works best in readonly scenarios. 0.3.4 Docs and examples split moved to separate file. 0.3.3 Fixed documentation layout again. 0.3.2 Fixed documentation layout. 0.3.1 Case insensitive filters: iStartsWith, iEndsWith and iContains. 0.3.0 Fix broken links in docs. 0.2.9 Support for row.delete(). Rollback only throws when error is present. 0.2.8 Guid accepts uppercase letters. Bugfix: null inserts on guid columns yielded wrong sql. 0.2.7 New method, toDto(), converts row to data transfer object. Bugfix: toJSON returned incorrect string on hasMany relations. 0.2.6 Fixed incorrect links in README. 0.2.5 Bugfix: caching on composite keys could give a crash #7. Improved sql compression on insert/update. 0.2.4 Bugfix: getMany with many-strategy and shallowFilter yields incorrect query #6. 0.2.3 Reformatted documentation. No code changes.

pgcenter - Command-line admin tool for observing and troubleshooting Postgres.

  •    Go

pgCenter is a command-line admin tool for observing and troubleshooting Postgres. Postgres provides various activity statistics that include detailed information about its behaviour: connections, statements, database operations, replication, resources usage and more. General purpose of the statistics is to help DBAs to monitor and troubleshoot Postgres. However, these statistics provided in textual form retrieved from SQL functions and views, and Postgres doesn't provide any tools for working with them. pgCenter's main goal is to help Postgres DBA manage statistics that theу have in their databases and see all the necessary data in convenient format based on builtin stats views and functions.

Solr - Blazing-fast, open source enterprise search platform

  •    Java

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites.

bdr - Bi-Directional Multi-Master Replication (BDR) for PostgreSQL (Postgres)

  •    C

Bi-Directional Multi-Master Replication (BDR) for PostgreSQL (Postgres)


Slony-I - Replication System for PostgreSQL

  •    C

Slony-I is a "master to multiple slaves" replication system with cascading and failover. Slony-I is a system designed for use at data centers and backup sites, where the normal mode of operation is that all nodes are available.

Citus - Scalable PostgreSQL for multi-tenant and real-time workloads

  •    C

Citus is a distributed database that scales across commodity servers using transparent sharding and replication. Citus extends the underlying database rather than forking it, giving developers and enterprises the power and familiarity of a relational database. As an extension, Citus supports new PostgreSQL releases, and allows you to benefit from new features while maintaining compatibility with existing PostgreSQL tools.

DBBalancer - Middleware btw Database client and server

  •    C++

DBBalancer is some sort of middleware that would sit in between of database clients and a server. Currently the only server supported is Postgres, but the architecture is open to embrace more servers in a future. It's a connection pool, a load balancer and a database replicator.

wal-e - Continuous Archiving for Postgres

  •    Python

WAL-E is a program designed to perform continuous archiving of PostgreSQL WAL files and base backups. To correspond on using WAL-E or to collaborate on its development, do not hesitate to send mail to the mailing list at wal-e@googlegroups.com (archives and subscription settings). Github issues are also currently being used to track known problems, so please feel free to submit those.

RethinkDB - Distributed JSON database

  •    C++

RethinkDB is built to store JSON documents, and scale to multiple machines with very little effort. It has a pleasant query language that supports really useful queries like table joins and group by, and is easy to setup and learn. It supports JSON data model, Distributed joins, subqueries, aggregation, atomic updates, Hadoop-style map/reduce.

Sequelize - An easy-to-use multi SQL dialect ORM for Node.js

  •    Javascript

Sequelize is a promise-based Node.js ORM for Postgres, MySQL, SQLite and Microsoft SQL Server. It features solid transaction support, relations, read replication and more.

PostgreSQL - Powerful, Open Source Object-Relational Database System

  •    C

PostgreSQL is a powerful, open source object-relational database system. It has more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness. It is fully ACID compliant, has full support for foreign keys, joins, views, triggers, and stored procedures (in multiple languages). It has native programming interfaces for most of the programming language.

Kallax - PostgreSQL typesafe ORM for the Go language

  •    Go

Kallax is a PostgreSQL typesafe ORM for the Go language. It aims to provide a way of programmatically write queries and interact with a PostgreSQL database without having to write a single line of SQL, use strings to refer to columns and use values of any type in queries.

Pincaster - NoSQL Database with a HTTP/JSON interface

  •    C

Pincaster is an in-memory, persistent data store for geographic data and key/dictionary pairs, with replication and a HTTP/JSON interface.

db4o

  •    Java

db4o open source object database, native to Java and .NET.

ToroDB Stampede - Provides better analytics on top of MongoDB and make it easier to migrate from MongoDB to SQL

  •    Java

ToroDB helps to transform your NoSQL data from a MongoDB replica set into a relational database in PostgreSQL. There are other solutions that are able to store the JSON document in a relational table using PostgreSQL JSON support, but it doesn't solve the real problem of 'how to really use that data'. ToroDB Stampede replicates the document structure in different relational tables and stores the document data in different tuples using those tables.

SQL Server Replication Explorer

  •    

SQL Server Replication Explorer is a client tool for browsing through Microsoft SQL Server replication topology. It can also be used for troubleshooting and monitoring of the Microsoft SQL Server replication.

SymmetricDS - Multi Database Replication

  •    Java

SymmetricDS is a database and file synchronization solution that is platform-independent, web-enabled, and database agnostic. SymmetricDS was built to make data replication across two to tens of thousands of databases and file systems fast, easy and resilient. It supports near real time, bi-directional data replication across large node networks over the WAN or LAN.

replication-manager - Signal 18 repman - Replication Manager for MySQL / MariaDB / Percona Server

  •    Go

replication-manager is an high availability solution to manage MariaDB 10.x and MySQL & Percona Server 5.7 GTID replication topologies. It includes third-party libraries released under their own licences. Please refer to the vendor directory for more information.