osync - A robust two way (bidirectional) file sync script based on rsync with fault tolerance, ACL support and time control

•    Shell

---

A two way filesync script running on bash Linux, BSD, Android, MacOSX, Cygwin, MSYS2, Win10 bash and virtually any system supporting bash). File synchronization is bidirectional, and can be run manually, as scheduled task, or triggered on file changes in daemon mode. It is a command line tool rsync wrapper with a lot of additional features baked in. This is a quickstart guide, you can find the full documentation on the author's site.

synchronization rsync-wrapper file-replication replication sync-files acl-replication time-control

MongoShake - MongoShake is a universal data replication platform based on MongoDB's oplog

•    Go

---

This is a brief introduction of Mongo-Shake, please visit english wiki or chinese wiki if you want to see more details including architecture, data flow, performance test, business showcase and so on. Mongo-Shake is developed and maintained by Nosql Team in Alibaba-Cloud. Mongo-Shake is a universal platform for services based on MongoDB's oplog. It fetches oplog from source mongo database, and replays in the target mongo database or sends to other ends in different tunnels. If the target side is mongo database which means replay oplog directly, it's like a syncing tool that used to copy data from source MongoDB to another MongoDB to build redundant replication or active-active replication. Except for this direct way, there are others tunnel types such like rpc, file, tcp, kafka. Receivers wrote by users must define their own interfaces to connecting to these tunnels respectively. Users can also define there own tunnel type which is pluggable. If connecting to a third-party message middleware like kafka, the consumer can get the subscriber data in an asynchronous way in pub/sub module flexibly. Here comes general data flow, The source can be either single mongod, replica set or sharding while target can be mongod or mongos. If the source is replica set, we suggest fetching data from secondary/hidden to ease the primary pressure. If the source is sharding, every shard should connect to Mongo-Shake. There can be several mongos on the target side to keep high availability, and different data will be hashed and written to different mongos.

replication mongodb-oplog active-active mongo-database mongoshake

litestream - Streaming replication for SQLite.

•    Go

---

Litestream is a standalone streaming replication tool for SQLite. It runs as a background process and safely replicates changes incrementally to another file or S3. Litestream only communicates with SQLite through the SQLite API so it will not corrupt your database. If you need support or have ideas for improving Litestream, please join the Litestream Slack or visit the GitHub Discussions. Please visit the Litestream web site for installation instructions and documentation.

replication sqlite s3

SQL Server Replication Explorer

•    

---

SQL Server Replication Explorer is a client tool for browsing through Microsoft SQL Server replication topology. It can also be used for troubleshooting and monitoring of the Microsoft SQL Server replication.

replication database distribution sql-server sql-server-2005 sql-server-2008 tools

ZooKeeper - Centralized service for maintaining configuration information

•    Java

---

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications.

service-discovery configuration-management service-coordination

gopli - DB replication tool to synchronize data with multi environments written in Golang.

•    Go

---

Database backup between remote hosts (or local) written in Golang. Write down setting file in toml.

toml replication database mysql

orchestrator - MySQL replication topology management and HA

•    Go

---

orchestrator actively crawls through your topologies and maps them. It reads basic MySQL info such as replication status and configuration.It provides with slick visualization of your topologies, including replication problems, even in the face of failures.

mysql replication high-availability management-system database-tools

Bucardo - Asynchronous PostgreSQL Replication System

•    Perl

---

Bucardo is an asynchronous PostgreSQL replication system, allowing for both multi-master and multi-slave operations.

database-replication replication db-tools

pglogical - Logical Replication extension for PostgreSQL 9

•    C

---

The pglogical extension provides logical streaming replication for PostgreSQL, using a publish/subscribe model. It is based on technology developed as part of the BDR project (http://2ndquadrant.com/BDR). To use pglogical the provider and subscriber must be running PostgreSQL 9.4 or newer.

postgresql replication logical-decoding database-replication subscription publish-subscribe data-transformation data-transport etl cdc zero-downtime

Tungsten Clustering and Replication

•    Java

---

Tungsten is a family of open source technologies for database clustering and replication. Tungsten includes replication, management, SQL routing, and proxying that improve database availability, protect data, and raise application throughput.

LizardFS - Software Defined Storage, Distributed, Parallel, Scalable, Fault-tolerant, Geo-Redundant and Highly available file system

•    C++

---

LizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.

distributed-filesystem filesystem nas distributed-systems distributed-computing distributed-storage fault-tolerance snapshot qos erasure-coding replication replicas geo-replication hsm hierarchical-storage hadoop

DBTransfer - Versatile and fast database transfer tool

•    Java

---

DBTransfer is a multi-platform java tool, that aims to replicate one database (JDBC or MS Access) to another (JDBC), using an one to one conversion.

database-tools database-replication replication

Perst - object-oriented embedded database

•    Java

---

Perst is McObject's open source, dual license, object-oriented embedded database system (ODBMS). Perst gives developers the ability to sort, store and retrieve objects in their applications with maximum speed and with low memory and storage overhead, while leveraging the object-oriented paradigm of Java and C#. It is available in one edition developed as an all-Java embedded database, and another implemented in C# (for Microsoft .NET Framework applications).

database object-database replication embedded persistance

Dragonfly - An Open-source P2P-based Image and File Distribution System

•    Go

---

Dragonfly is an open source intelligent P2P based image and file distribution system. Its goal is to tackle all distribution problems in cloud native scenarios. It is efficient with P2P and CDN technology which reduces image distribution time significantly and expedite delivery. it supports large-scale file distribution with improved downloading speed and success rate and lower consumption of bandwidth, especially cross-IDC bandwidth.

image-distribution file-distribution large-file registry containers docker-image p2p cncf cloud-native nydus kubernetes cdn

Citus - Scalable PostgreSQL for multi-tenant and real-time workloads

•    C

---

Citus is a distributed database that scales across commodity servers using transparent sharding and replication. Citus extends the underlying database rather than forking it, giving developers and enterprises the power and familiarity of a relational database. As an extension, Citus supports new PostgreSQL releases, and allows you to benefit from new features while maintaining compatibility with existing PostgreSQL tools.

database citus multi-tenant postgresql scale sharding sql distributed-database postgres

389 Directory Server - Powerful OpenSource LDAP

•    C

---

The enterprise-class Open Source LDAP server for Linux. It is hardened by real-world use, is full-featured, supports multi-master replication, and already handles many of the largest LDAP deployments in the world. OpenLDAP and Fedora Directory Server were both derived from the original University of Michigan slapd project. In 1996 the original developers of slapd became Netscape employees and developed Netscape Directory Server, which is now Fedora Directory Server.

ldap identity-management directory-server

Xenon - Decentralized Control Plane Framework

•    Java

---

Xenon is a framework for writing small REST-based services. (Some people call them microservices.) The runtime is implemented in Java and acts as the host for the lightweight, asynchronous services. The programming model is language agnostic (does not rely on Java specific constructs) so implementations in other languages are encouraged. The services can run on a set of distributed nodes. Xenon provides replication, synchronization, ordering, and consistency for the state of the services. Because of the distributed nature of Xenon, the services scale well and are highly available.

rest-framework web-framework web-application-framework

Copycat - A novel implementation of the Raft consensus algorithm

•    Java

---

Copycat is a fault-tolerant state machine replication framework. Built on the Raft consensus algorithm, it handles replication and persistence and enforces strict ordering of inputs and outputs, allowing developers to focus on single-threaded application logic. Its event-driven model allows for efficient client communication with replicated state machines, from simple key-value stores to wait-free locks and leader elections. You supply the state machine and Copycat takes care of the rest, making it easy to build robust, safe distributed systems.

raft distributed-systems consensus raft-consensus-algorithm state-machine consensus-algorithm replication database

Slony-I - Replication System for PostgreSQL

•    C

---

Slony-I is a "master to multiple slaves" replication system with cascading and failover. Slony-I is a system designed for use at data centers and backup sites, where the normal mode of operation is that all nodes are available.

replication postgres-tools db-tools database-utlities

Galera - Cluster for MySQL and MariaDB

•    C

---

Galera cluster provides distributed, multi master support for MySQL and MariaDB. Its feature include Synchronous Replication, True Multi-master, Active-Active Cluster Read and write to any node at any time, Automatic Node Provisioning, Multi-threaded Slave and lot more.

cluster mysql-cluster distributed multi-master replication mysql