go-sync - gosync is a library for Golang styled around zsync / rsync, written with the intent that it enables efficient differential file transfer in a number of ways

  •        30

The ZSync mechanism has the weakness that HTTP1.1 ranged requests are not always well supported by CDN providers and ISP proxies. When issues happen, they're very difficult to respond to correctly in software (if possible at all). Using HTTP 1.0 and fully completed GET requests would be better, if possible. There are some other issues too - ZSync doesn't (as far as I'm aware) solve any issues to do with storage of a files, which can get more and more onerous for large files that are not changing much from one version to another.

https://github.com/Redundancy/go-sync

Tags
Implementation
License
Platform

   




Related Projects

rsync - incremental file transfer program for Unix systems

  •    C

rsync is a file transfer program for Unix systems. rsync uses the rsync algorithm which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand. It can update whole directory trees and filesystems. It can use rsh, ssh or direct sockets as the transport.

sshync - Auto-sync files or directories over SSH.

  •    Javascript

Auto-sync files or directories over SSH using rsync and fs.watch(). Exclude files or directories by creating .sshyncignore in your source root (see repo root for example).

bitpocket - "DIY Dropbox" or "2-way directory (r)sync with proper deletion"

  •    Shell

bitpocket is a small but smart script that does 2-way directory synchronization. It uses rsync to do efficient data transfer and tracks local file creation/removal to avoid known rsync problem when doing 2-way syncing with deletion. bitpocket can use any server which you have ssh access to for its central storage. If you have gigabytes of free disk space on your hosting server you can finally make use of it.

gsync - gSync is an rsync based library for sending delta updates of files to a remote server.

  •    Go

A Go library implementing rsync algorithm described at https://www.samba.org/~tridge/phd_thesis.pdf and in more detail at http://paperswelove.org/2017/video/camilo-aguilar-rsync-algorithm/.

osync - A robust two way (bidirectional) file sync script based on rsync with fault tolerance, ACL support and time control

  •    Shell

A two way filesync script running on bash Linux, BSD, Android, MacOSX, Cygwin, MSYS2, Win10 bash and virtually any system supporting bash). File synchronization is bidirectional, and can be run manually, as scheduled task, or triggered on file changes in daemon mode. It is a command line tool rsync wrapper with a lot of additional features baked in. This is a quickstart guide, you can find the full documentation on the author's site.


Git Large File Storage - Git extension for versioning large files

  •    Go

Git LFS is a command line extension and specification for managing large files with Git. The client is written in Go, with pre-compiled binaries available for Mac, Windows, Linux, and FreeBSD.

Sync Gateway - Connects Couchbase Lite to Couchbase Server

  •    Go

Sync Gateway connects Couchbase Lite to Couchbase Server. It manages HTTP-based data access for Couchbase Lite mobile clients, Leverages Couchbase Server as it's horizontally scalable backing data store, Clustered into a horizontally scalable tier, Provides access control and data routing.

croc - Easily and securely send things from one computer to another :crocodile: :package:

  •    Go

Easily and securely transfer stuff from one computer to another. croc allows any two computers to directly and securely transfer files and folders. When sending a file, croc generates a random code phrase which must be shared with the recipient so they can receive the file. The code phrase encrypts all data and metadata and also serves to authorize the connection between the two computers in a intermediary relay. The relay connects the TCP ports between the two computers and does not store any information (and all information passing through it is encrypted).

IRSYNC (file transfer utility)

  •    C++

Flexible, secure file-transfer utility similar to rsync achieving high transfer rates for both large and small files on networks with both LANs and WANs (networks with low or high latencies). Supports interactive batch as well as scripting operation.

kaitai_struct - Kaitai Struct: declarative language to generate binary data parsers in C++ / C# / Go / Java / JavaScript / Lua / Perl / PHP / Python / Ruby

  •    Shell

Kaitai Struct is a declarative language used for describing various binary data structures laid out in files or in memory: i.e. binary file formats, network stream packet formats, etc. The main idea is that a particular format is described in Kaitai Struct language only once and then can be compiled with a ksc into source files in one of the supported programming languages. These modules will include a generated code for a parser that can read described data structure from a file / stream and give access to it in a nice, easy-to-comprehend API.

go-bindata - A small utility which generates Go code from any file

  •    Go

This package converts any file into managable Go source code. Useful for embedding binary data into a go program. The file data is optionally gzip compressed before being converted to a raw byte slice.It comes with a command line tool in the go-bindata sub directory. This tool offers a set of command line options, used to customize the output being generated.

FullSync - File Synchronization and Backup Tool

  •    Java

FullSync is a universal file synchronization and backup tool which is highly customizable and expandable. It is especially for developers, but the basic functionality is easy enough for everyone. It supports multiple protocols like FTP, SFTP, SMB/CIFS(Windows Shares) and local files, Scheduled execution of synchronizations using intervals or a crontab like scheduling. Publication and update of websites, synchronization of local directories and making backups of your data.

go-bindata - go-bindata.github.io

  •    Go

This package converts any file into managable Go source code. Useful for embedding binary data into a go program. The file data is optionally gzip compressed before being converted to a raw byte slice. It comes with a command line tool in the go-bindata sub directory. This tool offers a set of command line options, used to customize the output being generated.

curl - A command line tool and library for transferring data with URL syntax, supporting HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, FILE, IMAP, SMTP, POP3, RTSP and RTMP

  •    C

curl is used in command lines or scripts to transfer data. It is also used in cars, television sets, routers, printers, audio equipment, mobile phones, tablets, settop boxes, media players and is the internet transfer backbone for thousands of software applications affecting billions of humans daily.

csync - A professional file synchronizer for normal users

  •    C

sync is a lightweight utility to synchronize files between two directories on a system or between multiple systems. It synchronizes bidirectionally and allows the user to keep two copies of files and directories in sync. csync uses widely adopted protocols, such as smb or sftp, so that there is no need for a server component. It is a user-level program which means you don’t need to be a superuser or administrator.

PerlRsync

  •    C

Perl implentation of rsync that can talk directly to a native rsync program or rsync daemon. The file system I/O is abstracted into a seperate module, which allows an rsync interface to be developed for data other than regular file systems.

Audio Data Transfer Library

  •    

ADTL or Audio Data Transfer Library provides a light library that allows developers to transfer binary data over a sound card. This library is intended for low-bandwidth radio transmission for use in autonomous or r/c related applications.

QSync

  •    

QSync is a Qt-based frontend to the rsync file transfer utility, with an emphasis on initial ease-of-use, but also with the ability to be just as powerful as rsync itself. It will eventually support many, if not all, of rsync's many options.

socket.io-stream - Stream for Socket.IO

  •    Javascript

This is the module for bidirectional binary data transfer with Stream API through Socket.IO. If you are not familiar with Stream API, be sure to check out the docs. I also recommend checking out the awesome Stream Handbook.