{"id":13646838,"url":"https://github.com/kaiakz/rsync-os","last_synced_at":"2026-03-12T15:39:32.071Z","repository":{"id":47746265,"uuid":"267630117","full_name":"kaiakz/rsync-os","owner":"kaiakz","description":"A rsync-compatible tool for object storage","archived":false,"fork":false,"pushed_at":"2022-04-05T07:34:46.000Z","size":613,"stargazers_count":47,"open_issues_count":4,"forks_count":11,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-12-18T11:28:50.162Z","etag":null,"topics":["rsync","rsync-protocol","s3-storage"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kaiakz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-28T15:43:55.000Z","updated_at":"2025-09-01T00:43:29.000Z","dependencies_parsed_at":"2022-07-22T02:18:23.494Z","dependency_job_id":null,"html_url":"https://github.com/kaiakz/rsync-os","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/kaiakz/rsync-os","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaiakz%2Frsync-os","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaiakz%2Frsync-os/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaiakz%2Frsync-os/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaiakz%2Frsync-os/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kaiakz","download_url":"https://codeload.github.com/kaiakz/rsync-os/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaiakz%2Frsync-os/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30431026,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-12T14:34:45.044Z","status":"ssl_error","status_checked_at":"2026-03-12T14:09:33.793Z","response_time":114,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["rsync","rsync-protocol","s3-storage"],"created_at":"2024-08-02T01:03:09.489Z","updated_at":"2026-03-12T15:39:32.024Z","avatar_url":"https://github.com/kaiakz.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# RSYNC-OS\n## A port of rsync written in Go, with object storage support\n\n# Introduction\nrsync-os is a Golang implementation of Rsync built from scratch. Its goal is to be a modern rsync-compatible tool with features like:\n* Supports file storage and object storage\n* Uses a subset of rsync wire protocol to avoid blocks efficient transfer (For sender and receiver)\n* Rsync APIs\n* Safe \n\n![client](https://raw.githubusercontent.com/kaiakz/rsync-os/master/docs/client.jpg)\n\n## Usage\n### Use minio as backend\n1. install \u0026 run minio, you need to configure the `config.toml`.\n2. `go build`\n3. `./rsync-os rsync://[USER@]HOST[:PORT]/SRC minio`, for example, `./rsync-os rsync://mirrors.tuna.tsinghua.edu.cn/ubuntu minio`\n\n## Roadmap\n### Client\n#### Rsync wire protocol 27:\n- [x] Connect to rsync daemon by rsync://: Handshake, Fetches the file list, Requests \u0026 download files\n- [ ] Handles error\n#### Storage backend\n- [x] Minio: supports regular files, supports folder \u0026 symlink\n- [x] Minio: caches file list in boltdb\n- [ ] Local\n#### Other\n- [x] CLI\n\n## Detailed Information\n#### openrsync has a really good [documentation](https://github.com/kristapsdz/openrsync/blob/master/README.md) to describe how rsync algorithm works. \n\n#### Why do this?\nJust as [rsyn](https://github.com/sourcefrog/rsyn#why-do-this) said, \"The rsync C code is quite convoluted, with many interacting options and parameters stored in global variables affecting many different parts of the control flow, including how structures are encoded and decoded.\" I would like to provide a rsync written in clean and understandable Golang code. \n\nrsync has [a bad performance](https://github.com/tuna/rsync/blob/master/README-huai.md). Inspired by rsync-huai, rsync-os stores the file list in database to avoid recursively generating the list.\n\nModernized rsync: rsync-os supports both file storage and object storage.\n\n#### What's the difference between rsync and rsync-os\nrsync-os is the express edition of rsync, with object storage support. It uses a subset of rsync wire protocol(without rolling block checksum).\n\n#### rsync-os and rclone are completely different\nrclone does not support rsync wire protocol although it is called \"rsync for object storage\". With rclone you can't transfer files between rsync and object storage.\n\n#### Why we don't need rolling block checksum for regular file?\nIn the rsync algorithm, rsync requires random access of files to do the block exchange. But object storage does not support that.\n\nrsync-os simplifies the rsync algorithm to avoid random reading and writing, since rsync-os don't need to do a rolling checksum scanning the file. \n\nAs a client, when a file has different size or modified time compared to the remote file, rsync-os just pretend 'the file does not exist here', then send a reply to download the entire file from the server and finally replace it.\n\nAs a server, TBC\n\n#### HandShake\nrysnc-os supports rsync protocol 27. \nNow it sends the arguments \"--server--sender-l-p-r-t\" to the remote rsyncd by default.\n\n#### The File List\nAccording to the arguments rsync-os sent, the file list should contain path, size, modified time \u0026 mode. \n \n#### Request the file\nrsync-os always saves the file list in its database(local file list). rsync2os doesn't compare each file with the file list from remote server(remote file list), but the latest local file list. If the file in the local file list has different size, modified time, or doesn't exist, rsync2os will download the whole file(without [block exchange](https://github.com/kristapsdz/openrsync#block-exchange)). To to do that, rsync2os sends the empty block checksum with the file's index of the remote file list. \n\n#### Download the file\nThe rsync server sends the entire file as a stream of bytes.\n\n#### Multiplex \u0026 De-Multiplex\nMost rsync transmissions are wrapped in a multiplexing envelope protocol. The code written in C to multiplex \u0026 de-multiplex is obscure. \nUnlike rsync, rsync-os reimplements this part: It just does multiplexing \u0026 de-multiplexing in a goroutine.\n![de-multiplex](https://raw.githubusercontent.com/kaiakz/rsync-os/master/docs/demux.jpg)\n\n### Limitations\n* Do not support block exchange for regular files. If a file was modified, just downloads the whole file.\n* rsync-os can only act as client/receiver now.\n\n# Reference\n* [rsync](https://rsync.samba.org/)\n* [openrsync](https://github.com/openbsd/src/tree/master/usr.bin/rsync), a BSD-liscesed rsync\n* [rsync-huai](https://github.com/tuna/rsync), a modified version rsync by Tsinghua University TUNA Association\n* [yajsync](https://github.com/perlundq/yajsync), a port of rsync written in Java\n* [rsyn](https://github.com/sourcefrog/rsyn), wire-compaible rsync in Rust\n* [acrosync-library](https://github.com/gilbertchen/acrosync-library)\n* [repositoryd](https://github.com/APNIC-net/repositoryd), An rsync-compatible RPKI repository daemon.\n* https://rsync.samba.org/resources.html\n* https://github.com/boundary/wireshark/blob/master/epan/dissectors/packet-rsync.c\n* https://tools.ietf.org/html/rfc5781","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaiakz%2Frsync-os","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkaiakz%2Frsync-os","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaiakz%2Frsync-os/lists"}