{"id":13509894,"url":"https://github.com/Redundancy/go-sync","last_synced_at":"2025-03-30T14:31:54.753Z","repository":{"id":16368466,"uuid":"19118716","full_name":"Redundancy/go-sync","owner":"Redundancy","description":"gosync is a library for Golang styled around zsync / rsync, written with the intent that it enables efficient differential file transfer in a number of ways. NB: I am unable to contribute to this at the moment","archived":false,"fork":false,"pushed_at":"2020-08-08T16:12:10.000Z","size":201,"stargazers_count":582,"open_issues_count":14,"forks_count":66,"subscribers_count":23,"default_branch":"master","last_synced_at":"2024-11-01T10:35:04.224Z","etag":null,"topics":["binary-data","file-transfer","go","rsync","zsync"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Redundancy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-04-24T18:07:24.000Z","updated_at":"2024-09-07T20:10:00.000Z","dependencies_parsed_at":"2022-08-28T13:11:11.087Z","dependency_job_id":null,"html_url":"https://github.com/Redundancy/go-sync","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Redundancy%2Fgo-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Redundancy%2Fgo-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Redundancy%2Fgo-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Redundancy%2Fgo-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Redundancy","download_url":"https://codeload.github.com/Redundancy/go-sync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246332247,"owners_count":20760448,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary-data","file-transfer","go","rsync","zsync"],"created_at":"2024-08-01T02:01:16.240Z","updated_at":"2025-03-30T14:31:52.171Z","avatar_url":"https://github.com/Redundancy.png","language":"Go","funding_links":[],"categories":["Go","go"],"sub_categories":[],"readme":"Go-Sync\n------\n[![Build Status](https://travis-ci.org/Redundancy/go-sync.svg?branch=master)](https://travis-ci.org/Redundancy/go-sync)\n[![GoDoc](https://godoc.org/github.com/Redundancy/go-sync?status.svg)](https://godoc.org/github.com/Redundancy/go-sync)\n\n# The Command-line tool has moved!\nIn order to split issues between the library and the CLI tool, as well as correctly vendor dependencies, the command-line tool code has been moved to its own repository: https://github.com/Redundancy/gosync-cmd\n\n# Why *not* use a Zsync mechanism?\n\nConsider if a binary differential sync mechanism is appropriate to your use case:\n\nThe ZSync mechanism has the weakness that HTTP1.1 ranged requests are not always well supported by CDN providers and ISP proxies. When issues happen, they're very difficult to respond to correctly in software (if possible at all). Using HTTP 1.0 and fully completed GET requests would be better, if possible.\n\nThere are some other issues too - ZSync doesn't (as far as I'm aware) solve any issues to do with storage of a files, which can get more and more onerous for large files that are not changing much from one version to another.\n\nOn a project I worked on, we switched instead to storing individual files that were part of a larger build (like an ISO) by filename and hashes, mainly maintaining an index of which files comprised the full build. By doing this, we significantly decreased the required storage (new files were only required when they changed), allowed multiple versions to sit efficiently side by side and very simple file serving to be used efficiently (with a tiny library to resolve and fetch files).\n\n# The GoSync library\n\ngosync is a library inspired by zsync and rsync.\nHere are the goals:\n\n### Fast\nUsing the concurrency and performance features of Golang, Go-sync is designed to take advantage of multiple processors and multiple HTTP connections to make the most of modern hardware and minimize the impact of the bandwidth latency product.\n\n### Cross Platform\nWorks on Windows and Linux, without cygwin or fuss.\n\n### Easy\n\nA new high-level interface designed to reduce the work of implementing block transfer in your application:\n```golang\nfs := \u0026BasicSummary{...}\n\nrsync, err := MakeRSync(\n    localFilename,\n    referencePath,\n    outFilename,\n    fs,\n)\n\nif err != nil {\n    return err\n}\n\nerr = rsync.Patch()\n\nif err != nil {\n    return err\n}\n\nreturn rsync.Close()\n```\n\n### Extensible\nAll functionality is based on interfaces, allowing customization of behavior:\n\n```golang\n// Here, the input version is a local string\ninputFile := bytes.NewReader(localVersionAsBytes)\n\n// And the output is a buffer\npatchedFile := bytes.NewBuffer(nil)\n\n// This information is meta-data on the file that should be loaded / provided\n// You can also provide your own implementation of the FileSummary interface\nsummary := \u0026BasicSummary{\n    ChecksumIndex:  referenceFileIndex,\n    // Disable verification of hashes for downloaded data (not really a good idea!)\n    ChecksumLookup: nil,\n    BlockCount:     uint(blockCount),\n    BlockSize:      blockSize,\n    FileSize:       int64(len(referenceAsBytes)),\n}\n\nrsync := \u0026RSync{\n    Input:  inputFile,\n    Output: patchedFile,\n    // An in-memory block source\n    Source: blocksources.NewReadSeekerBlockSource(\n        bytes.NewReader(referenceAsBytes),\n        blocksources.MakeNullFixedSizeResolver(uint64(blockSize)),\n    ),\n    Index:   summary,\n    Summary: summary,\n    OnClose: nil,\n}\n```\n\nReuse low level objects to build a new high level library, or implement a new lower-level object to add a new transfer protocol (for example).\n\n### Tested\nGoSync has been built from the ground up with unit tests.\nThe GoSync command-line tool has acceptance tests, although not everything is covered.\n\n**HOWEVER** this library has never been used in production against real-world network problems, and I cannot personally guarantee that it will work as intended.\n\n## Current State\nThe GoSync library is fairly well unit-tested, but not tested through exposure to real-world network conditions. As an example, the HTTP client used is a default HTTP client, and is therefore lacking decent timeouts. As such, I would not recommend depending on the code in production unless you're willing to validate the results and debug issues like that.\n\nIn terms of activity, I have been extremely busy with other things for the last few months, and will continue to be. I do not expect to put a huge amount more work into this, since we solved our problem in a simpler (and significant more elegant) way explained in a section above.\n\n### Request for Enhancement\nIf the library or tool are still something that you feel would be useful, here are some issues and ideas for work that could be done.\n\n#### GZip support - Performance / Efficiency Enhancement (!)\nIn order to be more efficient in the transfer of data from the source to the client, gosync should support compressed blocks. This requires changing any assumptions about the offset of a block, and the length of a block to read (especially when merging block ranges), then adding a compression / decompression call to the interfaces.\n\nIn terms of the CLI tool, this probably means that gosync should build a version of the source file where each block is independently compressed and store the block-sizes in the index. It can then rebuild the offsets incrementally.\n\n#### Patch payloads - Feature\nGiven a known original version, and a known desired state, it would be possible to create a \"patch\", which has enough information to store the required blocks for the transformation only, and only enough of the index to validate that it's transforming the correct file.\n\n#### Patched file Validation - Feature (!)\nGoSync should validate the full MD5 and length of a file after it is done with patching it. This should be minimally expensive, and help increase confidence that GoSync has produced the correct result.\n\nThis one is pretty simple. :)\n\n#### Network Error handling - Improvement (!!)\nThe HTTP Blocksource does not handle connection / read timeouts and other myriad possible network failures . Handling these correctly is important to making it robust and production-ready.\n\nRolled into this is to correctly identify resumable errors (including rate-limiting, try-again-later and temporary errors) and back-off strategies.\n\n#### Rate limiting - Feature\nIn order to be a good network denizen, GoSync should be able to support rate-limiting.\n\n#### Better / Consistent naming - Improvement\nThe current naming of some packages and concepts is a bit broken. The RSync object, for example, has nothing to do with RSync. Blocks and Chunks are used interchangeably for a byte range.\n\n### Testing\n\nAll tests are run by Travis-CI\n\n#### Unit tests\n    go test github.com/Redundancy/go-sync/...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRedundancy%2Fgo-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRedundancy%2Fgo-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRedundancy%2Fgo-sync/lists"}