{"id":15460275,"url":"https://github.com/schollz/patchitup","last_synced_at":"2025-04-22T10:43:35.188Z","repository":{"id":57590884,"uuid":"122505260","full_name":"schollz/patchitup","owner":"schollz","description":"Backup your file to your remote server using minimum bandwidth.","archived":false,"fork":false,"pushed_at":"2018-02-26T20:20:10.000Z","size":98,"stargazers_count":23,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-11T19:06:45.832Z","etag":null,"topics":["cloud-storage","golang-library","patching"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/schollz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-22T16:35:55.000Z","updated_at":"2021-02-11T04:07:22.000Z","dependencies_parsed_at":"2022-09-26T19:43:02.881Z","dependency_job_id":null,"html_url":"https://github.com/schollz/patchitup","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schollz%2Fpatchitup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schollz%2Fpatchitup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schollz%2Fpatchitup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schollz%2Fpatchitup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/schollz","download_url":"https://codeload.github.com/schollz/patchitup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249259146,"owners_count":21239422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-storage","golang-library","patching"],"created_at":"2024-10-01T23:21:19.429Z","updated_at":"2025-04-16T16:31:46.585Z","avatar_url":"https://github.com/schollz.png","language":"Go","funding_links":["https://www.paypal.me/ZackScholl/5.00"],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003cimg\n    src=\"https://raw.githubusercontent.com/schollz/patchitup/master/.github/logo.png?token=AGPyE539EM60WRIJo3VVDBASqzozHprrks5amLWtwA%3D%3D\"\n    width=\"260px\" border=\"0\" alt=\"patchitup\"\u003e\n\u003cbr\u003e\n\u003ca href=\"https://github.com/schollz/patchitup/releases/latest\"\u003e\u003cimg src=\"https://img.shields.io/badge/version-0.1.0-brightgreen.svg?style=flat-square\" alt=\"Version\"\u003e\u003c/a\u003e\n\u003cimg src=\"https://img.shields.io/badge/coverage-75%25-green.svg?style=flat-square\" alt=\"Code Coverage\"\u003e\n\u003ca href=\"https://godoc.org/github.com/schollz/patchitup/patchitup\"\u003e\u003cimg src=\"https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square\" alt=\"Code Coverage\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.paypal.me/ZackScholl/5.00\"\u003e\u003cimg src=\"https://img.shields.io/badge/donate-$5-brown.svg?style=flat-square\" alt=\"Donate\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003eBackup your file to a cloud server using minimum bandwidth.\u003c/p\u003e\n\n*patchitup* is a way to keep the cloud up-to-date through incremental patches. In a nutshell, this is a pure-Golang library and a CLI tool for creating a client+server that exchange incremental gzipped patches to overwrite a remote copy to keep it up-to-date with the client's local file. \n\n\u003cem\u003e\u003cstrong\u003eWhy?\u003c/strong\u003e\u003c/em\u003e I wrote this program to reduce the bandwidth usage when backing up SQLite databases to a remote server from Raspberry Pis. I have deployed some software on Raspberry Pis that periodically [dumps the database to SQL text](http://www.sqlitetutorial.net/sqlite-dump/). Since Raspberry Pi's can die sometimes, I want to keep their data stored remotely. As the databases can get fairly large, a patch from SQL text will only ever be the changed/new records. *patchitup* allows the client to just send to the cloud only the changed/new records and still maintain the exact copy on the cloud. This can massively reduce bandwidth between the client and the cloud. \n\n\u003cem\u003e\u003cstrong\u003eWhy not git?\u003c/strong\u003e\u003c/em\u003e While *git* basically  does this already, its not terribly easy to setup a *git* server to support multiple users (though [gitolite](https://github.com/sitaramc/gitolite) does a great job of simplifying the process).  Also, most of the features of *git* are not necessary for my use-case.\n\n_Note: for an encrypted version see [schollz/patchitup-encrypted](https://github.com/schollz/patchitup-encrypted)._\n\n# Quickstart\n\nIn addition to being a Golang library, the *patchitup* is a server+client. To try it, first install *patchitup* with Go:\n\n```\n$ go install -u -v github.com/schollz/patchitup/...\n```\n\nThen start a *patchitup* server:\n\n```\n$ patchitup -host\nRunning at http://0.0.0.0:8002\n```\n\nThen you can patch a file:\n\n```\n$ patchitup -u me -s http://localhost:8002 -f SOMEFILE\n2018-02-23 08:56:44 [INFO] patched 2.4 kB (62.8%) to remote 'SOMEFILE' for 'me'\n2018-02-23 08:56:44 [INFO] remote server is up-to-date\n\n$ vim SOMEFILE # make some edits\n\n$ patchitup -u me -s http://localhost:8002 -f SOMEFILE\n2018-02-23 08:57:40 [INFO] patched 408 B (9.9%) to remote 'SOMEFILE' for 'me'\n2018-02-23 08:57:40 [INFO] remote server is up-to-date\n```\n\nThe first time you patch will basically just send up the gzipped file. Subsequent edits will just send up the patches. The percentage (e.g. `9.9%`) specifies the percentage of the entire file size that is being sent (to get an idea of bandwidth savings). The server also will log bandwidth usage.\n\n\n# How does it work?\n\n_Note:_ *patchitup* does **not** work for binary files (yet).\n\nWhy not just do \"`diff -u old new \u003e patch \u0026\u0026 rsync patch your@server:`\"? Well, *patchitup* keeps things organized a lot better and uses `gzip` by default to reduce the bandwidth cost even further. Also, in order to patch a remote file you first need a copy of the remote file to create the patch. In *patchitup*, if the local copy of remote file is not available, a local copy of the remote file is reconstructed it in a way that can massively reduce bandwidth (i.e. instead of just downloading the remote file). To reconstruct a local copy of remote file:\n\n1. The client asks the remote server for a hash of every line and its corresponding line number in the remote file. \n2. The client checks to see if any lines are needed (i.e. the set of line hashes that do not exist in the current local file). The client then asks the remote server for the actual lines corresponding to the missing hashes.\n3. The client uses these data (the local line hashes, the remote line hashes, and the hash line numbers) to reconstruct a copy of the remote file for doing the patching.\n\nOnce the local copy of the remote file is established, a patch is created and gzipped and sent to the server for overwriting the current remote copy. A current remote copy is cached locally so that it need not be reconstructed the next time.\n\nA more detailed flow chart:\n\n\u003ccenter\u003e\n\t\u003cimg src=\"https://user-images.githubusercontent.com/6550035/36574282-e0335014-17f9-11e8-92ba-1a474deaae76.png\"\u003e\n\u003c/center\u003e\n\n# Roadmap\n\nI would love PRs.\n\nSome ideas I'd like to add:\n\n- [ ] Built-in security (authentication tokens?)\n- [ ] Encryption option (to keep data on server private)\n\n# License\n\nMIT\n\n# Thanks\n\nLogo designed by \u003ca rel=\"nofollow\" target=\"_blank\" href=\"https://www.vecteezy.com\"\u003ewww.Vecteezy.com\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschollz%2Fpatchitup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fschollz%2Fpatchitup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschollz%2Fpatchitup/lists"}