{"id":16729102,"url":"https://github.com/keithduncan/catbus","last_synced_at":"2025-10-14T08:13:41.471Z","repository":{"id":144668838,"uuid":"137311256","full_name":"keithduncan/catbus","owner":"keithduncan","description":"Fast tarball transport. Content-aware transport for tar archives modelled on git upload-pack and receive-pack.","archived":false,"fork":false,"pushed_at":"2019-02-08T11:51:53.000Z","size":975,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-15T16:37:16.272Z","etag":null,"topics":["deployment","file-upload","network","rust-crate","tarball"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/keithduncan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-14T05:42:10.000Z","updated_at":"2024-07-22T11:02:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"cdd5ba1f-fdc3-47da-9288-39af2c2a4eac","html_url":"https://github.com/keithduncan/catbus","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/keithduncan/catbus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithduncan%2Fcatbus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithduncan%2Fcatbus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithduncan%2Fcatbus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithduncan%2Fcatbus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/keithduncan","download_url":"https://codeload.github.com/keithduncan/catbus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithduncan%2Fcatbus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279018307,"owners_count":26086334,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deployment","file-upload","network","rust-crate","tarball"],"created_at":"2024-10-12T23:26:56.762Z","updated_at":"2025-10-14T08:13:41.450Z","avatar_url":"https://github.com/keithduncan.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# catbus\n\nFast tarball transport. A content-aware transport for tar archives modelled on git `upload-pack` / `receive-pack`.\n\n![](nekobasu.gif)\n\nSave bytes and time when transferring a sequence of tar archives containing\nlargely the same content over time by sourcing already received parts from\na library-of-parts cache.\n\ncatbus allows you to slim down the bytes necessary to transport a given tar archive by\nconstructing an archive on the remote end from previously received tar archives.\n\n## How it works\n\n![](protocol.png)\n\n_Generated using https://sequencediagram.org/_\n\n1. An index of the tar archive to be transport is precomputed and stored beside the tar archives using `catbus index`.\n1. A receiver invokes `catbus transport receive-index` connected to a sender `catbus transport upload-index`. A reliable transport is assumed, such as SSH tunnels.\n1. The sender writes the index to the receiver\n1. The receiver tries to source the parts from the previously received local library of parts\n1. The receiver requests missing parts from the sender\n1. The sender generates an in-memory archive of the missing parts and sends it\n1. The receiver inserts the missing parts and serialises the full archive\n\n## How to build it\n\ncatbus is built using cargo and rust.\n\n```\ncd ~/catbus\ncargo build\n```\n\n## When to use it\n\ncatbus may be useful when you have a library of archives like application builds that\nyou need to transport to a fleet of hosts in a network efficient manner. By pre-computing\nthe archives from local parts we trade CPU for network bandwidth.\n\nA minimal set of parts is then transported to each host requesting the archive based on\nthe parts they need. A new host with no local parts will simply request the entire archive.\n\nThis is similar to how git negotiates the objects needed for a fetch but with a simpler\nreplacement for repacking the object database over time. Instead, old parts are pruned\nwhen the transport cache is expired on a first in first out basis based on age.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeithduncan%2Fcatbus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkeithduncan%2Fcatbus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeithduncan%2Fcatbus/lists"}