{"id":16338896,"url":"https://github.com/vorner/pgz","last_synced_at":"2025-06-24T15:32:25.057Z","repository":{"id":66117426,"uuid":"91106994","full_name":"vorner/pgz","owner":"vorner","description":"Parallel gzip","archived":false,"fork":false,"pushed_at":"2022-06-18T13:29:13.000Z","size":22,"stargazers_count":12,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-20T00:07:12.236Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vorner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-12T15:50:45.000Z","updated_at":"2025-04-07T19:37:50.000Z","dependencies_parsed_at":"2023-02-20T18:01:01.868Z","dependency_job_id":null,"html_url":"https://github.com/vorner/pgz","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vorner/pgz","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vorner%2Fpgz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vorner%2Fpgz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vorner%2Fpgz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vorner%2Fpgz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vorner","download_url":"https://codeload.github.com/vorner/pgz/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vorner%2Fpgz/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261703200,"owners_count":23196920,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T23:53:01.901Z","updated_at":"2025-06-24T15:32:25.050Z","avatar_url":"https://github.com/vorner.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The parallel gzip\n\nThis is an implementation of a parallel gzip. It works by splitting the input\ninto chunks (by default by 32MBs, but this can be configured).  Each chunk is\ncompressed independently and the results are concatenated together. Such result\ncan be read and decompressed by the usual gzip implementation.\n\nThe motivation is to speed up transfers of large amounts of data across a fast\nnetwork through ssh. The ssh throughput is limited by either its compression or\nencryption routines, which are single-threaded. This allows turning compression\noff in ssh and using multiple cores to compress the data. As the decompression\nis much faster, it is not necessary to use parallel *decompression*.\n\n## Limitations\n\nThere are certain limitations:\n\n* The compressed representation is slightly different than from the usual\n  sequential gzip. Technically, the output is multiple concatenated gzips, but\n  decompression tools commonly accept that. Furthermore, due to the\n  independent chunks, the compression ratio is likely to be a bit worse.\n* It uses more memory, to buffer the chunks.\n\n## License\n\nLicensed under either of\n\n * Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)\n * MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)\n\nat your option.\n\n### Contribution\n\nUnless you explicitly state otherwise, any contribution intentionally\nsubmitted for inclusion in the work by you, as defined in the Apache-2.0\nlicense, shall be dual licensed as above, without any additional terms\nor conditions.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvorner%2Fpgz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvorner%2Fpgz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvorner%2Fpgz/lists"}