https://github.com/vorner/pgz
Parallel gzip
https://github.com/vorner/pgz
Last synced: about 1 year ago
JSON representation
Parallel gzip
- Host: GitHub
- URL: https://github.com/vorner/pgz
- Owner: vorner
- License: apache-2.0
- Created: 2017-05-12T15:50:45.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2022-06-18T13:29:13.000Z (about 4 years ago)
- Last Synced: 2025-06-20T00:07:12.236Z (about 1 year ago)
- Language: Rust
- Size: 21.5 KB
- Stars: 12
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE-APACHE
Awesome Lists containing this project
README
# The parallel gzip
This is an implementation of a parallel gzip. It works by splitting the input
into chunks (by default by 32MBs, but this can be configured). Each chunk is
compressed independently and the results are concatenated together. Such result
can be read and decompressed by the usual gzip implementation.
The motivation is to speed up transfers of large amounts of data across a fast
network through ssh. The ssh throughput is limited by either its compression or
encryption routines, which are single-threaded. This allows turning compression
off in ssh and using multiple cores to compress the data. As the decompression
is much faster, it is not necessary to use parallel *decompression*.
## Limitations
There are certain limitations:
* The compressed representation is slightly different than from the usual
sequential gzip. Technically, the output is multiple concatenated gzips, but
decompression tools commonly accept that. Furthermore, due to the
independent chunks, the compression ratio is likely to be a bit worse.
* It uses more memory, to buffer the chunks.
## License
Licensed under either of
* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
* MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
at your option.
### Contribution
Unless you explicitly state otherwise, any contribution intentionally
submitted for inclusion in the work by you, as defined in the Apache-2.0
license, shall be dual licensed as above, without any additional terms
or conditions.