Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/shampoofactory/lzfse_rust
A pure Rust LZFSE implementation.
https://github.com/shampoofactory/lzfse_rust
compression compression-library decompression decompression-library lzfse rust rust-lang
Last synced: 3 months ago
JSON representation
A pure Rust LZFSE implementation.
- Host: GitHub
- URL: https://github.com/shampoofactory/lzfse_rust
- Owner: shampoofactory
- License: apache-2.0
- Created: 2021-04-06T16:12:16.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-04-06T14:07:42.000Z (over 1 year ago)
- Last Synced: 2024-10-01T17:08:48.870Z (3 months ago)
- Topics: compression, compression-library, decompression, decompression-library, lzfse, rust, rust-lang
- Language: Rust
- Homepage:
- Size: 3.25 MB
- Stars: 6
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE-APACHE
Awesome Lists containing this project
README
# lzfse_rust
Rust LZFSE implementation.
## Documentation
https://docs.rs/lzfse_rust
## Install
Simply configure your `Cargo.toml`:
```toml
[dependencies]
lzfse_rust = "0.2"
```
## Overview
This crate provides two LZFSE engines: one operating over user supplied memory buffers, and one operating over internal ring buffers.
The memory buffered engine works directly with input and output buffers that we supply.
It is exposed via `LzfseEncoder` and `LzfseDecoder` objects.
We would consider this engine when operating on `&[u8]` and `Vec` objects.
The ring buffered engine works by streaming data in and out of it's internal ring buffers.
It is exposed via `LzfseRingEncoder` and `LzfseRingDecoder` objects.
We would consider this engine when operating on IO streams, or when we want to expose a `Read` or `Write` interface.
Check the documentation for additional information and examples.
## Examples
This program compresses data from `stdin` into `stdout`. This example can be found in
`examples/compress_ring.rs`
```rust
use lzfse_rust::LzfseRingEncoder;
use std::io;
fn main() -> io::Result<()> {
let mut rdr = io::stdin();
let mut wtr = io::stdout();
let mut encoder = LzfseRingEncoder::default();
encoder.encode(&mut rdr, &mut wtr)?;
Ok(())
}
```
This program decompresses data from `stdin` into `stdout`. This example can be found in
`examples/decompress_ring.rs`
```rust
use lzfse_rust::LzfseRingDecoder;
use std::io;
fn main() -> io::Result<()> {
let mut rdr = io::stdin();
let mut wtr = io::stdout();
let mut decoder = LzfseRingDecoder::default();
decoder.decode(&mut rdr, &mut wtr)?;
Ok(())
}
```
# Command line tool: lzfoo
A fast, memory efficient and stream capable [lzfse](https://github.com/lzfse/lzfse) command line tool clone.
Powered by [lzfse_rust](https://github.com/shampoofactory/lzfse_rust).
Install.
```
$ cargo install lzfoo
```
Compress `a.txt` to `a.txt.lzfse`:
```
$ lzfoo -encode -i a.txt -o a.txt.lzfse
```
Compress with stdin/ stdout:
```
$ lzfoo -encode -i < a.txt > a.txt.lzfse
```
```
$ echo "semper fidelis" | lzfoo -encode > a.txt.lzfse
```
Decompress `a.txt.lzfse` to `a.txt`:
```
$ lzfoo -decode -i a.txt.lzfse -o a.txt
```
Decompress with stdin/ stdout:
```
$ lzfoo -decode -i < a.txt.lzfse > a.txt
```
Check the [lzfoo crate](https://github.com/shampoofactory/lzfse_rust/tree/main/lzfoo) for details.
## Testing
This crate comes with a comprehensive test suite that is divided into unit tests and integration tests.
The unit tests check that the library's internal components are working, whilst the integration tests check that library as a whole is working.
Extended tests are available but may take hours to complete.
Unit tests:
```
$ cargo test
```
Unit tests, extended:
```
$ cargo test -- --ignored
```
Integration tests:
```
$ cargo test --manifest-path test/Cargo.toml
```
Integration tests, extended:
```
$ cargo test --manifest-path test/Cargo.toml -- --ignored
```
Additional integration tests are available.
Notably, validation tests with the reference LZFSE implementation.
These are described in [test](https://github.com/shampoofactory/lzfse_rust/tree/main/test), along with instructions on how to build and run them.
## Performance
As a library and with the stated test machine, `lzfse_rust` outperforms `lzfse_ref`. However results will vary with different machines.
The benchmarks are powered with [Criterion](https://github.com/bheisler/criterion.rs) and formatted with [critcmp](https://github.com/BurntSushi/critcmp).
The machine is an 8GB Intel i5-2500K running [Ubuntu](https://ubuntu.com/) 18.04 (64 bit).
The dataset is taken from [Snappy](https://github.com/google/snappy).
The benchmark source is [here](https://github.com/shampoofactory/lzfse_rust/tree/main/bench), along with instructions on how to build and run the benchmarks.
```
group new/lzfse_ref/ new/rust/ new/rust_ring/
----- -------------- --------- --------------
decode/snap_uflat00_html 1.24 119.6±0.11µs 816.8 MB/sec 1.07 103.3±0.03µs 945.7 MB/sec 1.00 96.2±0.04µs 1014.9 MB/sec
decode/snap_uflat01_urls 1.18 1407.8±3.84µs 475.6 MB/sec 1.02 1211.8±0.11µs 552.5 MB/sec 1.00 1193.0±3.27µs 561.2 MB/sec
decode/snap_uflat02_jpg 1.07 353.7±0.05µs 331.9 MB/sec 1.00 330.7±0.91µs 355.0 MB/sec 1.02 336.1±0.85µs 349.3 MB/sec
decode/snap_uflat04_pdf 1.07 243.6±0.07µs 400.8 MB/sec 1.00 227.7±0.04µs 429.0 MB/sec 1.00 228.7±0.04µs 427.0 MB/sec
decode/snap_uflat05_html4 1.13 140.1±0.08µs 2.7 GB/sec 1.00 123.4±0.36µs 3.1 GB/sec 1.12 138.4±0.04µs 2.8 GB/sec
decode/snap_uflat06_txt1 1.22 469.4±0.05µs 309.0 MB/sec 1.09 420.8±1.21µs 344.7 MB/sec 1.00 384.3±0.05µs 377.4 MB/sec
decode/snap_uflat07_txt2 1.20 410.2±1.18µs 291.0 MB/sec 1.10 373.4±0.01µs 319.7 MB/sec 1.00 340.9±0.02µs 350.2 MB/sec
decode/snap_uflat08_txt3 1.25 1255.5±0.12µs 324.2 MB/sec 1.09 1096.9±3.32µs 371.0 MB/sec 1.00 1006.9±0.15µs 404.2 MB/sec
decode/snap_uflat09_txt4 1.18 1628.8±0.25µs 282.1 MB/sec 1.10 1511.5±3.03µs 304.0 MB/sec 1.00 1376.4±0.11µs 333.9 MB/sec
decode/snap_uflat10_pb 1.17 101.7±0.04µs 1112.3 MB/sec 1.04 90.2±0.03µs 1254.1 MB/sec 1.00 86.7±0.04µs 1304.3 MB/sec
decode/snap_uflat11_gaviota 1.28 486.0±0.05µs 361.7 MB/sec 1.09 413.2±0.05µs 425.4 MB/sec 1.00 379.5±0.03µs 463.1 MB/sec
encode/snap_uflat00_html 1.83 1500.0±3.76µs 65.1 MB/sec 1.00 821.1±0.09µs 118.9 MB/sec 1.10 905.9±0.11µs 107.8 MB/sec
encode/snap_uflat01_urls 1.45 13.1±0.00ms 51.3 MB/sec 1.00 9.0±0.00ms 74.2 MB/sec 1.01 9.1±0.00ms 73.8 MB/sec
encode/snap_uflat02_jpg 1.11 2.1±0.01ms 55.4 MB/sec 1.00 1910.0±5.33µs 61.5 MB/sec 1.12 2.1±0.00ms 54.7 MB/sec
encode/snap_uflat04_pdf 1.11 1695.1±0.19µs 57.6 MB/sec 1.00 1528.3±0.15µs 63.9 MB/sec 1.12 1705.8±0.16µs 57.3 MB/sec
encode/snap_uflat05_html4 5.10 4.4±0.00ms 89.7 MB/sec 1.00 854.4±0.08µs 457.2 MB/sec 1.12 954.2±0.11µs 409.4 MB/sec
encode/snap_uflat06_txt1 1.36 3.6±0.01ms 40.4 MB/sec 1.00 2.6±0.00ms 55.1 MB/sec 1.01 2.7±0.01ms 54.6 MB/sec
encode/snap_uflat07_txt2 1.34 3.1±0.01ms 38.5 MB/sec 1.01 2.3±0.00ms 51.2 MB/sec 1.00 2.3±0.00ms 51.8 MB/sec
encode/snap_uflat08_txt3 1.37 9.5±0.00ms 42.6 MB/sec 1.00 7.0±0.01ms 58.5 MB/sec 1.02 7.1±0.02ms 57.6 MB/sec
encode/snap_uflat09_txt4 1.34 12.3±0.04ms 37.3 MB/sec 1.00 9.3±0.00ms 49.7 MB/sec 1.00 9.2±0.00ms 49.9 MB/sec
encode/snap_uflat10_pb 1.95 1568.4±3.90µs 72.1 MB/sec 1.00 802.5±0.08µs 140.9 MB/sec 1.14 915.6±0.11µs 123.5 MB/sec
encode/snap_uflat11_gaviota 1.49 3.5±0.00ms 50.2 MB/sec 1.00 2.4±0.00ms 74.8 MB/sec 1.01 2.4±0.00ms 73.9 MB/sec
```
```
Key:
lzfse_ref: lzfse reference library
rust : lzfse_rust
rust_ring: lzfse_rust ring
Column: 1 2 3
1: relative time lower is better, 1.00 is the fastest
2: mean time ± standard deviation lower is better
3: throughput higher is better
```
## Minimum Rust version policy
This crate's minimum supported `rustc` version is `1.51.0`.
## License
Licensed under either of
* Apache License, Version 2.0
([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
* MIT license
([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
at your option.
## Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in the work by you, as defined in the Apache-2.0 license, shall be
dual licensed as above, without any additional terms or conditions.
## Alternatives
* [lzfse-rs](https://github.com/citruz/lzfse-rs) - bindings to the reference [LZFSE](https://github.com/lzfse/lzfse) implementation.