{"id":22281084,"url":"https://github.com/killingspark/zstd-rs","last_synced_at":"2025-05-14T09:08:22.868Z","repository":{"id":37587888,"uuid":"207613379","full_name":"KillingSpark/zstd-rs","owner":"KillingSpark","description":"zstd format implementation in pure rust","archived":false,"fork":false,"pushed_at":"2025-04-01T11:06:13.000Z","size":27906,"stargazers_count":297,"open_issues_count":10,"forks_count":35,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-03T02:05:51.092Z","etag":null,"topics":["compression","decompression","rust","zstd"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KillingSpark.png","metadata":{"files":{"readme":"Readme.md","changelog":"Changelog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-10T16:48:19.000Z","updated_at":"2025-04-02T23:11:16.000Z","dependencies_parsed_at":"2024-04-18T18:30:39.062Z","dependency_job_id":"fdb0627a-938d-47a8-9207-779b624ea53f","html_url":"https://github.com/KillingSpark/zstd-rs","commit_stats":{"total_commits":329,"total_committers":11,"mean_commits":29.90909090909091,"dds":0.1550151975683891,"last_synced_commit":"fa7bd9c7b34343679399d88e6234da495fd997ad"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KillingSpark%2Fzstd-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KillingSpark%2Fzstd-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KillingSpark%2Fzstd-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KillingSpark%2Fzstd-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KillingSpark","download_url":"https://codeload.github.com/KillingSpark/zstd-rs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248154967,"owners_count":21056541,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","decompression","rust","zstd"],"created_at":"2024-12-03T16:13:47.078Z","updated_at":"2025-04-10T03:43:49.011Z","avatar_url":"https://github.com/KillingSpark.png","language":"Rust","readme":"# Ruzstd (a pure rust zstd format implementation)\n\n[![Released API docs](https://docs.rs/ruzstd/badge.svg)](https://docs.rs/ruzstd)\n[![CI](https://github.com/killingspark/zstd-rs/workflows/CI/badge.svg)](https://github.com/killingspark/zstd-rs/actions?query=workflow%3ACI)\n\n\n# What is this\n\nA pure Rust implementation of the Zstandard compression format, as defined in [RFC8878](https://www.rfc-editor.org/rfc/rfc8878.pdf).\n\nThis crate contains a fully operational implementation of the decompression portion of the standard.\nIt also provides a compressor which is usable, but it does not yet reach the speed, ratio or configurability of the original zstd library.\n\nThis crate is currently actively maintained.\n\n# Current Status\n\nFeature complete on the decoder side.\n\nOn the compression side:\n- Support for generating compressed blocks at any compression level\n  - [x] Uncompressed\n  - [x] Fastest (roughly level 1)\n  - [ ] Default (roughly level 3)\n  - [ ] Better (roughly level 7)\n  - [ ] Best (roughly level 11)\n- [ ] Checksums\n- [ ] Dictionaries\n\n## Speed\nIn terms of speed this library is behind the original C implementation which has a rust binding located [here](https://github.com/gyscos/zstd-rs).\n\nMeasuring with the 'time' utility the original zstd and my decoder both decoding the same enwik9.zst file from a ramfs, my decoder is about 3.5 times slower. Enwik9 is highly compressible, for less compressible data (like a ubuntu installation .iso) my decoder comes close to only being 1.4 times slower.\n\n\n# How can you use it?\n\n## Compression\n\nThe easiest is to use the provided `compress`/`compress_to_vec` functions\n\n```rust, no_run\nuse ruzstd::encoding::{compress, compress_to_vec, CompressionLevel};\nlet data: \u0026[u8] = todo!();\n// Either\nlet mut compressed = Vec::new();\ncompress(data, \u0026mut compressed, CompressionLevel::Fastest);\n// or\nlet compressed = compress_to_vec(data, CompressionLevel::Fastest);\n ```\n\n Or you can use the `FrameDecoder` manually to compress data. This allows you to process encoded data while it is being encoded instead of collecting into a big vector.\n\n## Decompression\n\nAdditionally to the descriptions and the docs you can have a look at the zstd / zstd_streaming binaries. They showcase how this library can be used.\n\n### Easy\n\nThe easiest is to wrap the io::Read into a StreamingDecoder which itself implements io::Read. It will decode blocks as necessary to fulfill the read requests\n\n```rust, no_run\nuse ruzstd::decoding::StreamingDecoder;\nuse ruzstd::io::Read;\n\nlet mut source: \u0026[u8] = todo!(\"Get a reader from a File or any other source\");\nlet mut decoder = StreamingDecoder::new(\u0026mut source).unwrap();\n\nlet mut result = Vec::new();\ndecoder.read_to_end(\u0026mut result).unwrap();\n```\n\nThis might be a problem if you are accepting user provided data. Frames can be REALLY big when decoded. If this is the case you should either check how big the frame\nactually is or use the memory efficient approach described below.\n\n### Memory efficient\n\nIf memory is a concern you can decode frames partially. There are two ways to do this:\n\n#### Streaming decoder\n\nUse the StreamingDecoder and use a while loop to fill your buffer (see src/bin/zstd_stream.rs for an example). This is the\nrecommended approach.\n\n#### Use the lower level FrameDecoder\n\nFor an example see the src/bin/zstd.rs file. Basically you can decode the frame until either a\ngiven block count has been decoded or the decodebuffer has reached a certain size. Then you can collect no longer needed bytes from the buffer and do something with them, discard them and resume decoding the frame in a loop until the frame has been decoded completely.\n\n## Roadmap\n\n1. More Performance optimizations\n    1. sequence_decoding and reverse_bitreader::get_bits. Those account for about 50% of the whole time used in decoding\n    2. Matching suffixes. This accounts for \u003e60% of the whole time used in encoding\n2. Implement encoder features\n    1. More levels\n    2. Dictionaries\n    3. Checksums\n\n## Testing\n\nTests take two forms.\n\n1. Tests using well-formed files that have to decode correctly and are checked against their originals\n1. Tests using malformed input that have been generated by the fuzzer. These don't have to decode (they are garbage) but they must not make the decoder panic\n\n## Fuzzing\n\nFuzzing has been done on\n\n1. Random input with no initial corpus\n2. The \\*.zst in /fuzz_decodecorpus\n\n\n### You want to help fuzz?\n\nUse `cargo +nightly fuzz run decode` or some other fuzz target to run the fuzzer. It is seeded with files created with decodecorpus.\n\nIf the fuzzer finds a crash it will be saved to the artifacts dir by the fuzzer. Run `cargo test artifacts` to run the artifacts tests.\nThis will tell you where the decoder panics exactly. If you are able to fix the issue please feel free to do a pull request. If not please still submit the offending input and I will see how to fix it myself.\n\n# Contributing\n\nContributions will be published under the same MIT license as this project. Please make an entry in the Changelog.md file when you make a PR.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkillingspark%2Fzstd-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkillingspark%2Fzstd-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkillingspark%2Fzstd-rs/lists"}