{"id":46602767,"url":"https://github.com/dupontcyborg/compress-utils","last_synced_at":"2026-05-16T05:23:15.986Z","repository":{"id":258925480,"uuid":"873627188","full_name":"dupontcyborg/compress-utils","owner":"dupontcyborg","description":"Multi-algorithm compression \u0026 decompression library for C, C++ \u0026 Python","archived":false,"fork":false,"pushed_at":"2025-12-13T16:58:01.000Z","size":446,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-06T09:40:09.708Z","etag":null,"topics":["brotli","bzip2","c","compression","cpp","decompression","gzip","lz4","lzma","python","xz","zlib","zstd"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dupontcyborg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-16T13:35:23.000Z","updated_at":"2025-12-07T04:07:17.000Z","dependencies_parsed_at":"2024-10-23T05:16:36.058Z","dependency_job_id":"d997991d-2bb9-4a12-8127-140ef4f2450b","html_url":"https://github.com/dupontcyborg/compress-utils","commit_stats":null,"previous_names":["dupontcyborg/compression-utils"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/dupontcyborg/compress-utils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dupontcyborg%2Fcompress-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dupontcyborg%2Fcompress-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dupontcyborg%2Fcompress-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dupontcyborg%2Fcompress-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dupontcyborg","download_url":"https://codeload.github.com/dupontcyborg/compress-utils/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dupontcyborg%2Fcompress-utils/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30221511,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T14:02:48.375Z","status":"ssl_error","status_checked_at":"2026-03-07T14:02:43.192Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["brotli","bzip2","c","compression","cpp","decompression","gzip","lz4","lzma","python","xz","zlib","zstd"],"created_at":"2026-03-07T16:34:51.363Z","updated_at":"2026-05-16T05:23:15.979Z","avatar_url":"https://github.com/dupontcyborg.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# compress-utils\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/algorithms-6-green?style=flat\" alt=\"Algorithms\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/languages-3-yellow?style=flat\" alt=\"Languages\"/\u003e\n  \u003cimg src=\"https://img.shields.io/github/license/dupontcyborg/compress-utils\" alt=\"License\"/\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/github/actions/workflow/status/dupontcyborg/compress-utils/pr_build_and_test.yml\" alt=\"GitHub Actions Workflow Status\"/\u003e\n  \u003cimg src=\"https://img.shields.io/github/v/release/dupontcyborg/compress-utils\" alt=\"GitHub Release\"/\u003e\n  \u003cimg src=\"https://img.shields.io/github/languages/code-size/dupontcyborg/compress-utils\" alt=\"Code Size\"/\u003e\n\u003c/p\u003e\n\nA unified, high-performance interface for six compression algorithms — **Zstandard, Brotli, zlib, bzip2, LZ4, XZ/LZMA** — exposed identically across multiple languages.\n\n```\n                      ┌─────────────────────────────┐\n   Your application → │  C / C++ / Python / JS / TS │\n                      └──────────────┬──────────────┘\n                                     │\n                      ┌──────────────▼──────────────┐\n                      │   compress-utils C ABI      │\n                      │  (one library, six algos)   │\n                      └──────────────┬──────────────┘\n                                     │\n                ┌───────┬───────┬────┴────┬───────┬──────┐\n              zstd   brotli   zlib     bz2     lz4    xz\n```\n\nThe C library is the canonical surface. Every other binding is a thin shim — same allocation model, same error codes, same streaming protocol. Add a binding for any language that speaks C ABI; the work is mostly making the language's idioms (strings, exceptions, generators) feel natural on top of a uniform substrate.\n\n## Pick your language\n\n| Language | Install                                              | Docs                                          |\n|----------|------------------------------------------------------|-----------------------------------------------|\n| **C**    | Build from source ([instructions below](#building)) | [`include/compress_utils.h`](include/compress_utils.h) — the canonical ABI |\n| **C++**  | Header-only; built alongside C                       | [bindings/cpp/README.md](bindings/cpp/README.md) |\n| **Python** | `pip install compress-utils` | [bindings/python/README.md](bindings/python/README.md) |\n| **JS / TS (WASM)** | `npm install compress-utils` | [bindings/wasm/README.md](bindings/wasm/README.md) |\n| Go, Rust, Swift, Java | _Planned — all consume the C ABI directly_ |  |\n\nFor now each binding's README has its own installation + quickstart. A cross-cutting `docs/` is planned for architecture, allocation model, and per-algorithm notes — tracked in [TODO.md](TODO.md#documentation-plan-planned-2026-05-11).\n\n## Supported algorithms\n\n| Algorithm                                          | Strength               | Wire format produced            |\n|----------------------------------------------------|------------------------|---------------------------------|\n| [Zstandard](https://github.com/facebook/zstd)      | High speed, high ratio | ZSTD frame with content size    |\n| [Brotli](https://github.com/google/brotli)         | Web-optimized          | Raw Brotli stream               |\n| [zlib](https://github.com/madler/zlib)             | Ubiquitous (gzip-compatible) | zlib wrapper (RFC 1950)   |\n| [bzip2](https://sourceware.org/bzip2)              | High ratio             | bzip2 stream                    |\n| [LZ4](https://github.com/lz4/lz4)                  | Highest speed          | LZ4 frame (interoperable with `lz4` CLI / `.lz4` files) |\n| [XZ / LZMA](https://github.com/tukaani-project/xz) | Highest ratio          | XZ stream with CRC64            |\n\nAll algorithms expose the same API surface and the same level scale (`1` fastest → `10` smallest). The library maps each user level to the algorithm's native range so you don't need to remember that ZSTD goes 1–22 and zlib goes 1–9.\n\n## Building\n\n### Prerequisites\n\n- CMake 3.17+\n- A C11 compiler (Clang, GCC, MSVC)\n- A C++20 compiler (only for the C++ binding and the pybind11 module)\n- Python 3.10+ and `pybind11-stubgen` (only for the Python binding)\n\n### Build\n\n```sh\ngit clone https://github.com/dupontcyborg/compress-utils.git\ncd compress-utils\n./build.sh                  # Linux / macOS\n# or:\npowershell -File build.ps1  # Windows\n```\n\nThe default build produces:\n\n- `dist/c/lib/libcompress_utils.{dylib,so,dll}` — the shared C library, self-contained (all six algorithms baked in).\n- `dist/c/include/compress_utils.h` — the public C header.\n- `dist/cpp/include/compress_utils.hpp` — the header-only C++ binding.\n- `bindings/python/compress_utils/` — the importable Python package, including auto-generated `.pyi` type stubs.\n\nUseful flags:\n\n- `--algorithms=zstd,zlib` — limit which compressors are included (smaller binary).\n- `--languages=cpp,python,wasm` — limit which bindings are built (C is always built; it's the core).\n- `--release` — Release build (LTO, `-O3` / `/O2`).\n- `--clean` — force a clean rebuild.\n- `--skip-tests` — don't build/run the test suite.\n\nFor raw CMake usage (without `build.sh`):\n\n```sh\ncmake -B build -DCMAKE_BUILD_TYPE=Release\ncmake --build build -j\nctest --test-dir build\n```\n\n## Testing\n\nEach binding has its own test suite, all wired through ctest:\n\n| Target | What it covers |\n|--------|----------------|\n| `test_compress_utils` (C) | One-shot, streaming with tight buffers, cross-API round-trip, error codes, edge cases |\n| `test_compress_utils_cpp` (C++) | `cu::` namespace surface, RAII semantics, exception translation |\n| `test_compress_utils_py` (Python) | Same surface via pybind11, plus 1MB random/repetitive cases, string-vs-enum spellings |\n\nPlus a libFuzzer harness (`-DENABLE_FUZZ=ON`, clang only) at `tests/fuzz/fuzz_decompress.c`.\n\n## Project status\n\nPre-1.0. The C ABI is the source of truth for cross-language behavior — see [`include/compress_utils.h`](include/compress_utils.h) for the contract. Open work (additional language bindings, doc site, CMake package config, fuzz corpora, interop tests against canonical compressors per language) is tracked in [TODO.md](TODO.md).\n\n## AI disclosure\n\nThis project was built with substantial use of large language models. Specifically:\n\n- Architecture and design: human (me, [@dupontcyborg](https://nico.codes), a senior software engineer).\n- Implementation: basically entirely LLM-driven. Most of the C core, all of the C++/Python/WASM bindings, and the test suites were drafted by Mr. Claude\n- Review: me again.\n\nBugs and typos are most likely my own.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\n## Acknowledgments\n\nThis project wraps six battle-tested upstream compression libraries. See [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdupontcyborg%2Fcompress-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdupontcyborg%2Fcompress-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdupontcyborg%2Fcompress-utils/lists"}