{"id":28455146,"url":"https://github.com/imagemagick/deflate","last_synced_at":"2025-06-28T16:32:27.181Z","repository":{"id":196273385,"uuid":"695641985","full_name":"ImageMagick/deflate","owner":"ImageMagick","description":"libdeflate for the Windows build of ImageMagick","archived":false,"fork":false,"pushed_at":"2025-05-17T09:09:57.000Z","size":342,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-06T21:15:42.183Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ImageMagick.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS.md","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-09-23T19:25:54.000Z","updated_at":"2025-05-17T09:10:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"675732b4-0165-4d6a-a3ad-804e7afc8322","html_url":"https://github.com/ImageMagick/deflate","commit_stats":null,"previous_names":["imagemagick/deflate"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ImageMagick/deflate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImageMagick%2Fdeflate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImageMagick%2Fdeflate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImageMagick%2Fdeflate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImageMagick%2Fdeflate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ImageMagick","download_url":"https://codeload.github.com/ImageMagick/deflate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImageMagick%2Fdeflate/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262460254,"owners_count":23314698,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-06T21:15:44.127Z","updated_at":"2025-06-28T16:32:27.168Z","avatar_url":"https://github.com/ImageMagick.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Overview\n\nlibdeflate is a library for fast, whole-buffer DEFLATE-based compression and\ndecompression.\n\nThe supported formats are:\n\n- DEFLATE (raw)\n- zlib (a.k.a. DEFLATE with a zlib wrapper)\n- gzip (a.k.a. DEFLATE with a gzip wrapper)\n\nlibdeflate is heavily optimized.  It is significantly faster than the zlib\nlibrary, both for compression and decompression, and especially on x86 and ARM\nprocessors.  In addition, libdeflate provides optional high compression modes\nthat provide a better compression ratio than the zlib's \"level 9\".\n\nlibdeflate itself is a library.  The following command-line programs which use\nthis library are also included:\n\n* `libdeflate-gzip`, a program which can be a drop-in replacement for standard\n  `gzip` under some circumstances.  Note that `libdeflate-gzip` has some\n  limitations; it is provided for convenience and is **not** meant to be the\n  main use case of libdeflate.  It needs a lot of memory to process large files,\n  and it omits support for some infrequently-used options of GNU gzip.\n\n* `benchmark`, a test program that does round-trip compression and decompression\n  of the provided data, and measures the compression and decompression speed.\n  It can use libdeflate, zlib, or a combination of the two.\n\n* `checksum`, a test program that checksums the provided data with Adler-32 or\n  CRC-32, and optionally measures the speed.  It can use libdeflate or zlib.\n\nFor the release notes, see the [NEWS file](NEWS.md).\n\n## Table of Contents\n\n- [Building](#building)\n  - [Using CMake](#using-cmake)\n  - [Directly integrating the library sources](#directly-integrating-the-library-sources)\n  - [Supported compilers](#supported-compilers)\n- [API](#api)\n- [Bindings for other programming languages](#bindings-for-other-programming-languages)\n- [DEFLATE vs. zlib vs. gzip](#deflate-vs-zlib-vs-gzip)\n- [Compression levels](#compression-levels)\n- [Motivation](#motivation)\n- [License](#license)\n\n# Building\n\n## Using CMake\n\nlibdeflate uses [CMake](https://cmake.org/).  It can be built just like any\nother CMake project, e.g. with:\n\n    cmake -B build \u0026\u0026 cmake --build build\n\nBy default the following targets are built:\n\n- The static library (normally called `libdeflate.a`)\n- The shared library (normally called `libdeflate.so`)\n- The `libdeflate-gzip` program, including its alias `libdeflate-gunzip`\n\nBesides the standard CMake build and installation options, there are some\nlibdeflate-specific build options.  See `CMakeLists.txt` for the list of these\noptions.  To set an option, add `-DOPTION=VALUE` to the `cmake` command.\n\nPrebuilt Windows binaries can be downloaded from\nhttps://github.com/ebiggers/libdeflate/releases.\n\n## Directly integrating the library sources\n\nAlthough the official build system is CMake, care has been taken to keep the\nlibrary source files compilable directly, without a prerequisite configuration\nstep.  Therefore, it is also fine to just add the library source files directly\nto your application, without using CMake.\n\nYou should compile both `lib/*.c` and `lib/*/*.c`.  You don't need to worry\nabout excluding irrelevant architecture-specific code, as this is already\nhandled in the source files themselves using `#ifdef`s.\n\nIf you are doing a freestanding build with `-ffreestanding`, you must add\n`-DFREESTANDING` as well (matching what the `CMakeLists.txt` does).\n\n## Supported compilers\n\n- gcc: v4.9 and later\n- clang: v3.9 and later (upstream), Xcode 8 and later (Apple)\n- MSVC: Visual Studio 2015 and later\n- Other compilers: any other C99-compatible compiler should work, though if your\n  compiler pretends to be gcc, clang, or MSVC, it needs to be sufficiently\n  compatible with the compiler it pretends to be.\n\nThe above are the minimums, but using a newer compiler allows more of the\narchitecture-optimized code to be built.  libdeflate is most heavily optimized\nfor gcc and clang, but MSVC is supported fairly well now too.\n\nThe recommended optimization flag is `-O2`, and the `CMakeLists.txt` sets this\nfor release builds.  `-O3` is fine too, but often `-O2` actually gives better\nresults.  It's unnecessary to add flags such as `-mavx2` or `/arch:AVX2`, though\nyou can do so if you want to.  Most of the relevant optimized functions are\nbuilt regardless of such flags, and appropriate ones are selected at runtime.\nFor the same reason, flags like `-mno-avx2` do *not* cause all code using the\ncorresponding instruction set extension to be omitted from the binary; this is\nworking as intended due to the use of runtime CPU feature detection.\n\nIf using gcc, your gcc should always be paired with a binutils version that is\nnot much older than itself, to avoid problems where the compiler generates\ninstructions the assembler cannot assemble.  Usually systems have their gcc and\nbinutils paired properly, but rarely a mismatch can arise in cases such as the\nuser installing a newer gcc version without a proper binutils alongside it.\nSince libdeflate v1.22, the CMake-based build system will detect incompatible\nbinutils versions and disable some optimized code accordingly.  In older\nversions of libdeflate, or if CMake is not being used, a too-old binutils can\ncause build errors like \"no such instruction\" from the assembler.\n\n# API\n\nlibdeflate has a simple API that is not zlib-compatible.  You can create\ncompressors and decompressors and use them to compress or decompress buffers.\nSee libdeflate.h for details.\n\nThere is currently no support for streaming.  This has been considered, but it\nalways significantly increases complexity and slows down fast paths.\nUnfortunately, at this point it remains a future TODO.  So: if your application\ncompresses data in \"chunks\", say, less than 1 MB in size, then libdeflate is a\ngreat choice for you; that's what it's designed to do.  This is perfect for\ncertain use cases such as transparent filesystem compression.  But if your\napplication compresses large files as a single compressed stream, similarly to\nthe `gzip` program, then libdeflate isn't for you.\n\nNote that with chunk-based compression, you generally should have the\nuncompressed size of each chunk stored outside of the compressed data itself.\nThis enables you to allocate an output buffer of the correct size without\nguessing.  However, libdeflate's decompression routines do optionally provide\nthe actual number of output bytes in case you need it.\n\nWindows developers: note that the calling convention of libdeflate.dll is\n\"cdecl\".  (libdeflate v1.4 through v1.12 used \"stdcall\" instead.)\n\n# Bindings for other programming languages\n\nThe libdeflate project itself only provides a C library.  If you need to use\nlibdeflate from a programming language other than C or C++, consider using the\nfollowing bindings:\n\n* C#: [LibDeflate.NET](https://github.com/jzebedee/LibDeflate.NET)\n* Delphi: [libdeflate-pas](https://github.com/zedxxx/libdeflate-pas)\n* Go: [go-libdeflate](https://github.com/4kills/go-libdeflate)\n* Java: [libdeflate-java](https://github.com/astei/libdeflate-java)\n* Julia: [LibDeflate.jl](https://github.com/jakobnissen/LibDeflate.jl)\n* Nim: [libdeflate-nim](https://github.com/gemesa/libdeflate-nim)\n* Perl: [Gzip::Libdeflate](https://github.com/benkasminbullock/gzip-libdeflate)\n* PHP: [ext-libdeflate](https://github.com/pmmp/ext-libdeflate)\n* Python: [deflate](https://github.com/dcwatson/deflate)\n* Ruby: [libdeflate-ruby](https://github.com/kaorimatz/libdeflate-ruby)\n* Rust: [libdeflater](https://github.com/adamkewley/libdeflater)\n\nNote: these are third-party projects which haven't necessarily been vetted by\nthe authors of libdeflate.  Please direct all questions, bugs, and improvements\nfor these bindings to their authors.\n\nAlso, unfortunately many of these bindings bundle or pin an old version of\nlibdeflate.  To avoid known issues in old versions and to improve performance,\nbefore using any of these bindings please ensure that the bundled or pinned\nversion of libdeflate has been upgraded to the latest release.\n\n# DEFLATE vs. zlib vs. gzip\n\nThe DEFLATE format ([rfc1951](https://www.ietf.org/rfc/rfc1951.txt)), the zlib\nformat ([rfc1950](https://www.ietf.org/rfc/rfc1950.txt)), and the gzip format\n([rfc1952](https://www.ietf.org/rfc/rfc1952.txt)) are commonly confused with\neach other as well as with the [zlib software library](http://zlib.net), which\nactually supports all three formats.  libdeflate (this library) also supports\nall three formats.\n\nBriefly, DEFLATE is a raw compressed stream, whereas zlib and gzip are different\nwrappers for this stream.  Both zlib and gzip include checksums, but gzip can\ninclude extra information such as the original filename.  Generally, you should\nchoose a format as follows:\n\n- If you are compressing whole files with no subdivisions, similar to the `gzip`\n  program, you probably should use the gzip format.\n- Otherwise, if you don't need the features of the gzip header and footer but do\n  still want a checksum for corruption detection, you probably should use the\n  zlib format.\n- Otherwise, you probably should use raw DEFLATE.  This is ideal if you don't\n  need checksums, e.g. because they're simply not needed for your use case or\n  because you already compute your own checksums that are stored separately from\n  the compressed stream.\n\nNote that gzip and zlib streams can be distinguished from each other based on\ntheir starting bytes, but this is not necessarily true of raw DEFLATE streams.\n\n# Compression levels\n\nAn often-underappreciated fact of compression formats such as DEFLATE is that\nthere are an enormous number of different ways that a given input could be\ncompressed.  Different algorithms and different amounts of computation time will\nresult in different compression ratios, while remaining equally compatible with\nthe decompressor.\n\nFor this reason, the commonly used zlib library provides nine compression\nlevels.  Level 1 is the fastest but provides the worst compression; level 9\nprovides the best compression but is the slowest.  It defaults to level 6.\nlibdeflate uses this same design but is designed to improve on both zlib's\nperformance *and* compression ratio at every compression level.  In addition,\nlibdeflate's levels go [up to 12](https://xkcd.com/670/) to make room for a\nminimum-cost-path based algorithm (sometimes called \"optimal parsing\") that can\nsignificantly improve on zlib's compression ratio.\n\nIf you are using DEFLATE (or zlib, or gzip) in your application, you should test\ndifferent levels to see which works best for your application.\n\n# Motivation\n\nDespite DEFLATE's widespread use mainly through the zlib library, in the\ncompression community this format from the early 1990s is often considered\nobsolete.  And in a few significant ways, it is.\n\nSo why implement DEFLATE at all, instead of focusing entirely on\nbzip2/LZMA/xz/LZ4/LZX/ZSTD/Brotli/LZHAM/LZFSE/[insert cool new format here]?\n\nTo do something better, you need to understand what came before.  And it turns\nout that most ideas from DEFLATE are still relevant.  Many of the newer formats\nshare a similar structure as DEFLATE, with different tweaks.  The effects of\ntrivial but very useful tweaks, such as increasing the sliding window size, are\noften confused with the effects of nontrivial but less useful tweaks.  And\nactually, many of these formats are similar enough that common algorithms and\noptimizations (e.g. those dealing with LZ77 matchfinding) can be reused.\n\nIn addition, comparing compressors fairly is difficult because the performance\nof a compressor depends heavily on optimizations which are not intrinsic to the\ncompression format itself.  In this respect, the zlib library sometimes compares\npoorly to certain newer code because zlib is not well optimized for modern\nprocessors.  libdeflate addresses this by providing an optimized DEFLATE\nimplementation which can be used for benchmarking purposes.  And, of course,\nreal applications can use it as well.\n\n# License\n\nlibdeflate is [MIT-licensed](COPYING).\n\nI am not aware of any patents or patent applications relevant to libdeflate.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimagemagick%2Fdeflate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimagemagick%2Fdeflate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimagemagick%2Fdeflate/lists"}