{"id":13438710,"url":"https://github.com/dropbox/rust-brotli","last_synced_at":"2025-05-13T17:14:03.084Z","repository":{"id":38107487,"uuid":"56421556","full_name":"dropbox/rust-brotli","owner":"dropbox","description":"Brotli compressor and decompressor written in rust that optionally avoids the stdlib","archived":false,"fork":false,"pushed_at":"2025-04-22T07:07:43.000Z","size":14672,"stargazers_count":843,"open_issues_count":33,"forks_count":85,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-05-03T00:04:54.723Z","etag":null,"topics":["brotli","brotli-compression","brotli-compressor","brotli-decompressor","brotli-encoder","compression","compressor","decompression","rust","rustlang","safe"],"latest_commit_sha":null,"homepage":"https://dropbox.tech/infrastructure/-broccoli--syncing-faster-by-syncing-less","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dropbox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.BSD-3-Clause","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-04-17T06:01:45.000Z","updated_at":"2025-04-24T03:26:10.000Z","dependencies_parsed_at":"2023-02-10T00:15:37.266Z","dependency_job_id":"99d1b53f-36e3-43ea-8bd5-d67a09b82be7","html_url":"https://github.com/dropbox/rust-brotli","commit_stats":{"total_commits":895,"total_committers":24,"mean_commits":"37.291666666666664","dds":"0.47821229050279335","last_synced_commit":"ac9ae8ba7305fa0990261d61aca5f7aeb8772b3f"},"previous_names":[],"tags_count":69,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Frust-brotli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Frust-brotli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Frust-brotli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Frust-brotli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dropbox","download_url":"https://codeload.github.com/dropbox/rust-brotli/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253990498,"owners_count":21995776,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["brotli","brotli-compression","brotli-compressor","brotli-decompressor","brotli-encoder","compression","compressor","decompression","rust","rustlang","safe"],"created_at":"2024-07-31T03:01:07.776Z","updated_at":"2025-05-13T17:13:58.068Z","avatar_url":"https://github.com/dropbox.png","language":"Rust","funding_links":[],"categories":["Libraries","Rust","库 Libraries","库"],"sub_categories":["Compression","压缩 Compression","压缩"],"readme":"# rust-brotli\n\n[![crates.io](https://img.shields.io/crates/v/brotli.svg)](https://crates.io/crates/brotli)\n[![Build Status](https://travis-ci.org/dropbox/rust-brotli.svg?branch=master)](https://travis-ci.org/dropbox/rust-brotli)\n\n## What's new in 8.0.0\n* Fixed LZ77 to comply with the specification\n  * No longer seed the context by the end of the lz77 dictionary. Instead use\n    zero for the seed as would happen without a dictionary. This matches the\n    behavior of brotli with a custom dictionary as specified in\n  * https://datatracker.ietf.org/doc/draft-vandevenne-shared-brotli-format/\n\n## What's new in 7.0.0\n* Fix error with short writes\n* allow quality=10 for certain APIs and make it default to 9.5\n\n## What's new in 6.0.0\n  * Remove unused SIMD use statements\n  * hide a few warnings - these are TODOs, and should be fixed in separate PRs\n  * do NOT build SIMD as part of MSRV -- doesn't make any sense to combine nightly with MSRV\n\n## What's new in 5.0.0\n* The FFI is no longer active by default to avoid ODR issues if multiple versions of brotli are included in several dependent crates.\n\n## What's new in 4.0.0\nPinned to a rust-brotli-decompressor that can disable the ffi with the ffi-api\nflag.\nThis can help avoid symbol conflicts with other brotli libs.\n\n## What's new in 3.5\nUpdated SIMD support.\nBetter CI integration.\nCleaned up some of the clippy warnings.\n\n## What's new in 3.4\nBrotli decompressor's reader and writer has better behavior when operating upon brotli streams with extra bits at the end.\nOptional features like stdsimd are now tested or disabled for now.\n\n## What's new in 3.2\n* into_inner conversions for both Reader and Writer classes\n\n## What's new in 3.0\n* A fully compatible FFI for drop-in compatibiltiy with the https://github.com/google/brotli binaries\n  * custom allocators fully supported\n* Multithreaded compression so multiple threads can operate in unison on a single file\n* Concatenatability mode to add the feature requested in https://github.com/google/brotli/issues/628\n  * binary tool catbrotli can accomplish this if the first file was specified with -apendable and the second with -catable\n* validation mode where a file is double-checked to be able to be decompressed with the same settings; useful for benchmarking or fuzzing\n* Magic Number: where the brotli file can have a useful header with a few magic bytes, concatability info and a final output size for pre-allocating memory\n\n## What's new in 2.5\n* In 2.5 The callback also passes down an allocator to make new StaticCommands and PDFs and 256 bit floating point vectors.\n* In 2.4 The callback with the compression intermediate representation now passes a full metablock at a time. Also these items are mutable\nin case futher optimization is desired\n\n## What's new in 2.3\n\n* Flush now produces output instead of calling finish on the stream. This allows you to use the writer abstraction to\nget immediate output without having to resort to the CompressStream internal abstraction\n\n## Project Requirements\n\nDirect no-stdlib port of the C brotli compressor to Rust\n\nno dependency on the Rust stdlib: this library would be ideal for decompressing within a rust kernel among other things.\n\nThis is useful to see how C and Rust compare in an apples-to-apples\ncomparison where the same algorithms and data structures and\noptimizations are employed.\n\n## Compression Usage\n\nRust brotli currently supports compression levels 0 - 11\nThey should be bitwise identical to the brotli C compression engine at compression levels 0-9\nRecommended lg_window_size is between 20 and 22\n\n### With the io::Read abstraction\n```rust\nlet mut input = brotli::CompressorReader::new(\u0026mut io::stdin(), 4096 /* buffer size */,\n                                              quality as u32, lg_window_size as u32);\n```\nthen you can simply read input as you would any other io::Read class\n\n### With the io::Write abstraction\n\n```rust\nlet mut writer = brotli::Compressor::new(\u0026mut io::stdout(), 4096 /* buffer size */,\n                                         quality as u32, lg_window_size as u32);\n```\n\nThere are also methods to build Compressor Readers or Writers using the with_params static function\n\neg:\n```rust\nlet params = BrotliEncoderParams::default();\n// modify params to fit the application needs\nlet mut writer = brotli::Compressor::with_params(\u0026mut io::stdout(), 4096 /* buffer size */,\n                                         params);\n```\nor for the reader\n```rust\nlet params = BrotliEncoderParams::default();\n// modify params to fit the application needs\nlet mut writer = brotli::CompressorReader::with_params(\u0026mut io::stdin(), 4096 /* buffer size */,\n                                                       params);\n```\n\n\n### With the Stream Copy abstraction\n\n```rust\nmatch brotli::BrotliCompress(\u0026mut io::stdin(), \u0026mut io::stdout(), \u0026brotli_encoder_params) {\n    Ok(_) =\u003e {},\n    Err(e) =\u003e panic!(\"Error {:?}\", e),\n}\n```\n\n## Decompression Usage\n\n### With the io::Read abstraction\n\n```rust\nlet mut input = brotli::Decompressor::new(\u0026mut io::stdin(), 4096 /* buffer size */);\n```\nthen you can simply read input as you would any other io::Read class\n\n### With the io::Write abstraction\n\n```rust\nlet mut writer = brotli::DecompressorWriter::new(\u0026mut io::stdout(), 4096 /* buffer size */);\n```\n\n### With the Stream Copy abstraction\n\n```rust\nmatch brotli::BrotliDecompress(\u0026mut io::stdin(), \u0026mut io::stdout()) {\n    Ok(_) =\u003e {},\n    Err(e) =\u003e panic!(\"Error {:?}\", e),\n}\n```\n\n### With manual memory management\n\nThere are 3 steps to using brotli without stdlib\n\n1. setup the memory manager\n2. setup the BrotliState\n3. in a loop, call BrotliDecompressStream\n\nin Detail\n\n```rust\n// at global scope declare a MemPool type -- in this case we'll choose the heap to\n// avoid unsafe code, and avoid restrictions of the stack size\n\ndeclare_stack_allocator_struct!(MemPool, heap);\n\n// at local scope, make a heap allocated buffers to hold uint8's uint32's and huffman codes\nlet mut u8_buffer = define_allocator_memory_pool!(4096, u8, [0; 32 * 1024 * 1024], heap);\nlet mut u32_buffer = define_allocator_memory_pool!(4096, u32, [0; 1024 * 1024], heap);\nlet mut hc_buffer = define_allocator_memory_pool!(4096, HuffmanCode, [0; 4 * 1024 * 1024], heap);\nlet heap_u8_allocator = HeapPrealloc::\u003cu8\u003e::new_allocator(4096, \u0026mut u8_buffer, bzero);\nlet heap_u32_allocator = HeapPrealloc::\u003cu32\u003e::new_allocator(4096, \u0026mut u32_buffer, bzero);\nlet heap_hc_allocator = HeapPrealloc::\u003cHuffmanCode\u003e::new_allocator(4096, \u0026mut hc_buffer, bzero);\n\n// At this point no more syscalls are going to be needed since everything can come from the allocators.\n\n// Feel free to activate SECCOMP jailing or other mechanisms to secure your application if you wish.\n\n// Now it's possible to setup the decompressor state\nlet mut brotli_state = BrotliState::new(heap_u8_allocator, heap_u32_allocator, heap_hc_allocator);\n\n// at this point the decompressor simply needs an input and output buffer and the ability to track\n// the available data left in each buffer\nloop {\n    result = BrotliDecompressStream(\u0026mut available_in, \u0026mut input_offset, \u0026input.slice(),\n                                    \u0026mut available_out, \u0026mut output_offset, \u0026mut output.slice_mut(),\n                                    \u0026mut written, \u0026mut brotli_state);\n\n    // just end the decompression if result is BrotliResult::ResultSuccess or BrotliResult::ResultFailure\n}\n```\n\nThis interface is the same interface that the C brotli decompressor uses\n\nAlso feel free to use custom allocators that invoke Box directly.\nThis example illustrates a mechanism to avoid subsequent syscalls after the initial allocation\n\n## Using the C interface\n\nrust-brotli is a drop-in replacement for the official https://github.com/google/brotli C\nimplementation. That means you can use it from any place that supports that library.\nTo build rust-brotli in this manner enter the c subdirectory and run make there\n\ncd c \u0026\u0026 make\n\nthis should build c/target/release/libbrotli.so and should build the vanilla\ncommand line tool in C for compressing and decompressing any brotli file.\n\nthe libbrotli.so in c/target/release should be able to replace any other libbrotli.so\nfile, but with all the advantages of using safe rust (except in the FFI bindings)\n\nThe code also allows a wider range of options, including forcing the prediction mode\n(eg UTF8 vs signed vs MSB vs LSB) and changing the weight of the literal cost from 540\n to other values.\n\nAdditionally the CATABLE and APPENDABLE options are exposed and allow concatenation of files\ncreated in this manner.\n\nSpecifically CATABLE files can be concatenated in any order using the catbrotli tool\nand APPENDABLE files can be the first file in a sequence of catable files...\neg you can combine\nappendable.br catable1.br catable2.br catable3.br\n\nor simply\ncatable0.br catable1.br catable2.br catable3.br\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdropbox%2Frust-brotli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdropbox%2Frust-brotli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdropbox%2Frust-brotli/lists"}