{"id":20468493,"url":"https://github.com/instantiations/es_compression","last_synced_at":"2025-08-21T01:32:23.305Z","repository":{"id":42499258,"uuid":"302713418","full_name":"instantiations/es_compression","owner":"instantiations","description":"Compression framework for Dart providing FFI implementations for Brotli, Lz4, Zstd (Zstandard) with ready-to-use prebuilt binaries for Win/Linux/Mac.","archived":false,"fork":false,"pushed_at":"2025-04-01T19:10:54.000Z","size":10059,"stargazers_count":46,"open_issues_count":0,"forks_count":8,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-08T01:42:24.029Z","etag":null,"topics":["brotli","compression","dart","dartlang","encoder-decoder","ffi-bindings","lz4","zstd"],"latest_commit_sha":null,"homepage":"https://www.instantiations.com","language":"Dart","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/instantiations.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-09T17:43:16.000Z","updated_at":"2025-04-01T19:10:58.000Z","dependencies_parsed_at":"2024-06-21T16:21:25.385Z","dependency_job_id":"219a84a1-73d0-468a-acbd-bdc91d914917","html_url":"https://github.com/instantiations/es_compression","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/instantiations/es_compression","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instantiations%2Fes_compression","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instantiations%2Fes_compression/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instantiations%2Fes_compression/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instantiations%2Fes_compression/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/instantiations","download_url":"https://codeload.github.com/instantiations/es_compression/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instantiations%2Fes_compression/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271415041,"owners_count":24755628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["brotli","compression","dart","dartlang","encoder-decoder","ffi-bindings","lz4","zstd"],"created_at":"2024-11-15T14:05:43.726Z","updated_at":"2025-08-21T01:32:23.300Z","avatar_url":"https://github.com/instantiations.png","language":"Dart","readme":"[![Language](https://img.shields.io/badge/language-Dart-blue.svg)](https://dart.dev)\n[![pub package](https://img.shields.io/pub/v/es_compression.svg)](https://pub.dartlang.org/packages/es_compression)\n[![Status](https://github.com/instantiations/es_compression/actions/workflows/main.yml/badge.svg)](https://github.com/instantiations/es_compression/actions/workflows/main.yml)\n\n# ES Compression: A Compression Framework for Dart\n\n## Description\nCompression framework for Dart providing FFI implementations for Brotli, Lz4, Zstd (Zstandard) with ready-to-use prebuilt binaries\nfor Win/Linux/Mac.\n\nThis work is an inspired port of the *Unified Compression Framework* from the [VAST Platform] (VA Smalltalk) language\nand development environment. \n\nSee the [Design Document](https://github.com/instantiations/es_compression/blob/master/doc/design_doc.md) for detailed\ninformation on how this package was designed and implemented.\n\nBelow is a simple example of what an encode/decode would look like:\n```dart\nimport 'dart:convert';\nimport 'package:es_compression/brotli.dart';\nimport 'package:es_compression/lz4.dart';\nimport 'package:es_compression/zstd.dart';\n\n/// Brotli, Lz4, Zstd usage example\nvoid main() {\n  final bytes = utf8.encode('Hello Dart');\n  for (final codec in [brotli, lz4, zstd]) {\n    final encoded = codec.encode(bytes);\n    final decoded = codec.decode(encoded);\n    print(utf8.decode(decoded));\n  }\n}\n```\n\n## Executables\nIn the `bin` subdirectory, the following executables are provided.\n\n| Executable    | Source                | Description                                            |\n| --------------| ----------------------|------------------------------------------------------- |\n| `escompress`  | `es_compress.dart`    | Encode/Decode files using brotli, gzip, lz4 and zstd   |\n\nIf you want to use escompress on the command line,\ninstall it using `pub global activate`:\n\n```console\n\u003e pub global activate es_compression\n```\n\n### escompress\n`escompress` is a program that will encode/decode files using `brotli`, `gzip`, `lz4` or `zstd`.\n\nThe user provides the input and output file names from the command line.\nBy default, the file extension of either the input or output file name is used to determine which algorithm and\nencode/decode mode to use.\nThe user can provide additional command line arguments to make these decisions explicitly.\nSome examples are provided below.\n\n#### Examples\n\nEncode *input.txt* to *output.lz4* using Lz4 compression:\n```console\n\u003e escompress -i\"input.txt\" -o\"output.lz4\"\n```\n\nEncode *input.txt* to *output.lz4* using Lz4 compression at compression level 3:\n```console\n\u003e escompress -l 3 -i\"input.txt\" -o\"output.lz4\"\n```\n\nDecode *input.brotli* to *output.txt* using Brotli compression:\n```console\n\u003e escompress -i\"input.brotli\" -o\"output.txt\"\n```\n\nEncode *input.txt* to *output.compressed* using Zstd compression:\n```console\n\u003e escompress -e -a zstd -i\"input.txt\" -o\"output.compressed\"\n```\n\nDecode *input.compressed* to *output.txt* using GZip compression:\n```console\n\u003e escompress -d -a gzip -i\"input.compressed\" -o\"output.txt\"\n```\n\nPrint help:\n```console\n\u003e escompress -h\n```\n\n## Examples\nIn the `example` subdirectory, the following examples are provided to demonstrate usage of the converters and the\nframework.\n\n| Example                   | Description                                                                               |\n| ------------------------- | ----------------------------------------------------------------------------------------- |\n| `brotli_example.dart`     | Encode/Decode in one-shot and streaming modes using the Brotli FFI-based implementation   |\n| `lz4_example.dart`        | Encode/Decode in one-shot and streaming modes using the Lz4 FFI-based implementation      |\n| `rle_example.dart`        | A simple RLE compression example designed to show how build custom codecs                 |\n| `zstd_example.dart`       | Encode/Decode in one-shot and streaming modes using the Zstd FFI-based implementation     |\n\nTo run (brotli shown below):\n```console\n\u003e dart example/brotli_example.dart\n```\n\n## Tests\nIn the `test` subdirectory, the following tests are provided for the compression framework and encoder/decoder\nimplementations.\n\n| Test                   | Description                                                   |\n| -----------------------| ------------------------------------------------------------- |\n| `benchmarks_test.dart` | Test benchmarks defined in `benchmark` subdirectory           |\n| `brotli_test.dart`     | Test encoding/decode of the Brotli FFI-based implementation   |\n| `buffer_test.dart`     | Test `CodecBuffer` and friends in the compression framework   |\n| `escompress_test.dart` | Test `escompress` binary in the `bin` subdirectory            |\n| `examples_test.dart`   | Test examples defined in the `example` subdirectory           |\n| `lz4_test.dart`        | Test encoding/decode of the Lz4 FFI-based implementation      |\n| `zstd_test.dart`       | Test encoding/decode of the Zstd FFI-based implementation     |\n\nTo run test suite:\n```console\n\u003e pub run test\n```\n\n## Benchmarks\nIn the `benchmark` subdirectory, the following benchmarks are provided to help understand encoder/decoder performance\nand tradeoffs involved with parameters like buffer sizing.\n\n| Benchmark                 | Description                                                        |\n| ------------------------- | ------------------------------------------------------------------ |\n| `brotli_benchmark.dart`   | Benchmark encoding/decode of the Brotli FFI-based implementation   |\n| `gzip_benchmark.dart`     | Benchmark encoding/decode of the GZip implementation from the Dart SDK |\n| `lz4_benchmark.dart`      | Benchmark encoding/decode of the Lz4 FFI-based implementation      |\n| `zstd_benchmark.dart`     | Benchmark encoding/decode of the Zstd FFI-based implementation     |\n\nTo run (lz4 shown below):\n```console\n\u003e dart benchmark/lz4_benchmark.dart\n```\n\n## Deployment\nFFI-based implementations will need access to the low-level shared libraries (i.e. .dll, .so, .dylib).\nThis package offers a flexible library loader that can be customized for end-user deployment needs.\n\nBy default, the resolution order is:\n- Environment Variable\n- Package-Relative\n- Script-Relative\n- OS-Dependent\n\nUser Provided Resolution: The user can override the above resolution with a user provided library path. The different\nstrategies for locating shared libraries are described below.\n\n### Android Deployment Instructions\n\nTo use the `es_compression` package on Android, you must ensure that your Android project includes native `.so` libraries for Brotli, Lz4, and Zstd.\n\n#### Steps\n\n1. **Download Prebuilt Android Native Libraries**\n\n   Download the precompiled `.so` files from:  \n   [android_lib_es_compression0_9_9 Releases](https://github.com/isong0623/android_lib_es_compression0_9_9/releases)\n\n2. **Rename Library Files**\n\n   Rename each file by adding the `lib` prefix. For example:\n\n   ```\n   esbrotli-android64.so → libesbrotli-android64.so\n   ```\n\n3. **Place `.so` Files in jniLibs**\n\n   Extract and organize them under `android/app/src/main/jniLibs`:\n\n   ```\n   android/app/src/main/jniLibs/\n   ├── arm64-v8a/\n   │   ├── libesbrotli-android64.so\n   │   ├── libeslz4-android64.so\n   │   └── libeszstd-android64.so\n   ├── armeabi-v7a/\n   │   ├── libesbrotli-android32.so\n   │   ├── libeslz4-android32.so\n   │   └── libeszstd-android32.so\n   ├── x86/\n   │   ├── libesbrotli-android32.so\n   │   ├── libeslz4-android32.so\n   │   └── libeszstd-android32.so\n   └── x86_64/\n       ├── libesbrotli-android64.so\n       ├── libeslz4-android64.so\n       └── libeszstd-android64.so\n   ```\n\n4. **Update `android/app/build.gradle`**\n\n   Open your `android/app/build.gradle` file and add the following inside the `android` block:\n\n   ```gradle\n   android {\n       sourceSets {\n           main.jniLibs.srcDirs += 'src/main/jniLibs'\n       }\n       defaultConfig {\n           ndk.abiFilters 'armeabi-v7a', 'arm64-v8a', 'x86', 'x86_64'\n       }\n   }\n   ```\n\nOnce completed, the `es_compression` package will be fully functional on Android with no manual native compilation required.\n\nThese steps have been tested by multiple contributors. Please let us know via [Issues](https://github.com/instantiations/es_compression/issues) if they work for your setup or need adjustment.\n\n### Environment Variable Resolution\nAn environment variable can be defined that provides the path to the shared library. This is either the path to the shared\nlibrary file or the directory which should contain the filename of the form *es{algo}_{os}{bitness}.{ext}*. For example,\nthe filename for lz4 on 64-bit windows would be *eslz4_win64.dll*.\n\n| Codec      | Environment Variable  |\n| -----------| --------------------- |\n| `brotli`   | BROTLI_LIBRARY_PATH   |\n| `lz4`      | LZ4_LIBRARY_PATH      |\n| `zstd`     | ZSTD_LIBRARY_PATH     |\n\n### Package-Relative Resolution\nPrebuilt shared libraries for Win/Linux/Mac are provided in the `blob` directory of each FFI codec implementation. These\nhave been built by the package maintainer using the [blob_builder](https://github.com/instantiations/es_compression/tree/master/tool/blob_builder)\ntool.\n\nThe distributed shared libs for a codec named *'xxx'* is expected to be located in `lib/src/xxx/blobs` by default.\n\n### Script-Relative Resolution\nAn attempt is made to find the shared library in the same directory as the running script. The name of the shared library\nis expected to be of the form *es{algo}_{os}{bitness}.{ext}*. For example, the filename for zstd on 64-bit linux would\nbe *eszstd_linux64.dll*.\n\n### OS-Dependent Resolution\nA call to `DynamicLibrary.open()` is made for the filename of the form *es{algo}_{os}{bitness}.{ext}* which will use\nthe resolution rules for the operating system.\n\n### User Provided Resolution\nUsers of this package have the option to override the library path early in the program.\n\nProvided FFI Codecs have static getters/setters for the `libraryPath`. Users should be sure set the `libraryPath`\nearly **before** the first use. A `StateError` will be thrown if a user attempts to set the `libraryPath` more than once.\n\n```dart\nfinal codec = ZstdCodec.libraryPath = '/path/to/shared/library.so';\n```\n\n### Code Signing\n\n#### Windows\nProvided dlls are digitally signed with an MS authenticode certificate owned by [Instantiations, Inc].\n\n#### Linux\n*N/A*\n\n#### Mac\nProvided dylibs are not currently signed, and recent versions of OSX will refuse to load them unless you allow\nit from the *Security \u0026 Privacy* dialog.\n\nThe build scripts have been provided [blob_builder](https://github.com/instantiations/es_compression/tree/master/tool/blob_builder)\nand gives you access to build and sign them yourself, if desired.\n\n*Instantiations may sign the libraries in the future, and if so, it will be noted in the changelog and here.*\n\n## Tools\nIn the `tool` subdirectory, the following tools are provided.\n\n### Blob Builder\n[blob_builder](https://github.com/instantiations/es_compression/tree/master/tool/blob_builder) is a `cmake`-based build\ngenerator which builds all the prebuilt shared libraries and copies them to their appropriate location in the dart library.\n\nThe maintainers use this tool to update the prebuilt shared libraries.\nIt can also be used to build custom versions of the various libraries by making adjustments to CMake variables that\ndefine the version level information.\n\nPrebuilt shared libraries for Win/Linux/Mac are provided in the `blobs` directory for each FFI codec implementation.\nThe distributed shared libs for a codec named *xxx* is expected to be located in `lib/src/xxx/blobs`.\n\nThere are instructions in the main [CMakeLists.txt](https://github.com/instantiations/es_compression/blob/master/tool/blob_builder/CMakeLists.txt)\nfile that describe how to run the tool.\n\n## Framework\nInitial codecs provided by this library use FFI-based implementations. However, the framework easily allows for pure\ndart implementations for use within a front-end web context. [rle_example.dart](https://github.com/instantiations/es_compression/blob/master/example/rle_example.dart)\nshows a simple example of how to accomplish this.\n\nThe `GZipCodec` from `dart:io` served as a great starting point for understanding how to put the VAST Platform framework\nabstractions in terms of Dart codecs, converters, filters, sinks.\n\nThe major compression framework abstractions are:\n- `CodecConverter` - Connects the compression framework to `Converter` in `dart:convert`.\n- `CodecFilter` - Direct processing of byte data and provides low-level compression implementation and hooks.\n- `CodecSink` - A type of `ByteConversionSink` for efficient transmission of byte data.\n- `CodecBuffer` - A buffer with a streaming API that is backed by either [native](https://github.com/instantiations/es_compression/blob/master/lib/src/framework/native/buffers.dart)\nor [dart](https://github.com/instantiations/es_compression/blob/master/lib/src/framework/dart/buffers.dart) heap bytes.\n\n## Features and bugs\nPlease file feature requests and bugs at the [issue tracker][tracker].\n\nThey will be reviewed and addressed on a best-effort basis by [Instantiations, Inc].\n\n[library loader]: https://github.com/instantiations/es_compression/blob/master/lib/src/framework/native/library/open_library.dart\n[tracker]: https://github.com/instantiations/es_compression/issues\n[VAST Platform]: https://www.instantiations.com/vast-platform/\n[Instantiations, Inc]: https://www.instantiations.com\n\n## About Us\n\nSince 1988, Instantiations has been building software to meet the diverse and evolutionary needs of our customers. We've now added Dart and Flutter to our toolbox.\n\t\nFor more information about our custom development or consulting services with Dart, Flutter, and other languages, please visit: https://www.instantiations.com/services/\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finstantiations%2Fes_compression","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finstantiations%2Fes_compression","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finstantiations%2Fes_compression/lists"}