Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mcmilk/zstdmt
Multithreading Library for Brotli, Lizard, LZ4, LZ5, Snappy and Zstandard
https://github.com/mcmilk/zstdmt
brotli lizard lz4 lz5 multithreading-library skippables-frame snappy zstandard zstd
Last synced: 12 days ago
JSON representation
Multithreading Library for Brotli, Lizard, LZ4, LZ5, Snappy and Zstandard
- Host: GitHub
- URL: https://github.com/mcmilk/zstdmt
- Owner: mcmilk
- License: other
- Created: 2016-07-21T05:04:46.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2024-07-12T07:36:39.000Z (4 months ago)
- Last Synced: 2024-08-01T19:31:23.312Z (3 months ago)
- Topics: brotli, lizard, lz4, lz5, multithreading-library, skippables-frame, snappy, zstandard, zstd
- Language: C
- Homepage:
- Size: 942 KB
- Stars: 178
- Watchers: 17
- Forks: 33
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGES.md
- License: LICENSE
Awesome Lists containing this project
README
# Multithreading Library for [Brotli], [Lizard], [LZ4], [LZ5], [Snappy-c], [LZFSE] and [Zstandard]
## Description
- works with skippables frame id 0x184D2A50 (12 bytes per compressed frame)
- brotli is supported the same way, it will encapsulate the real brotli stream
within an 16 byte frame header
## Generic skippable frame definition
- the frame header for [Lizard], [LZ4], [LZ5] and [Zstandard] is like this:
size | value | description
--------|-------------------|------------
4 bytes | 0x184D2A50U | magic for skippable frame
4 bytes | 4 | size of skippable frame
4 bytes | compressed size | size of the following frame (compressed data)
## [Brotli] frame definition
- the frame header for brotli is defined a bit different:
size | value | description
--------|-------------------|------------
4 bytes | 0x184D2A50U | magic for skippable frame (like zstd)
4 bytes | 8 | size of skippable frame
4 bytes | compressed size | size of the following frame (compressed data)
2 bytes | 0x5242U | magic for brotli "BR"
2 bytes | uncompressed size | allocation hint for decompressor (64KB * this size)
## [Snappy-c] frame definition
- the frame header for Snappy-c is defined a bit different:
size | value | description
--------|-------------------|------------
4 bytes | 0x184D2A50U | magic for skippable frame (like zstd)
4 bytes | 8 | size of skippable frame
4 bytes | compressed size | size of the following frame (compressed data)
2 bytes | 0x5053U | magic for Snappy-c "SP"
2 bytes | uncompressed size | allocation hint for decompressor (64KB * this size)
## [LZFSE] frame definition
- the frame header for Lzfse is defined a bit different:
size | value | description
--------|-------------------|------------
4 bytes | 0x184D2A50U | magic for skippable frame (like zstd)
4 bytes | 8 | size of skippable frame
4 bytes | compressed size | size of the following frame (compressed data)
2 bytes | 0x464CU | magic for Lzfse "LF"
2 bytes | uncompressed size | allocation hint for decompressor (64KB * this size)
## Usage of the Testutils
- see [programs](https://github.com/mcmilk/zstdmt/tree/master/programs)
## Usage of the Library
- see [lib](https://github.com/mcmilk/zstdmt/tree/master/lib)
[Brotli]:https://github.com/google/brotli/
[LZ4]:https://github.com/lz4/lz4/
[LZ5]:https://github.com/inikep/lz5/
[Zstandard]:https://github.com/facebook/zstd/
[Lizard]:https://github.com/inikep/lizard/
[Snappy-c]:https://github.com/andikleen/snappy-c
[LZFSE]:https://github.com/lzfse/lzfse
/TR 2020-10-15