Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/mcmilk/zstdmt

Multithreading Library for Brotli, Lizard, LZ4, LZ5, Snappy and Zstandard
https://github.com/mcmilk/zstdmt

brotli lizard lz4 lz5 multithreading-library skippables-frame snappy zstandard zstd

Last synced: 3 months ago
JSON representation

Multithreading Library for Brotli, Lizard, LZ4, LZ5, Snappy and Zstandard

Host: GitHub
URL: https://github.com/mcmilk/zstdmt
Owner: mcmilk
License: other
Created: 2016-07-21T05:04:46.000Z (almost 8 years ago)
Default Branch: master
Last Pushed: 2024-01-07T15:03:23.000Z (6 months ago)
Last Synced: 2024-01-08T16:30:08.424Z (6 months ago)
Topics: brotli, lizard, lz4, lz5, multithreading-library, skippables-frame, snappy, zstandard, zstd
Language: C
Homepage:
Size: 942 KB
Stars: 158
Watchers: 14
Forks: 32
Open Issues: 0
Metadata Files:
- Readme: README.md
- Changelog: CHANGES.md
- License: LICENSE

Lists

my-awesome-stars - mcmilk/zstdmt - Multithreading Library for Brotli, Lizard, LZ4, LZ5, Snappy and Zstandard (C)

README

        
# Multithreading Library for [Brotli], [Lizard], [LZ4], [LZ5], [Snappy-c], [LZFSE] and [Zstandard]

## Description

- works with skippables frame id 0x184D2A50 (12 bytes per compressed frame)

- brotli is supported the same way, it will encapsulate the real brotli stream

  within an 16 byte frame header

## Generic skippable frame definition

- the frame header for [Lizard], [LZ4], [LZ5] and [Zstandard] is like this:

size    | value             | description

--------|-------------------|------------

4 bytes | 0x184D2A50U       | magic for skippable frame

4 bytes | 4                 | size of skippable frame

4 bytes | compressed size   | size of the following frame (compressed data)

## [Brotli] frame definition

- the frame header for brotli is defined a bit different:

size    | value             | description

--------|-------------------|------------

4 bytes | 0x184D2A50U       | magic for skippable frame (like zstd)

4 bytes | 8                 | size of skippable frame

4 bytes | compressed size   | size of the following frame (compressed data)

2 bytes | 0x5242U           | magic for brotli "BR"

2 bytes | uncompressed size | allocation hint for decompressor (64KB * this size)

## [Snappy-c] frame definition

- the frame header for Snappy-c is defined a bit different:

size    | value             | description

--------|-------------------|------------

4 bytes | 0x184D2A50U       | magic for skippable frame (like zstd)

4 bytes | 8                 | size of skippable frame

4 bytes | compressed size   | size of the following frame (compressed data)

2 bytes | 0x5053U           | magic for Snappy-c "SP"

2 bytes | uncompressed size | allocation hint for decompressor (64KB * this size)

## [LZFSE] frame definition

- the frame header for Lzfse is defined a bit different:

size    | value             | description

--------|-------------------|------------

4 bytes | 0x184D2A50U       | magic for skippable frame (like zstd)

4 bytes | 8                 | size of skippable frame

4 bytes | compressed size   | size of the following frame (compressed data)

2 bytes | 0x464CU           | magic for Lzfse "LF"

2 bytes | uncompressed size | allocation hint for decompressor (64KB * this size)

## Usage of the Testutils

- see [programs](https://github.com/mcmilk/zstdmt/tree/master/programs)

## Usage of the Library

- see [lib](https://github.com/mcmilk/zstdmt/tree/master/lib)

[Brotli]:https://github.com/google/brotli/

[LZ4]:https://github.com/lz4/lz4/

[LZ5]:https://github.com/inikep/lz5/

[Zstandard]:https://github.com/facebook/zstd/

[Lizard]:https://github.com/inikep/lizard/

[Snappy-c]:https://github.com/andikleen/snappy-c

[LZFSE]:https://github.com/lzfse/lzfse

/TR 2020-10-15