https://github.com/ap--/concatenated-zlib
playground to test decompression of concatenated zlib chunks
https://github.com/ap--/concatenated-zlib
Last synced: 4 months ago
JSON representation
playground to test decompression of concatenated zlib chunks
- Host: GitHub
- URL: https://github.com/ap--/concatenated-zlib
- Owner: ap--
- Created: 2023-06-14T19:30:32.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-06-15T12:39:10.000Z (about 3 years ago)
- Last Synced: 2025-10-23T09:51:33.146Z (8 months ago)
- Language: Cython
- Size: 42 KB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# concatenated_zlib
this is a playground repository to experiment with an zlib and zlibng
decompressor that can handle zlib compressed concatenated chunks.
The code here is basically copied from the excellent `imagecodecs` package:
https://github.com/cgohlke/imagecodecs/
and modified to support concatenated input.
### description
The below fails to decompress concatenated zlib compressed blocks:
```python
import zlib
c0 = zlib.compress(b"abc")
c1 = zlib.compress(b"efg")
print(zlib.decompress(c0 + c1)) # returns b"abc"
```
This repo provides a zlib and zlibng based decompressor that supports concatenated
input:
```python
import zlib
from concatenated_zlib import zlib_concat_decode
from concatenated_zlib import zlibng_concat_decode
c0 = zlib.compress(b"abc")
c1 = zlib.compress(b"efg")
print(zlib_concat_decode(c0 + c1)) # returns b"abcefg"
print(zlibng_concat_decode(c0 + c1)) # returns b"abcefg"
```
### benchmarks
To reproduce:
```shell
# ON A MAC YOU CAN INSTALL THE DEPENDECIES VIA homebrew
brew install zlib zlib-ng libdeflate
# ^^^ install the corresponding *-dev packages on your OS accordingly
git clone --recursive https://github.com/ap--/concatenated-zlib.git
cd concatenated-zlib/
python3 -m venv venv
source venv/bin/activate
python3 -m pip install .
cd bench; python time_load_chunks.py
```
Run `cd bench; python time_load_chunks.py` to get some crude timing for the different
approaches... Weirdly enough the cython implementation doesn't currently beat
the plain python stdlib decompression of individual chunks. Needs more
investigation...
### related info
- https://stackoverflow.com/a/60208718 decompress concatenated zlib chunks