https://github.com/sergey-dryabzhinsky/python-zstd
Simple python bindings to Yann Collet ZSTD compression library
https://github.com/sergey-dryabzhinsky/python-zstd
c compression python zstandard zstd
Last synced: 8 months ago
JSON representation
Simple python bindings to Yann Collet ZSTD compression library
- Host: GitHub
- URL: https://github.com/sergey-dryabzhinsky/python-zstd
- Owner: sergey-dryabzhinsky
- License: bsd-2-clause
- Created: 2015-03-22T07:03:10.000Z (almost 11 years ago)
- Default Branch: master
- Last Pushed: 2025-04-04T10:19:58.000Z (9 months ago)
- Last Synced: 2025-04-06T17:03:39.425Z (9 months ago)
- Topics: c, compression, python, zstandard, zstd
- Language: C
- Homepage:
- Size: 1.3 MB
- Stars: 174
- Watchers: 8
- Forks: 30
- Open Issues: 9
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
=============
python-zstd
=============
.. |releaseW| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/build-wheels.yml/badge.svg?tag=v1.5.6.7
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/build-wheels.yml
.. |masterW| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/build-wheels.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/build-wheels.yml
.. |cpython27x64| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython27_x86_64.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython27_x86_64.yml
.. |cpython27x86| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython27_x86.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython27_x86.yml
.. |cpython27armhf| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython27_armhf.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython27_armhf.yml
.. |cpython34x86| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython34_x86.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython34_x86.yml
.. |cpython34x64| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython34_x86_64.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython34_x86_64.yml
.. |cpython35x86| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython35_x86.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython35_x86.yml
.. |cpython35x64| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython35_x86_64_u20.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython35_x86_64_u20.yml
.. |cpython36x86| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython36_x86.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython36_x86.yml
.. |cpython36x64| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython36_x86_64_u20.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython36_x86_64_u20.yml
.. |cpython37x86| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython37_x86.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython37_x86.yml
.. |cpython37x64| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython37_x86_64_u24.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython37_x86_64_u24.yml
.. |cpython38x86| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython38_x86.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython38_x86.yml
.. |cpython38x64| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython38_x86_64_u24.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython38_x86_64_u24.yml
.. |cpython39x86| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython39_x86.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython39_x86.yml
.. |cpython39x64| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython39_x86_64_u24.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython39_x86_64_u24.yml
.. |cpython310x86u16| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython310_x86_u16.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython310_x86_u16.yml
.. |cpython310x64u20| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython310_x86_64_u24.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython310_x86_64_u24.yml
.. |cpython311x86u16| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython311_x86_u16.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython311_x86_u16.yml
.. |cpython311x64u20| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython311_x86_64_u24.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython311_x86_64_u24.yml
.. |cpython312x86u16| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython312_x86_u16.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython312_x86_u16.yml
.. |cpython312x64u20| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython312_x86_64_u20.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython312_x86_64_u20.yml
.. |cpython313x64u20| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython313_x86_64_u24.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython313_x86_64_u24.yml
.. |cpython314x64u20| image:: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython314_x86_64_u24.yml/badge.svg
:target: https://github.com/sergey-dryabzhinsky/python-zstd/actions/workflows/Build_wheels_for_cpython314_x86_64_u24.yml
+----------------------+---------------------+
| wheels for: | status |
+======================+=====================+
| cpython 2.7 x86 | |cpython27x86| |
+----------------------+---------------------+
| cpython 2.7 x64 | |cpython27x64| |
+----------------------+---------------------+
| cpython 2.7 armhf | |cpython27armhf| |
+----------------------+---------------------+
| cpython 3.4 x86 | |cpython34x86| |
+----------------------+---------------------+
| cpython 3.4 x64 | |cpython34x64| |
+----------------------+---------------------+
| cpython 3.5 x86 | |cpython35x86| |
+----------------------+---------------------+
| cpython 3.5 x64 u20 | |cpython35x64| |
+----------------------+---------------------+
| cpython 3.6 x86 | |cpython36x86| |
+----------------------+---------------------+
| cpython 3.6 x64 u20 | |cpython36x64| |
+----------------------+---------------------+
| cpython 3.7 x86 | |cpython37x86| |
+----------------------+---------------------+
| cpython 3.7 x64 u20 | |cpython37x64| |
+----------------------+---------------------+
| cpython 3.8 x86 | |cpython38x86| |
+----------------------+---------------------+
| cpython 3.8 x64 u20 | |cpython38x64| |
+----------------------+---------------------+
| cpython 3.9 x86 | |cpython39x86| |
+----------------------+---------------------+
| cpython 3.9 x64 u22 | |cpython39x64| |
+----------------------+---------------------+
| cpython 3.10 x86 u16 | |cpython310x86u16| |
+----------------------+---------------------+
| cpython 3.10 x64 u20 | |cpython310x64u20| |
+----------------------+---------------------+
| cpython 3.11 x86 u16 | |cpython311x86u16| |
+----------------------+---------------------+
| cpython 3.11 x64 u20 | |cpython311x64u20| |
+----------------------+---------------------+
| cpython 3.12 x86 u16 | |cpython312x86u16| |
+----------------------+---------------------+
| cpython 3.12 x64 u20 | |cpython312x64u20| |
+----------------------+---------------------+
| cpython 3.13 x64 u20 | |cpython313x64u20| |
+----------------------+---------------------+
| cpython 3.14 x64 u24 | |cpython314x64u20| |
+----------------------+---------------------+
| Release | |releaseW| |
| 1.5.6.7 | |
+----------------------+---------------------+
| Master | |masterW| |
+----------------------+---------------------+
Simple python bindings to Yann Collet ZSTD compression library.
**Zstd**, short for Zstandard, is a new lossless compression algorithm,
which provides both good compression ratio *and* speed for your standard compression needs.
"Standard" translates into everyday situations which neither look for highest possible ratio
(which LZMA and ZPAQ cover) nor extreme speeds (which LZ4 covers).
It is provided as a BSD-license package, hosted on GitHub_.
.. _GitHub: https://github.com/facebook/zstd
WARNING!!!
----------
If you setup 1.0.0.99.1 version - remove it manualy to able to update.
PIP matching version strings not tuple of numbers.
Result generated by versions prior to 1.0.0.99.1 is not compatible with orignial Zstd
by any means. It generates custom header and can be read only by zstd python module.
As of 1.0.0.99.1 version it uses standard Zstd output, not modified.
To prevent data loss there is two functions now: ```compress_old``` and ```decompress_old```.
They are works just like in old versions prior to 1.0.0.99.1.
As of 1.1.4 version module build without them by default.
As of 1.3.4 version these functions are deprecated and will be removed in future releases.
As of 1.5.0 version these functions are removed.
DISCLAIMER
__________
These python bindings are kept simple and blunt.
Support of dictionaries and streaming is not planned.
LINKS
-----
* Zstandard: https://github.com/facebook/zstd
* More full-featured and compatible with Zstandard python bindings by Gregory Szorc: https://github.com/indygreg/python-zstandard
Build from source
-----------------
>>> $ git clone https://github.com/sergey-dryabzhinsky/python-zstd
>>> $ git submodule update --init
>>> $ apt-get install python-dev python3-dev python-setuptools python3-setuptools
>>> $ python setup.py build_ext clean
>>> $ python3 setup.py build_ext clean
And you need to install `libzstd` developer files at least version *1.4.0*:
>>> $ dnf install -y libzstd-devel
# or
>>> $ apt install -y libzstd-dev
# or
>>> $ apk add zstd-dev
or do manual installation of zstd from source.
Note: Zstd legacy format support disabled by default.
To build with Zstd legacy versions support - pass ``--legacy`` option to setup.py script:
>>> $ python setup.py build_ext --legacy clean
When using a PEP 517 builder you can use ``ZSTD_LEGACY`` environment variable instead:
>>> $ ZSTD_LEGACY=1 python -m build -w
Note: Python-Zstd legacy format support removed since 1.5.0.
If you need to convert old data - checkout 1.4.9.1 module version. Support of it disabled by default.
To build with python-zstd legacy format support (pre 1.1.2) - pass ``--pyzstd-legacy`` option to setup.py script:
>>> $ python setup.py build_ext --pyzstd-legacy clean
But beware! Legacy formats support state is unknown in this case.
And if your version not equal with python-zstd - tests may not pass.
If you're scared of threads you may pass option `--libzstd-no-threads`:
>>> $ python setup.py build_ext --libzstd-no-threads clean
When using a PEP 517 builder you can use ``ZSTD_THREADS`` environment variable instead:
>>> $ ZSTD_THREADS=0 python -m build -w
If you're meet some cpu instruction errorrs you may try to disable built-in optimizations and pass option `--libzstd-no-use-asm`:
>>> $ python setup.py build_ext --libzstd-no-use-asm clean
Or add more speed with option `--libzstd-use-asm-bmi2` to use instructions for new AMD CPU.
When using a PEP 517 builder you can use ``ZSTD_ASM`` environment variable instead:
And ``ZST_ASM_BMI2=1`` too for bmi2 use.
>>> $ ZSTD_ASM=0 python -m build -w
If you want bo build smaller module by size try to use option `--small`, but it will work slower.
>>> $ python setup.py build_ext --small clean
When using a PEP 517 builder you can use ``ZSTD_SMALL`` environment variable instead:
>>> $ ZSTD_SMALL=1 python -m build -w
If you want to build with existing distribution of libzstd just add ``--external`` option
>>> $ python setup.py build_ext --external clean
When using a PEP 517 builder you can use ``ZSTD_EXTERNAL`` environment variable instead:
>>> $ ZSTD_EXTERNAL=1 python -m build -w
If paths to header file ``zstd.h`` and libraries is uncommon - use common ``build`` params:
--libraries --include-dirs --library-dirs.
>>> $ python setup.py build_ext --external --include-dirs /opt/zstd/usr/include --libraries zstd --library-dirs /opt/zstd/lib clean
But If you want to force build with bundled distribution of libzstd just add ``--
libzstd-bundled`` option
>>> $ python setup.py build_ext --libzstd-bundled clean
When using a PEP 517 builder you can use ``ZSTD_BUNDLED`` environment variable instead:
>>> $ ZSTD_BUNDLED=1 python -m build -w
If you want to check if build w/o any warnings just add ``--
all-warnings`` option
>>> $ python setup.py build_ext --all-warnings clean
When using a PEP 517 builder you can use ``ZSTD_WARNINGS`` environment variable instead:
>>> $ ZSTD_WARNINGS=1 python -m build -w
If you want to treat all warnings as errors just add ``--
all-warnings-errors`` option
>>> $ python setup.py build_ext --all-warnings-errors clean
When using a PEP 517 builder you can use ``ZSTD_WERRORS`` environment variable instead:
>>> $ ZSTD_WERRORS=1 python -m build -w
Install from pypi
-----------------
>>> # for Python 2.7+
>>> $ pip install zstd
>>> # or for Python 3.4+
>>> $ pip3 install zstd
API
___
Error
Standard python Exception for zstd module
ZSTD_compress (data[, level, threads, strict]): string|bytes
Function, compress input data block via mutliple threads, return compressed block, or raises Error.
Params:
* **data**: string|bytes - input data block, length limited by 2Gb by Python API
* **level**: int - compression level, ultra-fast levels from -100 (ultra) to -1 (fast) available since zstd-1.3.4, and from 1 (fast) to 22 (slowest), 0 or unset - means default (3). Default - 3.
* **threads**: int - how many threads to use, from 0 to 200, 0 or unset - auto-tune by cpu cores count. Default - 0. Since: 1.4.4.1
* **strict**: int - strict behaviour, raise `zstd.Error` if threads number or compression level is beyond limitations. Default - 0. Since: 1.5.6.3
Aliases:
- *compress(...)*,
- *dumps(...)*,
- *encode(...)* since: 1.5.6.2
Exception if:
- level bigger than max level
Max number of threads:
- 32bit system: 64
- 64bit system: 256
If provided bigger number - silently set maximber (since 1.5.4.1)
Since: 0.1
ZSTD_uncompress (data): string|bytes
Function, decompress input compressed data block, return decompressed block, or raises Error.
Support compressed data with multiple/concatenated frames (blocks) (since 1.5.5.1).
Params:
* **data**: string|bytes - input compressed data block, length limited by 2Gb by Python API
Aliases:
- *decompress(...)*,
- *uncompress(...)*,
- *loads(...)*,
- *decode(...)* since: 1.5.6.2
Since: 0.1
ZSTD_check (data): int
Function, checks if input is zstd compressed data block, returns 1 if yes, 0 if no.
Support compressed data with multiple/concatenated frames (blocks) .
Params:
* **data**: string|bytes - input compressed data block, length limited by 2Gb by Python API
Aliases:
- *check(...)*,
- *verify(...)* since: 1.5.6.3
Since: 1.5.6.2
version (): string|bytes
Returns this module doted version string.
The first three digits are folow libzstd version.
Fourth digit - module revision number for that version.
Since: 1.3.4.3
ZSTD_version (): string|bytes
Returns ZSTD library doted version string.
Since: 1.3.4.3
ZSTD_version_number (): int
Returns ZSTD library version in format: MAJOR*100*100 + MINOR*100 + RELEASE.
Since: 1.3.4.3
ZSTD_threads_count (): int
Returns ZSTD determined CPU cores count.
Since: 1.5.4.1
ZSTD_max_threads_count (): int
Returns ZSTD library determined maximum working threads count.
Since: 1.5.4.1
ZSTD_max_compression_level (): int
Returns ZSTD library determined maximum number of compression level .
Since: 1.5.6.3
ZSTD_min_compression_level (): int
Returns ZSTD library determined minimum number of compression level .
Since: 1.5.6.3
ZSTD_external (): int
Returns 0 of 1 if ZSTD library linked as external.
Since: 1.5.0.2
ZSTD_legacy_support (): int
Returns 0 of 1 if ZSTD library built with legacy formats support.
Since: 1.5.6.3
ZSTD_with_threads (): int
Returns 0 of 1 if bundled ZSTD library build with threads support.
Since: 1.5.6.2
ZSTD_with_asm (): int
Returns 0 of 1 if bundled ZSTD library build with asm optimization s.
Since: 1.5.6.2
Removed
_______
ZSTD_compress_old (data[, level]): string|bytes
Function, compress input data block, return compressed block, or raises Error.
**DEPRECATED**: Returns not compatible with ZSTD block header
**REMOVED**: since 1.5.0
Params:
* **data**: string|bytes - input data block, length limited by 2Gb by Python API
* **level**: int - compression level, ultra-fast levels from -5 (ultra) to -1 (fast) available since zstd-1.3.4, and from 1 (fast) to 22 (slowest), 0 or unset - means default (3). Default - 3.
Since: 1.0.0.99.1
ZSTD_uncompress_old (data): string|bytes
Function, decompress input compressed data block, return decompressed block, or raises Error.
**DEPRECATED**: Accepts data with not compatible with ZSTD block header
**REMOVED**: since 1.5.0
Params:
* **data**: string|bytes - input compressed data block, length limited by 2Gb by Python API
Since: 1.0.0.99.1
Use
___
Module has simple API:
>>> import zstd
>>> dir(zstd)
['Error', 'ZSTD_compress', 'ZSTD_external', 'ZSTD_uncompress', 'ZSTD_version', 'ZSTD_version_number', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'compress', 'decompress', 'dumps', 'loads', 'uncompress', 'version']
>>> zstd.version()
'1.5.1.0'
>>> zstd.ZSTD_version()
'1.5.1'
>>> zstd.ZSTD_version_number()
10501
>>> zstd.ZSTD_external()
0
In python2
>>> data = "123456qwert"
In python3 use bytes
>>> data = b"123456qwert"
>>> cdata = zstd.compress(data, 1)
>>> data == zstd.decompress(cdata)
True
>>> cdata_mt = zstd.compress(data, 1, 4)
>>> cdata == cdata_mt
True
>>> data == zstd.decompress(cdata_mt)
True