https://github.com/openstack/pyeclib
A simple Python interface for implementing erasure codes. Mirror of code maintained at opendev.org.
https://github.com/openstack/pyeclib
Last synced: 3 months ago
JSON representation
A simple Python interface for implementing erasure codes. Mirror of code maintained at opendev.org.
- Host: GitHub
- URL: https://github.com/openstack/pyeclib
- Owner: openstack
- License: bsd-2-clause
- Created: 2016-05-12T09:19:58.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2025-09-17T22:24:10.000Z (3 months ago)
- Last Synced: 2025-10-09T19:04:31.740Z (3 months ago)
- Language: Python
- Homepage: https://opendev.org/openstack/pyeclib
- Size: 16.1 MB
- Stars: 73
- Watchers: 9
- Forks: 19
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- Changelog: ChangeLog
- License: LICENSE
- Authors: AUTHORS
Awesome Lists containing this project
README
PyEClib
-------
This library provides a simple Python interface for implementing erasure codes
and is known to work with Python 3.10 through 3.13. To obtain the best
possible performance, the library utilizes liberasurecode, which is a C based
erasure code library.
PyECLib supports a variety of Erasure Coding backends including the standard
Reed-Solomon implementations provided by Jerasure [1], liberasurecode [3],
Intel's ISA-L [4] and Phazr.IO's libphazr. It also provides support for a flat
XOR-based encoder and decoder (part of liberasurecode) - a class of HD
Combination Codes based on "Flat XOR-based erasure codes in storage systems:
Constructions, efficient recovery, and tradeoffs" in IEEE MSST 2010[2]).
These codes are well-suited to archival use-cases, have a simple construction
and require a minimum number of participating disks during single-disk
reconstruction (think XOR-based LRC code).
-----
Installation
============
Install pre-requisites:
* Python 3.10+ (including development packages), argparse, setuptools
* liberasurecode v1.4.0 or greater [3]
* Erasure code backend libraries, gf-complete and Jerasure [1], ISA-L [4], etc.
Install dependencies:
Debian/Ubuntu hosts::
$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
$ sudo pip install -U bindep -r test-requirements.txt
RHEL/CentOS hosts::
$ sudo yum install -y redhat-lsb python2-pip python-devel liberasurecode-devel
$ sudo pip install -U bindep -r test-requirements.txt
$ tools/test-setup.sh
If you want to confirm all dependency packages installed successfully, try::
$ sudo bindep -f bindep.txt
For CentOS, make sure to install the latest Openstack Cloud SIG repo
to be able to install the latest available version of liberasurecode-devel.
Install PyECLib::
$ sudo python setup.py install
Run test suite included::
$ ./.unittests
If the test suite fails because it cannot find any of the shared libraries,
then you probably need to add /usr/local/lib to the path searched when loading
libraries. The best way to do this (on Linux) is to add '/usr/local/lib' to::
/etc/ld.so.conf
and then make sure to run::
$ sudo ldconfig
-----
Getting started
===============
Examples of using PyECLib are provided in the "tools" directory:
Command-line encoder::
tools/pyeclib_encode.py
Command-line decoder::
tools/pyeclib_decode.py
Utility to determine what is needed to reconstruct missing fragments::
tools/pyeclib_fragments_needed.py
A configuration utility to help compare available EC schemes in terms of
performance and redundancy::
tools/pyeclib_conf_tool.py
PyEClib initialization::
ec_driver = ECDriver(k=,
m=,
ec_type=))
Supported ``ec_type`` values:
* ``liberasurecode_rs_vand`` => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]
* ``jerasure_rs_vand`` => Vandermonde Reed-Solomon encoding, based on Jerasure [1]
* ``jerasure_rs_cauchy`` => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]
* ``flat_xor_hd_3``, ``flat_xor_hd_4`` => Flat-XOR based HD combination codes, liberasurecode [3]
* ``isa_l_rs_vand`` => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]
* ``isa_l_rs_cauchy`` => Cauchy Reed-Solomon encoding (ISA-L variant) [4]
* ``shss`` => NTT Lab Japan's Erasure Coding Library [5]
* ``libphazr`` => Phazr.IO's erasure code library with built-in privacy [6]
-----
Code Maintenance
================
This library is currently mainly maintained by the Openstack Swift community.
For questions or any other help, come ask in #openstack-swift on OFTC.
-----
References
==========
[1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org
[2] Greenan, Kevin M et al, "Flat XOR-based erasure codes in storage systems", https://web.archive.org/web/20161001210233/https://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf
[3] liberasurecode, C API abstraction layer for erasure coding backends, https://opendev.org/openstack/liberasurecode
[4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
[5] Kota Tsuyuzaki , "NTT SHSS Erasure Coding backend"
[6] Jim Cheung , "Phazr.IO libphazr erasure code backend with built-in privacy"