{"id":14109700,"url":"https://github.com/openstack/pyeclib","last_synced_at":"2025-10-09T19:05:08.540Z","repository":{"id":44137413,"uuid":"58626986","full_name":"openstack/pyeclib","owner":"openstack","description":"A simple Python interface for implementing erasure codes. Mirror of code maintained at opendev.org.","archived":false,"fork":false,"pushed_at":"2025-09-17T22:24:10.000Z","size":16931,"stargazers_count":73,"open_issues_count":0,"forks_count":19,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-10-09T19:04:31.740Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://opendev.org/openstack/pyeclib","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openstack.png","metadata":{"files":{"readme":"README.rst","changelog":"ChangeLog","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2016-05-12T09:19:58.000Z","updated_at":"2025-09-22T08:48:03.000Z","dependencies_parsed_at":"2024-08-14T10:12:22.686Z","dependency_job_id":"4a90b4e5-55c5-46f4-b92b-6209b6785ede","html_url":"https://github.com/openstack/pyeclib","commit_stats":null,"previous_names":[],"tags_count":40,"template":false,"template_full_name":null,"purl":"pkg:github/openstack/pyeclib","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openstack%2Fpyeclib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openstack%2Fpyeclib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openstack%2Fpyeclib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openstack%2Fpyeclib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openstack","download_url":"https://codeload.github.com/openstack/pyeclib/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openstack%2Fpyeclib/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001983,"owners_count":26083243,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-14T10:02:27.252Z","updated_at":"2025-10-09T19:05:08.534Z","avatar_url":"https://github.com/openstack.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"PyEClib\n-------\n\nThis library provides a simple Python interface for implementing erasure codes\nand is known to work with Python 3.10 through 3.13. To obtain the best\npossible performance, the library utilizes liberasurecode, which is a C based\nerasure code library.\n\nPyECLib supports a variety of Erasure Coding backends including the standard\nReed-Solomon implementations provided by Jerasure [1], liberasurecode [3],\nIntel's ISA-L [4] and Phazr.IO's libphazr.  It also provides support for a flat\nXOR-based encoder and decoder (part of liberasurecode) - a class of HD\nCombination Codes based on \"Flat XOR-based erasure codes in storage systems:\nConstructions, efficient recovery, and tradeoffs\" in IEEE MSST 2010[2]).\nThese codes are well-suited to archival use-cases, have a simple construction\nand require a minimum number of participating disks during single-disk\nreconstruction (think XOR-based LRC code).\n\n-----\n\nInstallation\n============\n\nInstall pre-requisites:\n\n* Python 3.10+ (including development packages), argparse, setuptools\n* liberasurecode v1.4.0 or greater [3]\n* Erasure code backend libraries, gf-complete and Jerasure [1], ISA-L [4], etc.\n\nInstall dependencies:\n\nDebian/Ubuntu hosts::\n\n    $ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev\n    $ sudo pip install -U bindep -r test-requirements.txt\n\nRHEL/CentOS hosts::\n    \n    $ sudo yum install -y redhat-lsb python2-pip python-devel liberasurecode-devel\n    $ sudo pip install -U bindep -r test-requirements.txt\n    $ tools/test-setup.sh\n\nIf you want to confirm all dependency packages installed successfully, try::\n\n    $ sudo bindep -f bindep.txt\n\nFor CentOS, make sure to install the latest Openstack Cloud SIG repo\nto be able to install the latest available version of liberasurecode-devel.\n\nInstall PyECLib::\n\n    $ sudo python setup.py install\n\nRun test suite included::\n\n    $ ./.unittests\n\nIf the test suite fails because it cannot find any of the shared libraries,\nthen you probably need to add /usr/local/lib to the path searched when loading\nlibraries.  The best way to do this (on Linux) is to add '/usr/local/lib' to::\n\n    /etc/ld.so.conf\n\nand then make sure to run::\n\n    $ sudo ldconfig\n\n-----\n\nGetting started\n===============\n\nExamples of using PyECLib are provided in the \"tools\" directory:\n\nCommand-line encoder::\n\n    tools/pyeclib_encode.py\n\nCommand-line decoder::\n\n    tools/pyeclib_decode.py\n\nUtility to determine what is needed to reconstruct missing fragments::\n\n    tools/pyeclib_fragments_needed.py\n\nA configuration utility to help compare available EC schemes in terms of\nperformance and redundancy::\n\n    tools/pyeclib_conf_tool.py\n\nPyEClib initialization::\n\n    ec_driver = ECDriver(k=\u003cnum_encoded_data_fragments\u003e,\n                         m=\u003cnum_encoded_parity_fragments\u003e,\n                         ec_type=\u003cec_scheme\u003e))\n\nSupported ``ec_type`` values:\n\n* ``liberasurecode_rs_vand`` =\u003e Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]\n* ``jerasure_rs_vand`` =\u003e Vandermonde Reed-Solomon encoding, based on Jerasure [1]\n* ``jerasure_rs_cauchy`` =\u003e Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]\n* ``flat_xor_hd_3``, ``flat_xor_hd_4`` =\u003e Flat-XOR based HD combination codes, liberasurecode [3]\n* ``isa_l_rs_vand`` =\u003e Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]\n* ``isa_l_rs_cauchy`` =\u003e Cauchy Reed-Solomon encoding (ISA-L variant) [4]\n* ``shss`` =\u003e NTT Lab Japan's Erasure Coding Library [5]\n* ``libphazr`` =\u003e Phazr.IO's erasure code library with built-in privacy [6]\n\n-----\n\nCode Maintenance\n================\n\nThis library is currently mainly maintained by the Openstack Swift community.\nFor questions or any other help, come ask in #openstack-swift on OFTC.\n\n-----\n\nReferences\n==========\n\n[1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org\n\n[2] Greenan, Kevin M et al, \"Flat XOR-based erasure codes in storage systems\", https://web.archive.org/web/20161001210233/https://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf\n\n[3] liberasurecode, C API abstraction layer for erasure coding backends, https://opendev.org/openstack/liberasurecode\n\n[4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version\n\n[5] Kota Tsuyuzaki \u003ctsuyuzaki.kota@lab.ntt.co.jp\u003e, \"NTT SHSS Erasure Coding backend\"\n\n[6] Jim Cheung \u003csupport@phazr.io\u003e, \"Phazr.IO libphazr erasure code backend with built-in privacy\"\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenstack%2Fpyeclib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenstack%2Fpyeclib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenstack%2Fpyeclib/lists"}