{"id":17299961,"url":"https://github.com/f4exb/cm256cc","last_synced_at":"2025-04-14T11:25:45.282Z","repository":{"id":54581339,"uuid":"64006460","full_name":"f4exb/cm256cc","owner":"f4exb","description":"Fast GF(256) Cauchy MDS Block Erasure Codec in C++","archived":false,"fork":false,"pushed_at":"2024-02-29T12:51:43.000Z","size":1902,"stargazers_count":24,"open_issues_count":3,"forks_count":8,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-28T00:41:53.618Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/f4exb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-07-23T08:14:57.000Z","updated_at":"2025-02-09T16:17:31.000Z","dependencies_parsed_at":"2024-02-29T14:09:24.660Z","dependency_job_id":null,"html_url":"https://github.com/f4exb/cm256cc","commit_stats":{"total_commits":108,"total_committers":5,"mean_commits":21.6,"dds":"0.39814814814814814","last_synced_commit":"fbdffb2e74d926fa6ecf586268690496fe69483e"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f4exb%2Fcm256cc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f4exb%2Fcm256cc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f4exb%2Fcm256cc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f4exb%2Fcm256cc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/f4exb","download_url":"https://codeload.github.com/f4exb/cm256cc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248870311,"owners_count":21175015,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T11:24:56.081Z","updated_at":"2025-04-14T11:25:45.254Z","avatar_url":"https://github.com/f4exb.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cm256cc\nFast GF(256) Cauchy MDS Block Erasure Codec in C++\n\nThis is the rewrite in (as much as possible) clean C++ of [cm256](https://github.com/f4exb/cm256). In some contexts like Qt programs and plugins the original cm256 library does not work.\n\ncm256cc performance is on par or even better than cm256. This is particularly true for armv7 architecture (Raspberry Pi 2 and 3) and is the most significant with Raspberry Pi 2.\n\ncm256cc is a simple library for erasure codes.  From given data it generates\nredundant data that can be used to recover the originals.\n\nCurrently only g++ is supported, other versions of MSVC than Visual Studio 2013 may work. Optimizations for both SSE3 (x86_64) and Neon (armv7) are available.\n\nThe original data should be split up into equally-sized chunks.  If one of these chunks\nis erased, the redundant data can fill in the gap through decoding.\n\nThe erasure code is parameterized by three values (`OriginalCount`, `RecoveryCount`, `BlockBytes`).  These are:\n\n+ The number of blocks of original data (`OriginalCount`), which must be less than 256.\n+ The number of blocks of redundant data (`RecoveryCount`), which must be no more than `256 - OriginalCount`.\n\nFor example, if a file is split into 3 equal pieces and sent over a network, `OriginalCount` is 3.\nAnd if 2 additional redundant packets are generated, `RecoveryCount` is 2.\nIn this case up to 256 - 3 = 253 additional redundant packets can be generated.\n\n\n##### Building: Quick Setup\n\nThis is a classical cmake project. Make sure cmake and g++ is installed in your system. create a `build` directory and cd into it. If you install the library in a custom location say `opt/install/cm256cc` use the following command line for cmake:\n\n  - `cmake -Wno-dev -DCMAKE_INSTALL_PREFIX=/opt/install/cm256cc ..`\n  \nResult:\n\n  - Library will be installed as `/opt/install/cm256cc/lib/libcm256cc.so`\n  - Include files will be installed in `/opt/install/cm256cc/include/cm256cc`\n  - Binary test programs will be installed in `/opt/install/cm256cc/bin`\n\n##### Building: Use the library\n\nInclude the cm256cc library in your project and cm256.h header in your program. Have a look at example programs `cm256_test.cpp`, `transmit.cpp`and `receive.cpp` in the `unit_test` folder for usage. Consult the `cm256.h header` for details on the encoding / decoding method.\n\n## Compilation\n\nThis is a classical cmake project. You may install the software anywhere you like with the `-DCMAKE_INSTALL_PREFIX` definition on the cmake command line. \n\nThe cmake file will try to find the best compiler optimization options depending on the hardware you are compiling this project. This may not be suitable if you intend to distribute the software or include it in a distribution. In this case you can use the `-DENABLE_DISTRIBUTION=1` define on the command line to have just SSSE3 optimization for the x86 based systems and still NEON optimization for arm or arm64.\n\n## Usage\n\nDocumentation is provided in the header file [cm256.h](https://github.com/catid/cm256/raw/master/cm256.h).\n\nWhen your application starts up it should call `isInitialized()` to verify that the library is constructed properly:\n\n~~~\n\t#include \"cm256.h\"\n\n    CM256 cm256;\n\n\tif (!cm256.isInitialized()) {\n\t\t// library not initialized\n\t\texit(1);\n\t}\n~~~\n\nTo generate redundancy, use the `cm256_encode` function.  To solve for the original data use the `cm256_decode` function.\n\nExample usage:\n\n~~~\nbool ExampleFileUsage()\n{\n    CM256 cm256;\n\n    if (!cm256.isInitialized()) {\n        // library not initialized\n        exit(1);\n    }\n\n    CM256::cm256_encoder_params params;\n\n    // Number of bytes per file block\n    params.BlockBytes = 4321;\n\n    // Number of blocks\n    params.OriginalCount = 33;\n\n    // Number of additional recovery blocks generated by encoder\n    params.RecoveryCount = 12;\n\n    // Size of the original file\n    static const int OriginalFileBytes = params.OriginalCount * params.BlockBytes;\n\n    // Allocate and fill the original file data\n    uint8_t* originalFileData = new uint8_t[OriginalFileBytes];\n    memset(originalFileData, 1, OriginalFileBytes);\n\n    // Pointers to data\n    CM256::cm256_block blocks[256];\n    for (int i = 0; i \u003c params.OriginalCount; ++i)\n    {\n        blocks[i].Block = originalFileData + i * params.BlockBytes;\n    }\n\n    // Recovery data\n    uint8_t* recoveryBlocks = new uint8_t[params.RecoveryCount * params.BlockBytes];\n\n    // Generate recovery data\n    if (cm256.cm256_encode(params, blocks, recoveryBlocks))\n    {\n        exit(1);\n    }\n\n    // Initialize the indices\n    for (int i = 0; i \u003c params.OriginalCount; ++i)\n    {\n        blocks[i].Index = CM256::cm256_get_original_block_index(params, i);\n    }\n\n    //// Simulate loss of data, subsituting a recovery block in its place ////\n    blocks[0].Block = recoveryBlocks; // First recovery block\n    blocks[0].Index = cm256_get_recovery_block_index(params, 0); // First recovery block index\n    //// Simulate loss of data, subsituting a recovery block in its place ////\n\n    if (cm256.cm256_decode(params, blocks))\n    {\n        exit(1);\n    }\n\n    // blocks[0].Index will now be 0.\n\n    delete[] originalFileData;\n    delete[] recoveryBlocks;\n\n    return true;\n}\n~~~\n\nThe example above is just one way to use the `cm256_decode` function.\n\nThis API was designed to be flexible enough for UDP/IP-based file transfer where\nthe blocks arrive out of order.\n\n\n#### Comparisons with Other Libraries\n\nThe approach taken in CM256 is similar to the Intel Storage Acceleration Library (ISA-L) available here:\n\nhttps://01.org/intel%C2%AE-storage-acceleration-library-open-source-version/downloads\n\nISA-L more aggressively optimizes the matrix multiplication operation, which is the most expensive step of encoding.\n\nCM256 takes better advantage of the m=1 case and the first recovery symbol, which is also possible with the Vandermonde matrices supported by ISA-L.\n\nISA-L uses a O(N^3) Gaussian elimination solver for decoding.  The CM256 decoder solves the linear system using a fast O(N^2) LDU-decomposition algorithm from \"Pivoting and Backward Stability of Fast Algorithms for Solving Cauchy Linear Equations\" (T. Boros, T. Kailath, V. Olshevsky), which was hand-optimized for memory accesses.\n\n\n#### Credits\n\nThis software was written entirely by Christopher A. Taylor \u003cmrcatid@gmail.com\u003e and converted to clean C++ code by myself Edouard M. Griffiths \u003cf4exb06@gmail.com\u003e.  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ff4exb%2Fcm256cc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ff4exb%2Fcm256cc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ff4exb%2Fcm256cc/lists"}