{"id":18001442,"url":"https://github.com/egbakou/reedsolomon","last_synced_at":"2025-09-05T22:32:52.587Z","repository":{"id":65597591,"uuid":"517441716","full_name":"egbakou/reedsolomon","owner":"egbakou","description":"Reed-Solomon Erasure Coding in C#/.NET","archived":false,"fork":false,"pushed_at":"2023-09-06T10:49:49.000Z","size":345,"stargazers_count":26,"open_issues_count":0,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-21T11:50:33.500Z","etag":null,"topics":["erasure-coding","reed-solomon","storage"],"latest_commit_sha":null,"homepage":"https://twitter.com/lioncoding","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/egbakou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"egbakou","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2022-07-24T21:33:56.000Z","updated_at":"2024-10-19T04:51:54.000Z","dependencies_parsed_at":"2024-10-29T23:49:48.988Z","dependency_job_id":null,"html_url":"https://github.com/egbakou/reedsolomon","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/egbakou%2Freedsolomon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/egbakou%2Freedsolomon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/egbakou%2Freedsolomon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/egbakou%2Freedsolomon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/egbakou","download_url":"https://codeload.github.com/egbakou/reedsolomon/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245618562,"owners_count":20645023,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["erasure-coding","reed-solomon","storage"],"created_at":"2024-10-29T23:17:31.476Z","updated_at":"2025-03-26T08:30:47.108Z","avatar_url":"https://github.com/egbakou.png","language":"C#","funding_links":["https://github.com/sponsors/egbakou"],"categories":[],"sub_categories":[],"readme":"# \u003cimg src=\"./assets/logo.png\" alt=\"Icon\" width=\"60\" /\u003eReedSolomon.NET \n[![CI](https://github.com/egbakou/reedsolomon/actions/workflows/ci.yml/badge.svg)](https://github.com/egbakou/reedsolomon/actions/workflows/ci.yml) [![NuGet](https://img.shields.io/nuget/v/ReedSolomon.NET.svg?label=NuGet)](https://www.nuget.org/packages/ReedSolomon.NET/)\n\n\nReed-Solomon Erasure coding implementation in C#/.NET.\n\nThis is a .NET Standard 2.1 port of the [JavaReedSolomon](https://github.com/Backblaze/JavaReedSolomon) library released by [Backblaze](http://backblaze.com/), without additional optimizations.\n\n## What is Reed-Solomon?\nReed-Solomon Erasure Coding is a method of data protection that uses a specific type of error-correcting code. It is designed to provide protection against data loss in case of failures or errors. It works by dividing data into smaller blocks and then encoding each block with extra information. This extra information allows the system to reconstruct the original data even if some of the blocks are lost or corrupted. Reed-Solomon Erasure Coding is commonly used in object storage systems, distributed file systems, and other types of storage systems where data durability and reliability are important.\n\nHere is a well-explained article about the topic by [Backblaze](http://backblaze.com/) : https://www.backblaze.com/blog/reed-solomon/\n\n## Motivation\n\nI had a school project last year about creating an object storage system. I choose not to use Azure Blob Storage or Amazon S3 as I had exhausted my student credits. I researched various open-source alternatives such as minio, nextcloud, and seaweedfs, but ultimately decided to develop my own S3 compatible storage system from scratch. I set up four virtual machines, each with two disks. However, the virtual machines were in an unstable environment and I aimed to avoid data replication across them. Through my research, I discovered that minio uses the Reed-Solomon Erasure Coding method. Azure Blob Storage also uses Erasure Coding, but I am not sure if they use the Reed-Solomon version. This is how the project came to realization.\n\nI followed the [Java implementation ](https://github.com/Backblaze/JavaReedSolomon) released by [Backblaze](http://backblaze.com/).\n\n## Installation\n\nReedSolomon.NET can be [found here on NuGet](https://www.nuget.org/packages/ReedSolomon.NET) and can be installed by copying and pasting the following command into your Package Manager Console within Visual Studio (Tools \u003e NuGet Package Manager \u003e Package Manager Console).\n\n```\nInstall-Package ReedSolomon.NET\n```\n\nAlternatively, you can install ReedSolomon.NET via the CLI with the following command:\n\n```\ndotnet add package ReedSolomon.NET\n```\n\n## Usage\n\n Take a look at the `ReedSolomon.NET.Sample` project in the `src` folder. \n\nYou will find these three files: `SampleEncoder`, `SampleDecoder,` and `RemoteDecoder`. They show, in a very simple way, how to break a file into data chunks and parity chunks, and then how to take a subset of the shards and reconstruct the original file.\n\n- `SampleEncoder.cs`\n\nBreak a file into data chunks and parity chunks.\n\n![](assets/encoder.png)\n\n- `SampleDecoder.cs`\n\nReconstruct the original file from data chunks and/or parity chunks.\n\nIn the illustration below, at least 4 \"data chunks + Parity chunks\" must be available, otherwise, the reconstruction will not be possible.\n\n![](assets/decoder.png)\n\n## Performance\n\nThe performance of the inner loop depends on the specific processor you're running on. There are twelve different permutations of the loop in this library, and the `ReedSolomonBenchmark` class will tell you which one is faster for your particular application.\n\nThese are the speeds I got running the benchmark on an AMD Ryzen 5 computer.\n\n```powershell\nInputOutputByteTableCodingLoop 304.8 MB/s\nOutputInputByteTableCodingLoop 296.8 MB/s\nInputOutputByteExpCodingLoop 258.3 MB/s\nOutputByteInputExpCodingLoop 254.5 MB/s\nOutputInputByteExpCodingLoop 241.6 MB/s\nByteOutputInputTableCodingLoop 234.6 MB/s\nByteOutputInputExpCodingLoop 234.2 MB/s\nOutputByteInputTableCodingLoop 232.5 MB/s\nInputByteOutputTableCodingLoop 152.0 MB/s\nByteInputOutputTableCodingLoop 145.5 MB/s\nInputByteOutputExpCodingLoop 134.3 MB/s\nByteInputOutputExpCodingLoop 127.1 MB/s\n```\n\n## Limitation\n\n- File size can't exceed 2GB due to the byte array limitation.\n\nWhen I was working on my project, the maximum size of any file was 1GB, so I didn't care about this.\n\nIf you want to improve this project, check out the [NativeMemoryArray repo](https://github.com/Cysharp/NativeMemoryArray) to remove this limitation and feel free to make a PR.\n\n- See the [Go implementation](https://github.com/klauspost/reedsolomon) to improve the performance of this project.\n\n- You must run the benchmark on your hardware to detect the best inner loop to use.\n\n## Shortcomings\n\n- If the shard number isn't the same for the decoder as in the encoder, invalid output will be generated.\n- If values have changed in a shard, the original file cannot be reconstructed.\n- If two shards have been swapped, reconstruction will always fail. You need to supply the shards in the same order as they were given to you.\n\nThe solution for this is to save a metadata file containing:\n\n- File size.\n\n- The number of data/parity shards.\n\n- HASH of each shard.\n\n- Order of the shards.\n\nIf you save these properties, you should be able to detect file corruption in a shard and be able to reconstruct your data if you have the needed number of shards left.\n\nFor security reasons, you can also encrypt each shard.\n\n## Links\n\n- [Backblaze Open Sources Reed-Solomon Erasure Coding Source Code](https://www.backblaze.com/blog/reed-solomon/).\n- [JavaReedSolomon](https://github.com/Backblaze/JavaReedSolomon). Compatible java library by Backblaze.\n- [klauspost/reedsolomon: Reed-Solomon Erasure Coding in Go (github.com)](https://github.com/klauspost/reedsolomon)\n- [SDC 2018 - Accelerated Erasure Coding: The New Frontier of Software Defined Storage - YouTube](https://www.youtube.com/watch?v=4QFb2Zvr6yc)\n\n- https://github.com/bwbeach)\n\n## License\n\nThis project is licensed under the terms of the MIT license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fegbakou%2Freedsolomon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fegbakou%2Freedsolomon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fegbakou%2Freedsolomon/lists"}