{"id":37136399,"url":"https://github.com/openacid/reedsolomon","last_synced_at":"2026-01-14T15:53:49.772Z","repository":{"id":57561660,"uuid":"329202396","full_name":"openacid/reedsolomon","owner":"openacid","description":"Reed-Solomon Erasure Code engine in Go, could more than 15GB/s per core","archived":false,"fork":true,"pushed_at":"2021-01-13T06:52:43.000Z","size":631,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-06-20T06:36:02.017Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"templexxx/reedsolomon","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openacid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-13T05:21:48.000Z","updated_at":"2021-01-13T05:21:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/openacid/reedsolomon","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/openacid/reedsolomon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openacid%2Freedsolomon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openacid%2Freedsolomon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openacid%2Freedsolomon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openacid%2Freedsolomon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openacid","download_url":"https://codeload.github.com/openacid/reedsolomon/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openacid%2Freedsolomon/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28425558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T15:24:48.085Z","status":"ssl_error","status_checked_at":"2026-01-14T15:23:41.940Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-14T15:53:49.086Z","updated_at":"2026-01-14T15:53:49.765Z","avatar_url":"https://github.com/openacid.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reed-Solomon\n\n[![GoDoc][1]][2] [![MIT licensed][3]][4] [![Build Status][5]][6] [![Go Report Card][7]][8] [![Sourcegraph][9]][10]\n\n[1]: https://godoc.org/github.com/templexxx/reedsolomon?status.svg\n[2]: https://godoc.org/github.com/templexxx/reedsolomon\n[3]: https://img.shields.io/badge/license-MIT-blue.svg\n[4]: LICENSE\n[5]: https://github.com/templexxx/reedsolomon/workflows/unit-test/badge.svg\n[6]: https://github.com/templexxx/reedsolomon\n[7]: https://goreportcard.com/badge/github.com/templexxx/reedsolomon\n[8]: https://goreportcard.com/report/github.com/templexxx/reedsolomon\n[9]: https://sourcegraph.com/github.com/templexxx/reedsolomon/-/badge.svg\n[10]: https://sourcegraph.com/github.com/templexxx/reedsolomon?badge\n\n\n## Introduction:\n\n\u003e- Erasure Codes(based on Reed-Solomon Codes) engine in pure Go.\n\u003e\n\u003e- It's a kind of [Systematic Codes](https://en.wikipedia.org/wiki/Systematic_code), which means \nthe input data is embedded in the encoded output .\n\u003e\n\u003e- [High Performance](https://github.com/templexxx/reedsolomon#performance): More than 15GB/s per physics core. \n\u003e\n\u003e- High Reliability: \n\u003e  1. At least two companies are using this library in their storage system.\n    (More than dozens PB data)\n\u003e  2. Full test of galois field calculation and invertible matrices\n\u003e   (You can also find the [mathematical proof](invertible.jpg) in this repo).\n\u003e\n\u003e- Based on [Klauspost ReedSolomon](https://github.com/klauspost/reedsolomon) \n\u0026 [Intel ISA-L](https://github.com/01org/isa-l) with some additional changes/optimizations.\n\u003e\n\u003e- It's the backend of [XRS](https://github.com/templexxx/xrs) (Erasure Codes\nwhich can save about 30% I/O in reconstruction process).\n\n## Specification\n### Math\n\n\u003e- Coding over in GF(2^8).\n\u003e\n\u003e- Primitive Polynomial: x^8 + x^4 + x^3 + x^2 + 1 (0x1d).\n\u003e\n\u003e- [Cauchy Matrix](matrix.go) is the generator matrix.\n\u003e   \u003e-  Any submatrix of encoding matrix is invertible (See the proof [here](invertible.jpg)). \n\u003e\n\u003e- [Galois Field Tool](mathtool/gentbls.go): Generate primitive polynomial \nand it's log, exponent, multiply and inverse tables etc. \n\u003e\n\u003e- [Inverse Matrices Tool](mathtool/combi.go): Calculate the number of inverse matrices \nwith specific data \u0026 parity number.\n\u003e\n\n[XP](https://github.com/drmingdrmer) has written an excellent article ([Here, in Chinese](http://drmingdrmer.github.io/tech/distributed/2017/02/01/ec.html)) about how\nErasure Codes works and the math behind it. It's a good start to read it.\n\n### Accelerate\n\n\u003e- SIMD: [Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions](http://web.eecs.utk.edu/~jplank/plank/papers/FAST-2013-GF.html)\n\u003e\n\u003e- Reduce memory I/O: Write cache-friendly code. In the process of two matrices multiply, we will have to\nread data times, and keep the temporary results, then write to memory. If we could put more data into\nCPU's Cache but not read/write memory again and again, the performance should\nimprove a lot. \n\u003e\n\u003e- Cache inverse matrices: It'll save thousands ns, not much, but it's still meaningful\nfor small data.\n\u003e\n\u003e- ...\n\n[Here](http://www.templex.xyz/blog/101/reedsolomon.html) (in Chinese) is an article about\nhow to write a fast Erasure Codes engine. \n(Written by me years ago, need update, but the main ideas still work)\n\n## Performance\n\nPerformance depends mainly on:\n\n\u003e- CPU instruction extension.\n\u003e\n\u003e- Number of data/parity row vectors.\n\n**Platform:** \n\n*AWS c5d.xlarge (Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz)*\n\n**All test run on a single Core.**\n\n### Encode:\n\n`I/O = (data + parity) * vector_size / cost`\n\n*Base means no SIMD.*\n\n| Data  | Parity  | Vector size | AVX512 I/O (MB/S) |  AVX2 I/O (MB/S) |Base I/O (MB/S) |\n|-------|---------|-------------|-------------|---------------|---------------|\n|10|2|4KB|       29683.69   |    21371.43      |   910.45       |\n|10|2|1MB|     17664.67    |    \t15505.58      |   917.26       |\n|10|2|8MB|      10363.05    |      9323.60     |    914.62      |\n|10|4|4KB|      17708.62    |      12705.35    |    531.82      |\n|10|4|1MB|     11970.42    |     9804.57     |  536.31        |\n|10|4|8MB|      7957.9    |      6941.69     |    534.82      |\n|12|4|4KB|      16902.12    |       12065.14   |  511.95        |\n|12|4|1MB|      11478.86   |   9392.33       |   514.24       |\n|12|4|8MB|       7949.81   |   6760.49        |    513.06      |\n\n### Reconstruct:\n\n`I/O = (data + reconstruct_data_num) * vector_size / cost`\n\n| Data  | Parity  | Vector size | Reconstruct Data Num |  AVX512 I/O (MB/S) |\n|-------|---------|-------------|-------------|---------------|\n|10|4|4KB| 1         |      29830.36    |\n|10|4|4KB| 2        |     21649.61     |  \n|10|4|4KB| 3         |     17088.41      | \n|10|4|4KB| 4         |    14567.26       | \n\n### Update:\n\n`I/O = (2 + parity_num + parity_num) * vector_size / cost`\n\n| Data  | Parity  | Vector size | AVX512 I/O (MB/S) |\n|-------|---------|-------------|-------------|\n|10|4|4KB|      36444.13    |\n\n### Replace:\n\n`I/O = (parity_num + parity_num + replace_data_num) * vector_size / cost`\n\n| Data  | Parity  | Vector size | Replace Data Num |  AVX512 I/O (MB/S) |\n|-------|---------|-------------|-------------|---------------|\n|10|4|4KB| 1         |  78464.33        |  \n|10|4|4KB| 2        |     50068.71     |   \n|10|4|4KB| 3         |   38808.11        |  \n|10|4|4KB| 4         |    32457.60       |     \n|10|4|4KB| 5         |  28679.46         |  \n|10|4|4KB| 6         |    26151.85       |   \n\n**PS:**\n\n*And we must know the benchmark test is quite different with encoding/decoding in practice.\nBecause in benchmark test loops, the CPU Cache may help a lot.*\n\n## Links \u0026 Thanks\n\u003e- [Klauspost ReedSolomon](https://github.com/klauspost/reedsolomon): It's the\nmost commonly used Erasure Codes library in Go. Impressive performance, friendly API, \nand it can support multi platforms(with fast Galois Field Arithmetic). Inspired me a lot.\n\u003e\n\u003e- [Intel ISA-L](https://github.com/01org/isa-l): The ideas of Cauchy matrix and saving memory\nI/O are from it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenacid%2Freedsolomon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenacid%2Freedsolomon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenacid%2Freedsolomon/lists"}