{"id":15067744,"url":"https://github.com/somethingnew2-0/go-erasure","last_synced_at":"2025-07-08T10:37:56.575Z","repository":{"id":22590407,"uuid":"25932260","full_name":"somethingnew2-0/go-erasure","owner":"somethingnew2-0","description":"Erasure coding (Reed–Solomon coding) in Go","archived":false,"fork":false,"pushed_at":"2019-10-18T02:14:00.000Z","size":422,"stargazers_count":48,"open_issues_count":0,"forks_count":10,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-24T14:02:48.351Z","etag":null,"topics":["erasure-coding","golang","reed-solomon","reedsolomon","trie"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/somethingnew2-0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-10-29T17:15:46.000Z","updated_at":"2023-11-18T07:29:08.000Z","dependencies_parsed_at":"2022-08-21T08:30:10.409Z","dependency_job_id":null,"html_url":"https://github.com/somethingnew2-0/go-erasure","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/somethingnew2-0%2Fgo-erasure","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/somethingnew2-0%2Fgo-erasure/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/somethingnew2-0%2Fgo-erasure/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/somethingnew2-0%2Fgo-erasure/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/somethingnew2-0","download_url":"https://codeload.github.com/somethingnew2-0/go-erasure/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248251223,"owners_count":21072687,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["erasure-coding","golang","reed-solomon","reedsolomon","trie"],"created_at":"2024-09-25T01:26:59.661Z","updated_at":"2025-04-10T16:12:44.468Z","avatar_url":"https://github.com/somethingnew2-0.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"go-erasure [![Build Status](https://travis-ci.org/somethingnew2-0/go-erasure.svg?branch=master)](https://travis-ci.org/somethingnew2-0/go-erasure) [![Coverage Status](https://coveralls.io/repos/somethingnew2-0/go-erasure/badge.svg?branch=master)](https://coveralls.io/r/somethingnew2-0/go-erasure?branch=master)\n========\n\n*Disclaimer: I recommend the [klauspost/reedsolomon](https://github.com/klauspost/reedsolomon) erasure coding library over this one as it is more performant and has better support for multiple architectures.*\n\nGo bindings for erasure coding (Reed-Solomon coding).\n\nErasure coding is similar to RAID based parity encoding, but is more generalized and powerful.  When defining an erasure code, you specify a `k` and `m` variable. `m` is the number of shards you wish to encode and `k` is the number shards it takes to recreate your original data.  Hence `k` must be less than `m` and usually not equal (as that would be a pointless encoding). The real magic with erasure coding is that fact that ANY `k` of the `m` shards can recreate the original data.  For example, a erasure coding scheme of `k=8` and `m=12` means any four of the encoded shards can be lost while the original data can still be constructed from the valid remaining eight shards.\n\nThis library is aimed at simplicity and performance.  It only has three methods including a constructor which are all thread-safe! Internally it uses [Cgo](http://blog.golang.org/c-go-cgo) to utilize a complex C library.  For a more in-depth look into this library be sure to check out the [Intel® Storage Acceleration Library](https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version) and especially their corresponding [video](http://www.intel.com/content/www/us/en/storage/erasure-code-isa-l-solution-video.html).  One feature it does add is an optimization for decoding.  Since there are `m choose k` possible inverse matrices for decoding, this library caches them (via lazy-loading) so as reduce the amount of time decoding.  It does so by utilizing a [trie](http://en.wikipedia.org/wiki/Trie) where the sorted error list of shards is the key to the trie and the corresponding decode matrix is the value.\n\nI hope you find it useful and pull requests are welcome!\n\n## Usage\nSee the [GoDoc](https://godoc.org/github.com/somethingnew2-0/go-erasure) for an API reference\n\n### Encode and decode random data\n\n```go\npackage main\n\nimport (\n  \"bytes\"\n  \"log\"\n  \"math/rand\"\n  \n  \"github.com/somethingnew2-0/go-erasure\"\n)\n\nfunc corrupt(source, errList []byte, shardLength int) []byte {\n\tcorrupted := make([]byte, len(source))\n\tcopy(corrupted, source)\n\tfor _, err := range errList {\n\t\tfor i := 0; i \u003c shardLength; i++ {\n\t\t\tcorrupted[int(err)*shardLength+i] = 0x00\n\t\t}\n\t}\n\treturn corrupted\n}\n\nfunc main() {\n\tm := 12\n\tk := 8\n\tshardLength := 16 // Length of a shard\n\tsize := k * shardLength // Length of the data blob to encode\n\n\tcode := erasure.NewCode(m, k, size)\n\n\tsource := make([]byte, size)\n\tfor i := range source {\n\t\tsource[i] = byte(rand.Int63() \u0026 0xff) //0x62\n\t}\n\n\tencoded := code.Encode(source)\n\n\terrList := []byte{0, 2, 3, 4}\n\n\tcorrupted := corrupt(append(source, encoded...), errList, shardLength)\n\n\trecovered := code.Decode(corrupted, errList, true)\n\n\tif !bytes.Equal(source, recovered) {\n\t\tlog.Fatal(\"Source was not sucessfully recovered with 4 errors\")\n\t}\n}\n```\n\n\n## Development\n\nTo start run `source dev.sh` or more simply `. dev.sh` to setup the git hooks and GOPATH for this project.\n\nRun `go test` or `go test -bench .` to test the unit tests and benchmark tests.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsomethingnew2-0%2Fgo-erasure","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsomethingnew2-0%2Fgo-erasure","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsomethingnew2-0%2Fgo-erasure/lists"}