{"id":15352610,"url":"https://github.com/thomasjungblut/go-sstables","last_synced_at":"2025-05-16T13:06:48.402Z","repository":{"id":42862740,"uuid":"125091363","full_name":"thomasjungblut/go-sstables","owner":"thomasjungblut","description":"Go library for protobuf compatible sstables, a skiplist, a recordio format and other database building blocks like a write-ahead log. Ships now with an embedded key-value store.","archived":false,"fork":false,"pushed_at":"2025-05-05T10:14:18.000Z","size":3912,"stargazers_count":323,"open_issues_count":6,"forks_count":12,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-05T11:28:43.245Z","etag":null,"topics":["compaction","database","embedded-database","hacktoberfest","kaitai","kaitai-struct","key-value-store","memstore","merge","protobuf","recordio","skiplist","sst","sstable","sstables","write-ahead-log"],"latest_commit_sha":null,"homepage":"https://blog.thomasjungblut.com/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thomasjungblut.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-03-13T17:43:33.000Z","updated_at":"2025-05-05T10:14:21.000Z","dependencies_parsed_at":"2023-01-23T19:15:18.032Z","dependency_job_id":"54fb3d82-af01-44cf-84ef-f4a464b92d58","html_url":"https://github.com/thomasjungblut/go-sstables","commit_stats":{"total_commits":102,"total_committers":3,"mean_commits":34.0,"dds":0.07843137254901966,"last_synced_commit":"d5fa566e5d9d2fe98ff25dcbbfedc1fe6071f9f0"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomasjungblut%2Fgo-sstables","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomasjungblut%2Fgo-sstables/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomasjungblut%2Fgo-sstables/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomasjungblut%2Fgo-sstables/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thomasjungblut","download_url":"https://codeload.github.com/thomasjungblut/go-sstables/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254535829,"owners_count":22087399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compaction","database","embedded-database","hacktoberfest","kaitai","kaitai-struct","key-value-store","memstore","merge","protobuf","recordio","skiplist","sst","sstable","sstables","write-ahead-log"],"created_at":"2024-10-01T12:09:54.085Z","updated_at":"2025-05-16T13:06:48.390Z","avatar_url":"https://github.com/thomasjungblut.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![unit-test](https://github.com/thomasjungblut/go-sstables/actions/workflows/unit-test.yml/badge.svg)](https://github.com/thomasjungblut/go-sstables/actions/workflows/unit-test.yml)\n[![CodeQL](https://github.com/thomasjungblut/go-sstables/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/thomasjungblut/go-sstables/actions/workflows/codeql-analysis.yml)\n[![Go Reference](https://pkg.go.dev/badge/github.com/thomasjungblut/go-sstables.svg)](https://pkg.go.dev/github.com/thomasjungblut/go-sstables)\n\n## go-sstables\n\n`go-sstables` is a Go library that contains NoSQL database building blocks like a sequential record format (recordio), a\nsorted string table (sstable), a write-ahead-log (WAL), and a memory store (memstore) that stores key/value pairs in\nmemory using a skip list.\n\nYou can frequently find those in embedded key-value stores or databases as well, notable examples\nare [RocksDB](https://github.com/facebook/rocksdb) or [LevelDB](https://github.com/google/leveldb).\n\nThere is an example embedded key-value database in this library, you can find it in the simpledb folder - please don't\nuse it for any production workload.\n\nWhile plain `[]byte` are at the core of this library, there are wrappers and bindings for protobuf to enable more\nconvenient serialization.\n\n## Installation\n\nThis is a library as it does not contain any installable binary, which means you can just directly add it to your\ndependency via `go get`:\n\n\u003e go get -d github.com/thomasjungblut/go-sstables\n\n## Documentation\n\n[RocksDB has a great overview](https://github.com/facebook/rocksdb/wiki/RocksDB-Overview#3-high-level-architecture) of\nhow the components usually play together to get an idea:\n\n![rocksdb architecture overview](https://user-images.githubusercontent.com/62277872/119747261-310fb300-be47-11eb-92c3-c11719fa8a0c.png)\n\nYou will find basically all of those mentioned pieces above and all of them stitched together as SimpleDB. The\ndocumentation is now separated by package for easier browsing, each of those READMEs contain examples - there is\nalso [/examples](_examples) when you prefer browsing the code directly.\n\n* [RecordIO](recordio/README.md)\n    * [Benchmark](benchmark/README.md#recordio)\n* [sstables](sstables/README.md)\n    * [Benchmark](benchmark/README.md#sstable)\n* [Memstore](memstore/README.md)\n* [SkipList](skiplist/README.md)\n* [WriteAheadLog](wal/README.md)\n* [SimpleDB](simpledb/README.md)\n    * [Benchmark](benchmark/README.md#simpledb)\n\nYou can also find all interface and\nmethod [documentation on `pkg.go.dev`](https://pkg.go.dev/github.com/thomasjungblut/go-sstables/sstables#section-documentation)\n.\n\n## Kaitai Support\n\nAs you might want to read the data and files in other languages, I've added support for [Kaitai](https://kaitai.io/).  \nKaitai is a declarative schema file to define a binary format. From that `ksy` file you can generate code for a lot of\nother languages and read the data.\n\nCurrently, there is support for:\n\n* [RecordIO (v2)](kaitai/recordio_v2.ksy)\n* [RecordIO (v3)](kaitai/recordio_v3.ksy)\n* [RecordIO (v4)](kaitai/recordio_v4.ksy)\n\nYou can find more information on how to generate Kaitai readers in [kaitai/README.md](kaitai/README.md).\n\n\n## Development on go-sstables\n\n### Thank you\n\nI want to deeply thank Sébastien Heitzmann (@sebheitzmann) for his continued support, reviews and usage of this library.\n\n\n### Generating protobufs\n\nThis needs some pre-requisites installed, namely\nthe [protobuf compiler](https://github.com/protocolbuffers/protobuf/releases) and the go generator plugin. The latter\ncan be installed as a go package:\n\n```\ngo install google.golang.org/protobuf/cmd/protoc-gen-go\n```\n\nFull installation details can be found in\nthe [protobuf dev documentation](https://developers.google.com/protocol-buffers/docs/gotutorial#compiling-your-protocol-buffers)\n.\n\nOnce installed, one can generate the protobuf structs using:\n\n```\nmake compile-proto\n```\n\n### Releasing the Go Module\n\n[General Guidance](https://github.com/golang/go/wiki/Modules#releasing-modules-all-versions)\n\nIn short, run these commands:\n\n```\nmake unit-test\nmake release\ngit push --tags \n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthomasjungblut%2Fgo-sstables","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthomasjungblut%2Fgo-sstables","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthomasjungblut%2Fgo-sstables/lists"}