{"id":15449940,"url":"https://github.com/iand/gonudb","last_synced_at":"2025-04-19T22:58:27.317Z","repository":{"id":56134237,"uuid":"313794984","full_name":"iand/gonudb","owner":"iand","description":"Gonudb is an append-only key/value datastore written in Go.","archived":false,"fork":false,"pushed_at":"2023-12-11T10:35:08.000Z","size":108,"stargazers_count":19,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-13T19:22:55.566Z","etag":null,"topics":["database","go","key-value"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iand.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-18T01:58:31.000Z","updated_at":"2024-12-25T08:35:07.000Z","dependencies_parsed_at":"2023-10-04T15:15:10.342Z","dependency_job_id":"e9aa159b-b68a-4fda-96f9-5210512ddde9","html_url":"https://github.com/iand/gonudb","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iand%2Fgonudb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iand%2Fgonudb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iand%2Fgonudb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iand%2Fgonudb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iand","download_url":"https://codeload.github.com/iand/gonudb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249826664,"owners_count":21330673,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","go","key-value"],"created_at":"2024-10-01T21:02:41.890Z","updated_at":"2025-04-19T22:58:27.284Z","avatar_url":"https://github.com/iand.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gonudb \n\nGonudb is an append-only key/value datastore written in Go.\n\n[![Check Status](https://github.com/iand/gonudb/actions/workflows/check.yml/badge.svg)](https://github.com/iand/gonudb/actions/workflows/check.yml)\n[![Test Status](https://github.com/iand/gonudb/actions/workflows/test.yml/badge.svg)](https://github.com/iand/gonudb/actions/workflows/test.yml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/iand/gonudb)](https://goreportcard.com/report/github.com/iand/gonudb)\n[![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go\u0026logoColor=white)](https://pkg.go.dev/github.com/iand/gonudb)\n\n## Overview\n\nGonudb is a port of [NuDB](https://github.com/CPPAlliance/NuDB), a C++ key/value store.\n\nA Gonudb datastore comprises a data file holding keys and values stored sequentially and an\naccompanying key file which forms an on-disk hash table indexing the values stored in the data\nfile.\n\nDuring commits a log file is created to store bookkeeping information that may be used to repair the\ndatastore in the event of a failure.\n\nThe data file and key file are independent and a new key file may be rebuilt from the data file if\nnecessary, potentially with an alternate hashing scheme.\n\n\n## Installation\n\nExecute `go get github.com/iand/gonudb` within a Go module directory to add it to your module.\n\n## Usage\n\nGonudb is primarily a library. Import package `github.com/iand/gonudb` to use. A sample application\nthat demonstrates some simple inserts and fetches is provided in `cmd/gonudbsample`.\n\nAn admin tool can be found in `cmd/gonudbadmin` which provides some commands for inspecting and\nvalidating the files that comprise a store.\n\nInstall by executing `go install github.com/iand/gonudb/cmd/gonudbadmin` from the root of the \nrepository.\n\n - `gonudbadmin info` can be used to view charactistic information about any of the three files used by gonudb (data, key and log files).\n - `gonudbadmin verify` verifies the consistency of data and key files and shows some statistics on the data they hold.\n\n\n## Design\n\nGonudb shares the design ideals that motivated NuDB (but see Status below):\n\n 1. Writes should not block reads.\n 2. Reads should be limited only by the SSD's IOPS limit.\n 3. A read for a non-present key should require one IOP.\n 4. A read for a present key whose data can be read in a single IOP should only require two IOPs, one to figure out where it is and one to read it in.\n\nKeys and values are stored sequentially in an append only data file. The data file begins with a\nheader that contains characteristic information about the file such as the version of the encoding\nscheme, a datastore identifier and an application identifier. Data records follow immediately on\nfrom the header. Each record comprises the size of the value, followed by the size of the key,\nfollowed by the key, followed by the value data. The data file is considered to be immutable and\nthere are no delete or mutate operations.\n\nInserts are buffered in memory and periodically committed to disk. Clients are throttled based on\nthe rate at which data is flushed to disk. Values are immediately discoverable via their key and\nmay be read from memory or disk.\n\nKeys are hashed and written to buckets stored in the key file. As with the data file, the key file\nbegins with a header containing characteristic information. The key file's version, datastore\nidentifier and application identifier must match those in the data file header. Additionally the\nkey file header contains the hash salt, the block size of each bucket and the target load factor\nwhich determines when a bucket should be split. Buckets are a fixed size and written sequentially\nafter the header which enables them to the be easily located by index.\n\nEach bucket is assigned a range of hash values and entries within a bucket are ordered by hash. When\nthe number of entries in a bucket exceeds the load factor it undergoes a split and its entries are\nrehashed across the pair of buckets using the linear hashing algorithm. When a bucket exceeds its\ncapacity it is spilled to the data file and replaced with an empty bucket containing a pointer to\nthe spill record. A spilled bucket may spill multiple times with the resulting spill records\nforming a linked list in the data file.\n\nIn the best case reading a record from the datastore requires one read from the key file to load the\nrelevant bucket and a read from the data file to access the value. Additional reads from the data\nfile may be required to resolve hash collisions and to load spill records. Read performance is\nindependent of the size of the datastore and the size of buckets in the key file may be tuned to\nthe block size of the underlying physical media so loading a bucket may only take a single IOP.\n\n## Status\n\nVersion 0.1.0 is an alpha quality functional port of the original NuDB suitable for testing with \nexpendable loads. Correctness and safety has been prioritised over performance. Locks are broad in scope\nand treat reads and writes with equal priority. Future work will tune the locking bahaviour to \nbetter meet the goal of writes not blocking reads.\n\nHigh priority tasks include:\n\n * Add recover from partial writes\n * Add rekey admin function.\n * Tune locking strategy\n\nAdditional features under consideration:\n\n * Allow alternate hashing functions to be specified.\n\n## Author\n\nGo port written by:\n\n* [Ian Davis](http://github.com/iand) - \u003chttp://iandavis.com/\u003e\n\n## License\n\nDistributed under the Boost Software License, Version 1.0. (See accompanying file [LICENSE](LICENSE)\n or copy at http://www.boost.org/LICENSE_1_0.txt)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiand%2Fgonudb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiand%2Fgonudb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiand%2Fgonudb/lists"}