{"id":13628934,"url":"https://github.com/vmxdev/tkvdb","last_synced_at":"2026-01-30T23:03:34.110Z","repository":{"id":152103604,"uuid":"70838515","full_name":"vmxdev/tkvdb","owner":"vmxdev","description":"Trie key-value database","archived":false,"fork":false,"pushed_at":"2022-02-18T20:25:36.000Z","size":325,"stargazers_count":337,"open_issues_count":3,"forks_count":25,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-04-17T17:39:07.093Z","etag":null,"topics":["c-library","key-value-store","radix-tree"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vmxdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-10-13T18:59:37.000Z","updated_at":"2025-03-29T12:32:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"873bf725-e3df-4f0b-a8e4-7f2cc65b98c0","html_url":"https://github.com/vmxdev/tkvdb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vmxdev/tkvdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmxdev%2Ftkvdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmxdev%2Ftkvdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmxdev%2Ftkvdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmxdev%2Ftkvdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vmxdev","download_url":"https://codeload.github.com/vmxdev/tkvdb/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmxdev%2Ftkvdb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28922232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T22:32:35.345Z","status":"ssl_error","status_checked_at":"2026-01-30T22:32:31.927Z","response_time":66,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c-library","key-value-store","radix-tree"],"created_at":"2024-08-01T22:00:59.782Z","updated_at":"2026-01-30T23:03:34.092Z","avatar_url":"https://github.com/vmxdev.png","language":"C","funding_links":[],"categories":["C"],"sub_categories":[],"readme":"# tkvdb\nTrie (radix trie in fact) key-value database\n\n`tkvdb` is an embedded database library for key-value data. It is similar to Berkeley DB, LevelDB or SQLite4 LSM.\n\nKeys are always sorted in memcmp() order.\n\n## Supported operations\n\n  * Add a new key/value pair to the database.\n  * Delete an existing key from the database.\n  * Querying the database for a specific key.\n  * Iterating through a range of database keys (either forwards or backwards).\n\n## Portability\n\n`tkvdb` is written in ANSI C, without using platform or OS-specific functions.\n\nIt uses traditional `open/seek/read/write/close` API for operations with data files, memory allocation (`malloc/realloc/free`) and some string functions (`memset/memcpy`) for dealing with in-memory transactions.\n\nThere is no limitations for 32-bit CPU's, except for size of memory buffers.\n\n`tkvdb` was tested on Linux(x32/x64 CPU's and 32 bit ARM) and under Wine using mingw (hopefully it will work under Windows).\n\n## Python binding\n\nThanks to [@vladimir-g](https://github.com/vladimir-g), there is a python binding for `tkvdb`, see https://github.com/vladimir-g/python-tkvdb\n\n## Basic usage\n\nAPI is quite a simple, but requires some explanation.\n\ntkvdb database file (in simplified append-only form) looks like:\n\n![tkvdb database layout](docs/nonvac_db.png?raw=true \"tkvdb database layout\")\n\nA database file is a set of blocks - \"Transactions\" (probably not the best name for it).\nIn each block, there is a footer with a pointer to a current root node, signature and some additional DB file information.\nEach transaction is a small subtree (radix tree) which contains nodes of the database that was changed in this transaction.\nTree node may contain pointers (offsets in the file) to other, unchanged nodes from previous transactions. These pointers remain unchanged in a new subtree.\n\nSo, to modify the database you need to open it, create a transaction, make changes and \"commit\" (or \"rollback\").\nHowever, you may create transaction without the underlying database file. In this case, `tkvdb` will act as the RAM-only database.\nCommits and rollbacks will just drop all the data and reset transaction to initial state.\n\nTransactions in RAM-only mode uses less memory compared to transactions with underlying DB file (with the same key-values in transaction), since there is no need to hold file offsets for each node.\n\nHere is a simple example:\n\n```\ntkvdb_datum key, value;\n\ndb = tkvdb_open(\"db.tkvdb\", NULL);           /* optional, only if you need to keep data on disk */\n\ntransaction = tkvdb_tr_create(db, NULL);     /* pass NULL instead of db for RAM-only db */\n\ntransaction-\u003ebegin(transaction);             /* start */\ntransaction-\u003eput(transaction, \u0026key, \u0026val);   /* add key-value pair or overwrite existing */\ntransaction-\u003ecommit(transaction);            /* commit */\n\n/* you may reuse transaction later */\ntransaction-\u003ebegin(transaction);             /* start new transaction */\ntransaction-\u003eget(transaction, \u0026key, \u0026val);   /* get key-value pair */\ntransaction-\u003edel(transaction, \u0026key, 1);      /* delete all keys starting with key.data */\ntransaction-\u003erollback(transaction);          /* dismiss */\n\ntransaction-\u003efree(transaction);\n\ntkvdb_close(db);                             /* close on-disk database */\n```\n\n## Searching in database and cursors\n\nUse `transaction-\u003eget()` if you need to get a value by key.\nOn success, it returns `TKVDB_OK` and pointer to data in memory and length.\nYou can modify the value \"in place\" if a length is not changed.\n\nIf you need to iterate through the database (or through a part of the database) you may use cursors.\n\n```\nTKVDB_RES rc;\n\ntkvdb_cursor *cursor = tkvdb_cursor_create(transaction);\n\nrc = cursor-\u003efirst(cursor);                  /* position cursor to the first key-value pair of database */\nwhile (rc == TKVDB_OK) {\n\tkey = cursor-\u003ekey(cursor);           /* get pointer to key */\n\tkeysize = cursor-\u003ekeysize(cursor);   /* and size of a key */\n\tval = cursor-\u003eval(cursor);           /* pointer to value */\n\tvalsize = cursor-\u003evalsize(cursor);   /* and size of value */\n\n\trc = cursor-\u003enext(cursor);           /* jump to next key-value pair */\n}\n\ncursor-\u003efree(cursor);\n```\n\n`while` loop can be written in alternative way\n\n```\nwhile (rc == TKVDB_OK) {\n\ttkvdb_datum key = cursor-\u003ekey_datum(cursor);\n\ttkvdb_datum value = cursor-\u003eval_datum(cursor);\n\n\trc = cursor-\u003enext(cursor);\n}\n```\n\n\nTo iterate in reverse order use `cursor-\u003elast()` and `cursor-\u003eprev()`.\n\nIf you want to search a key-value pair in database by prefix use `cursor-\u003eseek(cursor, \u0026key, TKVDB_SEEK)`\nwhere `TKVDB_SEEK` can be:\n  * `TKVDB_SEEK_EQ` : search for the exact key match\n  * `TKVDB_SEEK_LE` : search for less (in terms of memcmp()) or equal key\n  * `TKVDB_SEEK_GE` : search for greater (in terms of memcmp()) or equal key\n\nAfter seeking to key-value pair you can use `cursor-\u003enext()` or `cursor-\u003eprev()`\n\n## Database and transaction parameters\n\nYou can tune some database or transaction parameters. Transaction parameters are inherited from database, but can be overridden.\nHere is an example:\n\n```\ntkvdb_params *params;\n\nparams = tkvdb_params_create();\ntkvdb_param_set(params, TKVDB_PARAM_TR_DYNALLOC, 0);      /* don't use dynamic nodes allocation */\ntkvdb_param_set(params, TKVDB_PARAM_TR_LIMIT, 1024*1024); /* memory block of 1M will be used for transaction */\n\ndb = tkvdb_open(\"db.tkvdb\", params);\ntransaction1 = tkvdb_tr_create(db, NULL);                  /* transactions of parent db will use theese parameters */\n\n/* and you can override parameters for some transactions with different values */\ntkvdb_param_set(params, TKVDB_PARAM_TR_LIMIT, 1024*1024*10);\ntransaction2 = tkvdb_tr_create(db, params);\n\n/* or use with RAM-only transaction */\ntransaction3 = tkvdb_tr_create(NULL, params);\n\ntkvdb_params_free(params);\n\n```\n\nTransaction parameter can be:\n  * `TKVDB_PARAM_TR_DYNALLOC` - if != `0` then `tkvdb` will allocate memory for nodes dynamically using `malloc()`. Else `tkvdb` will use simple builtin allocator (which can be faster). Default `1`\n  * `TKVDB_PARAM_TR_LIMIT` - memory limit for transaction. In case of overlimit transaction functions will return `TKVDB_ENOMEM`. When used with `TKVDB_PARAM_TR_DYNALLOC` == `0` memory will be allocated in `tkvdb_tr_create()` and this buffer will be used for transaction. Default `SIZE_MAX` (no limit)\n  * `TKVDB_PARAM_ALIGNVAL` - align values in memory. Must be power of two. `0` or `1` means value will not be aligned\n  * `TKVDB_PARAM_AUTOBEGIN` - start transaction automatically after creation, `commit()` and `rollback()`. `begin()` function ignored. Default `0` (you must call `begin()` before working with transaction)\n  * `TKVDB_PARAM_CURSOR_STACK_DYNALLOC` - allocate stack for cursors dynamically when needed (using `realloc()`). Default 1.\n  * `TKVDB_PARAM_CURSOR_STACK_LIMIT` - memory limit cursor stack (in bytes). No limits by default. Stack parameters applied also for `commit()` and `free()` operations for iteration through nodes.\n  * `TKVDB_PARAM_CURSOR_KEY_DYNALLOC` - allocate memory for cursor keys dynamically when needed (using `realloc()`). Default 1.\n  * `TKVDB_PARAM_CURSOR_KEY_LIMIT` - memory limit for cursor keys (in bytes). No limits by default.\n\n## Multithreading\n\n`tkvdb` does not use any OS-dependent synchronization mechanisms.\nYou must explicitly lock transaction update operations.\n\nHowever, on some CPU's (at least on x32/x64) we can guarantee that `transaction-\u003eput()` will never put the in-memory transaction in inconsistent state in RAM-only mode.\nUpdates of tree are lock-free and atomic.\nYou can use one writer and multiple readers without locks.\nBut be careful with `transaction-\u003erollback()` and `transaction-\u003ecommit()` - there is no such guarantees for theese functions, reading from transaction while resetting it can lead to unpredicatable consequences.\n\n## Bugs and caveats (sort of TODO)\n\n  * There is still no `vacuum` routine for database file. We have initial and bogus implementation, but it's not tested, so the database now is append-only. As a temporary workaround you can use `tkvdb-dump` and `tkvdb-restore`, see [utils](utils).\n  * There is no easy way to get N-th record of database. However, it's possible to implement such seeks using some nodes metadata.\n  * There is no publicly available benchmarks and nice performance charts. You can run `perf_test` from `extra` directory, it will show ops(inserts/updates and lookups) per second for 4 and 16 byte keys with different number of keys in transaction. Test is single-threaded and shows RAM-only operations. Depending on hardware you may get up to tens of millions ops per second (or even more than 100 millions lookups per second for short keys). Probably we will make more accurate, complete and readable performance tests.\n\n## Compiling and running tests\n\nUnit test:\n```sh\n$ cc -g -Wall -pedantic -Wextra -I. extra/tkvdb_test.c tkvdb.c -o tkvdb_test\n$ ./tkvdb_test\n```\n\nSimple performance test:\n```sh\n$ cc -O3 -Wall -pedantic -Wextra -I. extra/perf_test.c tkvdb.c -o perf_test\n$ ./perf_test\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmxdev%2Ftkvdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvmxdev%2Ftkvdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmxdev%2Ftkvdb/lists"}