{"id":27185064,"url":"https://github.com/threefoldtech/0-db","last_synced_at":"2025-04-09T17:10:22.182Z","repository":{"id":30897768,"uuid":"119703121","full_name":"threefoldtech/0-db","owner":"threefoldtech","description":"Fast write ahead persistent redis protocol key-value store","archived":false,"fork":false,"pushed_at":"2024-05-31T01:17:09.000Z","size":1147,"stargazers_count":39,"open_issues_count":24,"forks_count":10,"subscribers_count":30,"default_branch":"development-v2","last_synced_at":"2024-06-02T14:22:17.115Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/threefoldtech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-31T15:10:54.000Z","updated_at":"2024-05-31T01:16:04.000Z","dependencies_parsed_at":"2024-01-17T02:56:04.783Z","dependency_job_id":"cfff5e20-8d1b-4ee4-b380-a9b0743d0a7b","html_url":"https://github.com/threefoldtech/0-db","commit_stats":null,"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2F0-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2F0-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2F0-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2F0-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/threefoldtech","download_url":"https://codeload.github.com/threefoldtech/0-db/tar.gz/refs/heads/development-v2","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248074976,"owners_count":21043490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-09T17:10:21.475Z","updated_at":"2025-04-09T17:10:22.162Z","avatar_url":"https://github.com/threefoldtech.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 0-db [![codecov](https://codecov.io/gh/threefoldtech/0-db/branch/master/graph/badge.svg)](https://codecov.io/gh/threefoldtech/0-db)  \n0-db (zdb) is a super fast and efficient key-value store which makes data persistant\ninside an always append datafile, with namespaces support and data offloading.\n\nThe database is split in two part:\n- The database engine, which can be used as static or shared library\n- The network daemon using resp (redis protocol) to make a remote database\n\nIndexes are created to speedup restart/reload process, this index is always append too,\nexcept in sequential-mode (see below for more information).\n\nWe use it as backend for many of our blockchain work, 0-db is not a redis replacement and never will.\n\n# GitHub Package helper\n\nIn order to use GitHub Package Docker image, here are some environment variable you can set:\n- `DATADIR`: directory of data files\n- `INDEXDIR`: directory of index files\n- `DATASIZE`: maximum datafiles size\n- `ADMIN`: admin password\n- `PROTECT`: set to 1 to enable `--protect` which requires admin password to write on default namespace\n\n# Quick links\n1. [Build targets](#build-targets)\n2. [Build instructions](#build-instructions)\n3. [Running](#running)\n4. [Always append](#always-append)\n5. [Running modes](#running-modes)\n6. [Implementation](#implementation)\n7. [Supported commands](#supported-commands)\n8. [Namespaces](#namespaces)\n9. [Hook system](#hook-system)\n10. [Data offload](#data-offload)\n11. [Limitation](#limitation)\n12. [Tests](#tests)\n\n# Build targets\nCurrently supported system:\n* Linux (using `epoll`), kernel 3.17+, glibc 2.25+\n* MacOS and FreeBSD (using `kqueue`)\n\nCurrently supported hardware:\n* Any little-endian CPU\n\nCurrently optimized hardware:\n* `x86_64` with `SSE4.2`\n* `ARMv8` with `CRC32` flags (which include `Apple M1`)\n\nThis project doesn't support big-endian system for now.\n\n# Build instructions\nTo build the project (library, server, tools):\n* Type `make` on the root directory\n* The binaries will be placed on `bin/` directory\n\nYou can build each parts separatly by running `make` in each separated directories.\n\n\u003e By default, the code is compiled in debug mode, in order to use it in production, please use `make release`\n\n# Running\n\n0-db is made to be run in network server mode (using zdbd), documentation here is about the server.\nMore documentation will comes about the library itself. The library lacks of documentation.\n\nWithout argument, datafiles and indexfiles will be stored on the current working directory, inside\n`zdb-data` and `zdb-index` directories.\n\nIt's recommended to store index on SSD, data can be stored on HDD, fragmentation will be avoided as much\nas possible (only on Linux). Please avoid CoW (Copy-on-Write) for both index and data.\n\n## Default port\n\n0-db listens by default on port `9900` but this can be overidden on the commandline using `--port` option.\n\n# Always append\nData file (files which contains everything, included payload) are **in any cases** always append:\nany change will result in something appened to files. Data files are immuables. If any suppression is\nmade, a new entry is added, with a special flag. This have multiple advantages:\n- Very efficient in HDD (no seek when writing batch of data)\n- More efficient for SSD, longer life, since overwrite doesn't occures\n- Easy for backup or transfert: incremental copy work out-of-box\n- Easy for manipulation: data is flat\n- History support out-of-box, since all data are kept\n\nOf course, when we have advantages, some cons comes with them:\n- Any overwrite won't clean previous data\n- Deleting data won't actually delete anything in realtime\n- You need some maintenance to keep your database not exploding\n\nHopefuly, theses cons have their solution:\n- Since data are always append, you can at any time start another process reading that database\nand rewrite data somewhere else, with optimization (removed non-needed files). This is what we call\n`compaction`, and some tools are here to do so.\n- As soon as you have your new files compacted, you can hot-reload the database and profit, without\nloosing your clients (small freeze-time will occures, when reloading the index).\n\nData files are never reach directly, you need to always hit the index first.\n\nIndex files are always append, except when deleting or overwriting a key. Impact are really small\nonly a flag is edited, and new entries are always append anyway, but the database supports to walk\nover the keys, any update needs to invalidate the previous entry, in order to keep the chain in a\ngood health. The index is there mostly to have flexibility.\n\nOtherwise, index works like data files, with more or less the same data (except payload) and\nhave the advantage to be small and load fast (can be fully populated in memory for processing).\n\n# Running modes\nOn runtime, you can choose between multiple mode:\n* `user`: user-key mode\n* `seq`: sequential mode\n\n**Warning**: in any case, please ensure data and index directories used by 0-db are empty, or\ncontains only valid database namespaces directories.\n\nIf you run `zdbd` without `--mode` argument, server will runs in `mixed mode` and allows some\n`user` and `sequential` namespace on the same instance. Please see `NSSET` command to get more\ninformation on how to choose runtime mode.\n\n## User Key\nThis is a default mode, a simple key-value store.\nUser can `SET` their own keys, like any key-value store.\nAll the keys are kept in memory.\n\nEven in this mode, the key itself is returned by the `SET` command.\n\n## Sequential\nIn this mode, the key is a sequential key autoincremented.\n\nYou need to provide a null-length key, to generate a new key.\nIf you provide a key, this key should exists (a valid generated key), and the `SET` will update that key.\n\nProviding any other key will fails.\n\nThe id is a little-endian integer key. Keys are not kept in memory, based on the key-id, location\non disk can be known. Running a `zdbd` in sequential-mode-only have a **really** low memory footprint.\n\n# Implementation\nThis project doesn't rely on any dependencies, it's from scratch.\n\nA rudimental and very simplified RESP protocol is supported, allowing only some commands. See below.\n\nEach index files contains a 27 bytes headers containing a magic 4 bytes identifier,\na version, creation and last opened date and it's own file-sequential-id. In addition it contains\nthe mode used when it was created (to avoid mixing mode on different run).\n\nFor each entries on the index, on disk, an entry of 30 bytes + the id will be written.\nIn memory, 42 bytes (34 bytes + pointer for linked list) plus the key itself (limited to 256 bytes)\nwill be consumed.\n\nThe data (value) files contains a 26 bytes headers, mostly the same as the index one\nand each entries consumes 18 bytes (1 byte for key length, 4 bytes for payload length, 4 bytes crc,\n4 bytes for previous offset, 1 byte for flags, 4 byte for timestamp) plus the key and payload.\n\n\u003e We keep track of the key on the data file in order to be able to rebuild an index based only on datafile if needed.\n\nEach time a key is inserted, an entry is added to the data file, then on the index.\nWhenever the key already exists, it's appended to disk and entry in memory is replaced by the new one.\n\nEach time the server starts, it loads (basicly replay) the index in memory. The index is kept in memory \n**all the time** and only this in-memory index is reached to fetch a key, index files are\nnever read again except during startup, reload or slow query (slow queries mean, doing some SCAN/RSCAN/HISTORY requests).\n\nIn sequential-mode, key is the location on the index, no memory usage is needed, but lot of disk access are needed.\n\nWhen a key-delete is requested, the key is kept in memory and is flagged as deleted. A new entry is added\nto the index file, with the according flags. When the server restart, the latest state of the entry is used.\nIn direct mode, the flag is overwritten in place on the index.\n\n## Index\nThe current index in memory is a really simple implementation (to be improved).\n\nIt uses a rudimental kind-of hashtable. A list of branchs (2^24) is pre-allocated.\nBased on the crc32 of the key, we keep 24 bits and uses this as index in the branches.\n\nBranches are allocated only when used. Using 2^24 bits will creates 16 million index entries\n(128 MB on 64 bits system).\nEach branch (when allocated) points to a linked-list of keys (collisions).\n\nWhen the branch is found based on the key, the list is read sequentialy.\n\n## Read-only\nYou can run 0-db using a read-only filesystem (both for keys or data), which will prevent\nany write and let the 0-db serving existing data. This can, in the meantime, allows 0-db\nto works on disks which contains failure and would be remounted in read-only by the kernel.\n\nThis mode is not possible if you don't have any data/index already available.\n\n# Supported commands\n- `PING`\n- `SET \u003ckey\u003e \u003cvalue\u003e [timestamp]`\n- `GET \u003ckey\u003e`\n- `MGET \u003ckey\u003e [key ...]`\n- `DEL \u003ckey\u003e`\n- `STOP` (used only for debugging, to check memory leaks)\n- `EXISTS \u003ckey\u003e`\n- `CHECK \u003ckey\u003e`\n- `KEYCUR \u003ckey\u003e`\n- `INFO`\n- `NSNEW \u003cnamespace\u003e`\n- `NSDEL \u003cnamespace\u003e`\n- `NSINFO \u003cnamespace\u003e`\n- `NSLIST`\n- `NSSET \u003cnamespace\u003e \u003cproperty\u003e \u003cvalue\u003e`\n- `NSJUMP`\n- `SELECT \u003cnamespace\u003e [SECURE password]`\n- `DBSIZE`\n- `TIME`\n- `AUTH [password]`\n- `AUTH SECURE [password]`\n- `SCAN [cursor]`\n- `RSCAN [cursor]`\n- `WAIT command | * [timeout-ms]`\n- `HISTORY \u003ckey\u003e [binary-data]`\n- `FLUSH`\n- `HOOKS`\n- `INDEX DIRTY [RESET]`\n- `DATA RAW \u003cfileid\u003e \u003coffset\u003e`\n- `LENGTH \u003ckey\u003e`\n- `KEYTIME \u003ckey\u003e`\n\n`SET`, `GET` and `DEL`, `SCAN` and `RSCAN` supports binary keys.\n\nArguments `\u003cflags\u003e` are mandatory, arguments `[flags]` are optionals.\n\n\u003e Compared to real redis protocol, during a `SET`, the key is returned as response.\n\n## SET\nThis is the basic `SET key value` command, key can be binary.\n\nThis command returns the key if SET was done properly or `(nil)` if you\ntry to update a key without modification (avoid inserting already existing data).\n\n**Note:** admin user can specify an extra argument, timestamp, which will set the timestamp of the key\nto the specified timestamp and not the current timestamp. This is needed when doing replication.\n\n## GET (with MGET)\nRetreive data, key can be binary. Returns (nil) when key doesn't exists (not found, deleted).\n\n`GET` can only handle one key. There is `MGET` which supports multiple keys. Response is an array.\n\nThere is a hard-limit of 1023 keys at a time.\n\n## EXISTS\nReturns 1 or 0 if the key exists\n\n## CHECK\nCheck internally if the data is corrupted or not. A CRC check is done internally.\nReturns 1 if integrity is validated, 0 otherwise.\n\n## KEYCUR\nReturns a cursor from a key name. This cursor **should** be valid for life-time.\nYou can provide this cursor to SCAN family command in order to start walking from a specific\nkey. Even if that key was updated or deleted, the cursor contains enough data to know from\nwhere to start looking and you won't miss any new stuff.\n\nThis cursor is a binary key.\n\n## SCAN\nWalk forward over a dataset (namespace).\n\n- If `SCAN` is called without argument, it starts from first key (first in time) available in the dataset.\n- If `SCAN` is called with an argument, it starts from provided key cursor.\n\nIf the dataset is empty, or you reach the end of the chain, `-No more data` is returned.\n\nIf you provide an invalid cursor as argument, `-Invalid key format` is returned.\n\nOtherwise, an array (a little bit like original redis `SCAN`) is returned.\nThe first item of the array is the next id (cursor) you need to set to SCAN in order to continue walking.\n\nThe second element of the array is another array which contains one or more entries (keys). Each entries\ncontains 3 fields: the key, the size of the payload and the creation timestamp.\n\n**Note:** the amount of keys returned is not predictable, it returns as much as possible keys\nin a certain limited amount of time, to not block others clients.\n\nExample:\n```\n\u003e SCAN\n1) \"\\x87\\x00\\x00\\x00\\x10\\x00\\x00\\xcd4\\x00\\x00\\x87E{\\x88  # next key id to send as SCAN argument to go ahead\n2) 1) 1) \"\\x01\\x02\\x03\"\n      2) (integer) 16                 # size of payload in byte\n      3) (integer) 1535361488         # unix timestamp of creation time\n   2) 1) \"\\xa4\\x87\\xd4}\\xbe\\x84\\x1a\\xba\"\n      2) (integer) 6                  # size of payload in byte\n      3) (integer) 1535361485         # unix timestamp of creation time\n```\n\nBy calling `SCAN` with each time the key responded on the previous call, you can walk forward a complete\ndataset.\n\nThere is a special alias `SCANX` command which does exacly the same, but with another name.\nSome redis client library (like python) expect integer response and not binary response. Using `SCANX` avoid\nthis issue.\n\nIn order to start scanning from a specific key, you need to get a cursor from that key first,\nsee `KEYCUR` command\n\n## RSCAN\nSame as scan, but backward (last-to-first key)\n\n## NSNEW\nCreate a new namespace. Only admin can do this.\n\nBy default, a namespace is not password protected, is public and not size limited.\n\n## NSDEL\nDelete a namespace. Only admin can do this.\n\nWarning:\n- You can't remove the namespace you're currently using.\n- Any other clients using this namespace will be moved to a special state, awaiting to be disconnected.\n\n## NSINFO\nReturns basic informations about a namespace\n\n```\n# namespace\nname: default          # namespace name\nentries: 0             # amount of entries\npublic: yes            # public writable (yes/no)\npassword: no           # password protected (yes/no)\ndata_size_bytes: 0     # total data payload in bytes\ndata_size_mb: 0.00     # total data payload in MB\ndata_limits_bytes: 0   # namespace size limit (0 for unlimited)\nindex_size_bytes: 0    # index size in bytes (thanks captain obvious)\nindex_size_kb: 0.00    # index size in KB\nmode: userkey          # running mode (userkey/sequential)\n\n## new fields\nworm: no               # write-once-read-multiple mode enabled\nlocked: no             # lock (read-only or even write disabled) mode\n\nnext_internal_id: 0x00000000    # internal next key id\nstats_index_io_errors: 0        # amount of index read/write io error\nstats_index_io_error_last: 0    # last timestamp of index io error\nstats_index_faults: 0           # always 0 for now\nstats_data_io_errors: 0         # amount of data read/write io error\nstats_data_io_error_last: 0     # timestamp of last io error\nstats_data_faults: 0            # always 0 for now\n\nindex_disk_freespace_bytes: 57676599296    # free space on index partition (bytes)\nindex_disk_freespace_mb: 55004.69          # free space on index partition (megabytes)\ndata_disk_freespace_bytes: 57676599296     # free space on data partition (bytes)\ndata_disk_freespace_mb: 55004.69           # free space on data partition (metabytes)\n\ndata_path: /tmp/zdb-data/default           # namespace physical data path (only available for admin)\nindex_path: /tmp/zdb-index/default         # namespace physical index path (only available for admin)\n```\n\nFields `stats_index_` and `stats_data_` fields are useful to know if partition on which data and index\nlive had issues during running time.\n\n## NSLIST\nReturns an array of all available namespaces.\n\n## NSSET\nChange a namespace setting/property. Only admin can do this.\n\nProperties:\n* `maxsize`: set the maximum size in bytes, of the namespace's data set\n* `password`: lock the namespace by a password, use `*` password to clear it\n* `public`: change the public flag, a public namespace can be read-only if a password is set (0 or 1)\n* `worm`: « write only read multiple » flag which disable overwrite and deletion (0 or 1)\n* `mode`: change index mode (`user` or `seq`)\n* `lock`: set namespace in read-only or normal mode (0 or 1)\n* `freeze`: set namespace in read-write protected or normal mode (0 or 1)\n\nAbout mode selection: it's now possible to mix modes (user and sequential) on the same 0-db instance.\nThis is only possible if you don't provide any `--mode` argument on runtime, otherwise 0-db will be available\nonly on this mode.\n\nIt's only possible to change mode on a fully empty dataset (no deleted keys, nothing.), aka on a newly created\nnamespace. You can change `default` namespace aswell if it's empty.\n\nAs soon as there are a single object in the namespace, you won't be able to change mode.\n\n`LOCK` mode won't change anything for read queries, but any update (set, del, ...) will be\ndenied with an error message (eg: `Namespace is temporarily locked`).\n\n`FREEZE` mode will deny any operation on the specific namespace, read, write, update, delete operations\nwill be denied with an error message (eg: `Namespace is temporarily frozen`)\n\n## NSJUMP\nForce closing current index and data files and open the next id.\n\n## SELECT\nChange your current namespace. If the requested namespace is password-protected, you need\nto add the password as extra parameter. If the namespace is `public` but password protected,\nand you don't provide any password, the namespace will be accessible in read-only.\n\nYou can use SECURE password, like authentication (see below). A challenge is required first\n(using `AUTH SECURE CHALLENGE` command). See `AUTH` below for more information.\n\n```\n\u003e\u003e AUTH SECURE CHALLENGE\n749e5be04ca0471e\n\u003e\u003e SELECT hello SECURE 632ef9246e9f01a3453aec8f133d1f652cccebbb\nOK\n```\n\n## AUTH\nIf an administrator password is set, use `AUTH` command to authentificate yourself as `ADMIN`.\nThere is two way possible to request authentication.\n\n### Legacy plain-text authentication\nYou can authenticate yourself using the simple `AUTH \u003cpassword\u003e` way. This is still supported and valid.\nIn the futur this will be probably disabled for security reason.\n\nThere is no encryption between client and 0-db server, any network monitor could leak\nadministration password.\n\n### Secure authentication\nThere is a more advanced two-step authentication made to avoid plain-text password leak and safe\nagainst replay-attack.\n\nYou can authenticate yourself using the `AUTH SECURE` command, in two step.\n- First you request a challenge using `AUTH SECURE CHALLENGE`\n- Then you authenticate yourself using `AUTH SECURE sha1(challenge:password)`\n\nThe challenge is session-specific random nonce. It can be used only 1 time.\nLet assume the password is `helloworld`, the authentication workflow is:\n```\n\u003e\u003e AUTH SECURE CHALLENGE\n708f109fbef4d656\n\u003e\u003e AUTH SECURE 5af26c9c8bf4db0b342c42fc47e3bdae58da4578\nOK\n```\n\nThe secure password is constructed via `sha1(708f109fbef4d656:helloworld)`\n\nThis is the prefered method to use.\n\n## WAIT\nBlocking wait on command execution by someone else. This allows you to wait somebody else\ndoing some commands. This can be useful to avoid polling the server if you want to do periodic\nqueries (like waiting for a `SET`).\n\nWait takes one or two arguments: a command name to wait for and an optional timeout.\nThe event will only be triggered for clients on the same namespace as you (same `SELECT`),\nthe special command '`*`' can be used to wait on any commands.\n\nThe optional timeout argument is in milliseconds, by default, timeout is set to 5 seconds if no\ntimeout is provided. Timeout range can be set from 100ms to 30 minutes. Timeout precision is not\nalways exact and can be slightly different than expected, depending on server load.\n\nThis is the only blocking function right now. In server side, your connection is set `pending` and\nyou won't receive anything until someone executed the expected command or timeout occures.\n\nWhen the command is triggered by someone else, you receive `+COMMAND_NAME` as response. If your reached\nthe timeout, you receive `-Timeout` error.\n\n## HISTORY\nThis command allows you to go back in time, when your overwrite a key.\n\nYou always need to set the expected key as first argument: `HISTORY mykey`\n\nWithout more argument, you'll get information about the current state of the key.\nThe returned value are always the same format, an array made like this:\n\n1. A binary string which can be used to go deeper on the history\n2. The timestamp (unix) when the key was created\n3. The payload of the data at that time\n\nTo rollback in time, you can follow the history by calling again the same command, with\nas extra argument the first key received (a binary string). Eg: `HISTORY mykey \"\\x00\\x00\\x1b\\x00\\x00\\x00\"`\n\nWhen requesting an extra argument, you'll get the previous entry. And so on...\n\n## FLUSH\nTruncate a namespace contents. This is a really destructive command, everything is deleted and no\nrecovery is possible (history, etc. are deleted).\n\nThis is only allowed on private and password protected namespace. You need to select the namespace\nbefore running the command.\n\n## HOOKS\n\nThis command is reserved to admin. It lists running (or recently running) hooks and their status.\n\nThis list contains a list of hooks. Each entry contains 6 fields:\n```\n\u003e HOOKS\n1) 1) \"ready\"\n   2) (empty array)\n   3) (integer) 18621\n   4) (integer) 1615511907\n   5) (integer) 1615511907\n   6) (integer) 0\n```\n\nValues are:\n1. Hook Type (name)\n2. Extra arguments (depend of the type)\n3. Process ID (pid)\n4. Timestamp when hook were created\n5. Timestamp when hook finished\n6. Status Code (return code)\n\nA still running hook will have a positive created timestamp but zero\nfinished timestamp.\n\nExample of a hook triggered when 'namespace2' namespace were created:\n```\n\u003e HOOKS\n1) 1) \"namespace-created\"\n   2) 1) \"namespace2\"\n   3) (integer) 18447\n   4) (integer) 1615511807\n   5) (integer) 1615511818\n   6) (integer) 2\n```\n\nThe database keep a volatile list, only recent hooks are kept and list\nis cleaned up after some time for now it's 60 seconds. This could changes.\n\nIf no recent hooks were executed, list is empty:\n```\n\u003e HOOKS\n(empty array)\n```\n\n## INDEX\n\nThis command have only a small subset of commands. This covers some information about index.\n\n### INDEX DIRTY\nList the current namespace index files id which were modified since last reset\n\n### INDEX DIRTY RESET\nReset the dirty list\n\n## DATA\n\nThis command have small internal operation on raw data file.\n\n### DATA RAW\n\nAn internal call allows to retreive a raw data object from database only based on fileid and offset.\nThis method of access should only be made by a valid fileid and offset, some protection are in\nplace to avoid issues on wrong offset but not fully tested yet.\n    \nThis command (only for admin) returns an array with the object requested:\n  1. Key\n  2. Previous Offset\n  3. Integrity CRC32\n  4. Flags\n  5. Timestamp\n  6. Payload\n\nIn addition with NSINFO, this command can be used to query a database based on fielid/offset\nfrom another database used eg, for replication.\n\nYou can use fields `data_current_id` and `data_current_offset` from `NSINFO` to query valid offsets.\n\n## LENGTH\n\nReturns payload size of a key or `(nil)` if not found.\n\n```\n\u003e SET hello world\n\"hello\"\n\n\u003e LENGTH hello\n(integer) 5\n\n\u003e LENGTH notfound\n(nil)\n```\n\n## KEYTIME\n\nReturn last-update timestamp of a key or `(nil)` if not found.\n\n```\n\u003e SET hello world\n\"hello\"\n\n\u003e KEYTIME hello\n(integer) 1664996517\n\n\u003e KEYTIME notfound\n(nil)\n```\n\n# Namespaces\nA namespace is a dedicated directory on index and data root directory.\nA namespace is a complete set of key/data. Each namespace can be optionally protected by a password\nand size limited.\n\nYou are always attached to a namespace, by default, it's namespace `default`.\n\n## Protected mode\nIf you start the server using `--protect` flag, your `default` namespace will be set in read-only\nby default, and protected by the **Admin Password**.\n\nIf you're running protected mode, in order to do changes on default namespace, you need to explicitly\n`SELECT default [password]` to switch into read-write mode.\n\n# Hook System\nYou can request 0-db to call an external program/script, as hook-system. This allows the host\nmachine running 0-db to adapt itself when something happen.\n\nTo use the hook system, just set `--hook /path/to/executable` on argument.\nThe file must be executable, no shell are invoked. Executing a shell script with shebang is valid.\n\nWhen 0-db starts, it create it's own pseudo `identifier` based on listening address/port/socket.\nThis id is used on hooks arguments.\n\nFirst argument is `Hook name`, second argument is `Generated ID`, next arguments depends of the hook.\n\nCurrent supported hooks:\n\n| Hook Name             | Action                  | Blk | Arguments                  |\n| --------------------- | ----------------------- | --- | -------------------------- |\n| `ready`               | Server is ready         |  *  | (none)                     |\n| `close`               | Server closing (nicely) |  *  | (none)                     |\n| `jump-index`          | Index incremented       |     | See below                  |\n| `jump-data`           | Data incremented        |     | See below                  |\n| `crash`               | Server crashed          |     | (none)                     |\n| `namespaces-init`     | Pre-loading namespaces  |  *  | Index and data path        |\n| `namespace-created`   | New namespace created   |     | Namespace name             |\n| `namespace-updated`   | Namespace config update |     | Namespace name             |\n| `namespace-deleted`   | Namespace removed       |     | Namespace name             |\n| `namespace-reloaded`  | Namespace reloaded      |     | Namespace name             |\n| `namespace-closing`   | Unloading namespace     |     | Namespace name             |\n| `missing-data`        | Data file not found     |  *  | Missing filename           |\n\n**WARNING**: as soon as hook system is enabled, `ready` event needs to be handled correctly.\nIf hook returns something else than `0`, database initialization will be stopped.\n\nIn order to get `zdbd` starting with hook, `ready` _needs_ to returns `0` to valdate server that everything\nis okay and database can operate.\n\nNamespace configuration is any metadata which can be set via `NSSET` command. Any metadata change\nimply an update of `zdb-namespace` file.\n\nHook flagged `Blk` are blocking (waiting for hook to finish).\n\nSpecial notes about:\n - `jump-index`: arguments contains old index file and new index file, in addition, last argument\n    will contains a list (space separated) of id of dirty indexes. See INDEX DIRTY command above.\n - `jump-data`: arguments contains old data and new data file.\n - `namespaces-init`: blocking hook used to prepare or restore anything the database needs before\n    starting namespaces initialization. This hook can be used to restore state or create empty\n    namespaces. Arguments are:\n     - Index path\n     - Data path\n - `namespace-closing`: called when namespace is unloaded, basically this is only called on server\n    stop (gracefully or crash). This hook can be used to save namespace state.\n    Each namespace will trigger that hook with as arguments:\n     - Namespace name\n     - Current index filename\n     - Current data filename\n     - Dirty index list (see `jump-index` hook)\n\n# Data offload\nOne latest feature of 0-db is `data offloading`. When a datafile is full (reaches `--datasize`),\nthe file becomes immuable (thanks always-append).\n\nYou can offload theses datafiles to make some space locally and still get a fully working database\nif you're able to restore that file on demand.\n\nThe database only needs the index to fully operate, except when payload is requested (eg: with a `GET`).\nThis means even if datafiles are not present, you still can list, scan, delete and even update a key.\nIf you need the payload, the datafile needs to be restored.\n\n**WARNING**: index cannot be offloaded and needs to be always present and available, index is not immuable !\n\n## How to use offloader\nIn order to make offloader working correctly, you need to use the hook system.\n\nEach time a datafile is full and a new one is created, hook `jump-data` (and `jump-index`) are triggered.\nOn theses hooks, you can copy the datafile somewhere else (eg: a slow but safe backup solution). This file will\nnever be updated anymore.\n\nYou can remove any datafile **except the current active one**. You only can offload (remove)\nan immuable datafile. When that datafile will be needed, if it's not found, a hook will be triggered\n(`missing-data`) with the filename as argument. Note that during that time, 0-db will wait for the hook,\nblocking everything else.\n\n## Dirty index\nIf datafiles are immuable, indexfiles is another story. Previous files can be updated, this is needed\nto keep history working and other things. In order to keep the index sane in your backup solution, there\nis a new way in place to keep track of what was changed, to avoid saving the full index each time.\n\nYou can issue `INDEX DIRTY` command to zdb to get a list of index id which were updated since the last reset.\nThe reset is made via `INDEX DIRTY RESET` command.\n\nOn the `missing-data` hook, there is an extra argument which provide the dirty index list automatically and reset\nthe list after. You can use that list to copy index files updated in the same time.\n\n# Limitation\nBy default, datafiles are split when bigger than 256 MB.\n\nThe datafile id is stored on 32 bits, which makes maximum of 4,294,967,296 files.\nEach datafile cannot be larger than 4 GB.\n\nEach namespaces have their own datafiles, one namespace can contains maximum 16 EB (16,384 PB).\n\nDefault values (256 MB datafiles) set the limits to 1024 PB.\n\nThis limitation can be changed on startup via command line option `--datasize`, and provide (in bytes)\nthe size limit of a datafile. Setting `536870912` for example (which is 512 MB).\n\nPlease use always the same datasize accross multiple run, but using different size **should not** interfer.\n\n# Tests\nYou can run a sets of test on a running 0-db instance.\nTheses tests (optional) requires `hiredis` library.\n\nTo build the tests, type `make` in the `tests` directory.\n\nTo run the tests, run `./zdbtests` in the `tests` directory.\n\nWarning: for now, only a local 0-db using `/tmp/zdb.sock` unix socket is supported.\n\nWarning 2: please use an empty database, otherwise tests may fails as false-positive issue.\n\n# Repository Owner\n- [Maxime Daniel](https://github.com/maxux), Telegram: [@maxux](http://t.me/maxux)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthreefoldtech%2F0-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthreefoldtech%2F0-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthreefoldtech%2F0-db/lists"}