{"id":19665239,"url":"https://github.com/alttch/yedb","last_synced_at":"2025-02-27T03:44:08.050Z","repository":{"id":62590585,"uuid":"328758214","full_name":"alttch/yedb","owner":"alttch","description":"Simple and reliable key-value database","archived":false,"fork":false,"pushed_at":"2022-02-07T00:12:31.000Z","size":218,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-10T04:35:03.303Z","etag":null,"topics":["configs","database","key-value","reliable","rugged","serialization"],"latest_commit_sha":null,"homepage":"https://www.yedb.org","language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alttch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-11T18:37:17.000Z","updated_at":"2022-03-04T22:05:37.000Z","dependencies_parsed_at":"2022-11-03T22:48:06.930Z","dependency_job_id":null,"html_url":"https://github.com/alttch/yedb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alttch%2Fyedb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alttch%2Fyedb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alttch%2Fyedb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alttch%2Fyedb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alttch","download_url":"https://codeload.github.com/alttch/yedb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240974590,"owners_count":19887319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["configs","database","key-value","reliable","rugged","serialization"],"created_at":"2024-11-11T16:21:44.530Z","updated_at":"2025-02-27T03:44:08.032Z","avatar_url":"https://github.com/alttch.png","language":"Jinja","funding_links":[],"categories":[],"sub_categories":[],"readme":"# YEDB specifications\n\nVersion: **1**\n\n\u003cimg src=\"https://github.com/alttch/yedb/blob/main/yedb.png?raw=true\" align=\"right\"\u003e\n\n## 1. General\n\n- YEDB is the database format / engine, designed to store configuration files\n  and other reliable data.\n\n- YEDB is designed for reliability: the data survive any power loss, unless the\n  file system is broken.\n\n- YEDB is the data serialization / storage engine, used by\n  [Bohemia Automation](https://www.bohemia-automation.com) and\n  [AlterTech](https://www.altertech.com) in industrial and embedded software\n  IoT/IIoT products.\n\n- YEDB has the free data format which can be used under Apache 2.0 software\n  license.\n\n- YEDB uses a very simple data format: keys are stored as serialized regular\n  files. The format helps the database survive disasters and is highly\n  repairable.\n\n- YEDB libraries and client / server implementations are free, open-source and\n  provided under Apache 2.0 software license.\n\n## 1.1 Implementations\n\n- [yedb-py](https://github.com/alttch/yedb-py) - pure Python library and\n  client/server. Extra features: byte objects, database serialization format\n  conversion. \"pickle\" serialization format, which can store Python objects\n  directly (embedded only).\n\n- [yedb-rs](https://github.com/alttch/yedb-rs) - Rust version. Fast, reliable.\n  Byte objects are not supported.\n\n## 1.2 References\n\nThe following third-party technologies are mentioned in this document:\n\n- JSON: https://www.json.org\n\n- MessagePack: https://msgpack.org\n\n- CBOR: https://cbor.io \n\n- YAML: https://yaml.org\n\n- JSON RPC: https://www.jsonrpc.org\n\n- JSON Schema: https://json-schema.org\n\n## 2. Database format\n\n### 2.1 Common\n\n- all meta date/time values are represented as UNIX timestamps in nanoseconds\n  (unsigned 64-bit integer).\n\n- all integer numbers are stored with little-endian order.\n\n### 2.2 Meta information file\n\nThe file MUST be called \".yedb\" and stored in the database root directory. The\nfile MUST be serialized as JSON and contain the following fields:\n\n| Name      | Type   | Description                    |\n| --------- | ------ | ------------------------------ |\n| fmt       | String | data serialization format code |\n| created   | u64    | database creation timestamp    |\n| version   | u16    | engine version                 |\n| checksums | bool   | data storage checksums enabled |\n\n### 2.3 Key files\n\n#### 2.3.1 General\n\nKey values MUST be stored in regular files in \"keys\" subdirectory. The format\nMUST be kept to allow system administrators repair a database without any\nexternal tools. The full path tree, where a key file is stored, MUST represent\nthe key full name.\n\nExample: a key, named \"my/cool/key\" should be:\n\n- stored in \"keys/my/cool\" directory\n\n- named \"key\" with the extension, representing the data serialization format\n\n- keys, which have the path starting with dot (\".\") are considered as hidden\n\nWhen deleted, the engine SHOULD automatically remove unnecessary directories in\nthe path tree.\n\nThe root key SHOULD NOT have any value. Other \"parent\" keys MAY have values set.\n\n#### 2.3.2 File format\n\nThe database can have files of the same format only. Checksums are enabled or\ndisabled globally, for all keys in the database.\n\n##### 2.3.2.1 Checksums disabled\n\nIf checksums are disabled, key files contain serialized data as-is. This is\nmore easy for manually repairing the database, but less reliable for data\nintegrity.\n\n##### 2.3.2.2 Checksums enabled\n\n###### 2.3.2.2.1 Binary serialization formats\n\n| Byte range | Size | Value                               |\n| ---------- | ---- | ----------------------------------- |\n| 0-31       | 32   | SHA256-checksum of serialized data  |\n| 32-39      | 8    | Set timestamp                       |\n| 40-        |      | Serialized data                     |\n\n###### 2.3.2.2.2 Text serialization formats\n\n| String | Value                    \n| ------ | ------------------------------------------- |\n| 1      | SHA256-checksum of serialized data (HEX)    |\n| 2      | Set timestamp (hex)                         |\n| 3-N    | Serialized data                             |\n\n- the strings MUST have no white spaces\n\n- the file MUST have UNIX (new line = LF) encoding\n\n- the serialized data SHOULD have LF ending to let key files be edited with\n  regular text editors.\n\n### 2.4 Data serialization\n\n#### 2.4.1 Data formats\n\nThe current YEDB specifications document defines the following serialization\nformats:\n\n| Name    | Code | Mandatory  | File suffix | With c-sums | Type   |\n| ------- | ---- | ---------- | ----------- | ----------- | ------ |\n| json    | 1    | Y(default) | .json       | .jsonc      | text   |\n| msgpack | 2    | Y          | .mp         | .mpc        | binary |\n| cbor    | 3    | N          | .cb         | .cbc        | binary |\n| yaml    | 4    | N          | .yml        | .ymlc       | text   |\n\n#### 2.4.2 Data types\n\n| Name        | Mandatory |\n| ----------- | --------- |\n| null        | Y         |\n| boolean     | Y         |\n| number      | Y         |\n| string      | Y         |\n| array       | Y         |\n| object      | Y         |\n| bytes       | N         |\n\nThe database MAY implement additional data types.\n\n#### 2.4.3 Data type schemas\n\nThe database MAY implement strict type / structure checking for keys.\n\nIf implemented, the implementation MUST satisfy the following requirements:\n\n- Key schemas are defined as *.schema/path/to/key* or *.schema/path* keys.\n\n- A schema MUST be applied to individual keys or to all their sub keys, unless\n  the lower-level schema is defined. E.g. the schema named *.schema/group1* is\n  applied to all *group1* keys, unless some key *group1/key1* has the\n  individual schema at *.schema/group1/key1*.\n\n- The key schemas MUST implement [JSON Schema](https://json-schema.org).\n\n- The key schemas MAY extend JSON Schema and implement additional data types.\n\n## 3. Engine\n\n### 3.1 Basics\n\n- The database engine MUST support locking. Usually the locking can be performed\nwith an exclusively-locked file, respected by other engine instances.\n\n- The default lock file SHOULD be called \"db.lock\" and created in the database\n  root directory. The lock file SHOULD contain the owner process ID.\n\n- The engine MUST allow variable location of the lock file, allowing using of\n  databases on read-only file systems.\n\n- The embedded engine MUST be thread-safe out-of-the-box or provide\n  documentation how to implement thread-safe usage.\n\n- The embedded engine MAY offer asynchronous read/writes if keys are stored on\n  different physical drives.\n\n- Client/server architecture is optional.\n\n- The engine MAY implement data caching for read operations ONLY.\n\n- Engine methods MAY accept keys with leading slashes (\"/path/to/key\"), but\n  MUST represent all keys without them.\n\n#### 3.1.1 Writing and flushing data\n\nIf data flushing is enabled, the key and database data MUST be written into\ntemporary files. After, these files MUST be flushed with the corresponding\nsystem call (e.g.  *fd.flush(); libc::fsync(fd)*).\n\nAfter the successful flushing:\n\n- the key MUST be renamed to its actual name\n\n- the key directory MUST be flushed as well\n\n- if \"write\\_modified\\_only\" parameter is set, the engine MUST make sure the\n  key value is changed before writing any data to the disk.\n\n### 3.2 Public database object variables\n\nThe database object MUST have the following variables, defined either as public\nor provide setters for them:\n\n| Name                  | Type   | Default | Description                            |\n| --------------------- | ------ | ------- | -------------------------------------- |\n| auto\\_repair          | bool   | true    | Auto-repair the database when opened   |\n| auto\\_flush           | bool   | true    | Flush key data to disk immediately     |\n| lock\\_ ex             | bool   | true    | Lock the database exclusively on open  |\n| write\\_modified\\_only | bool   | true    | Write to disk modified key values only |\n\n### 3.3 Mandatory methods\n\n| Name                   | Args                          | Brief description                                        |\n| ---------------------- | ----------------------------- | -------------------------------------------------------- |\n| purge                  |                               | Remove all except key files and meta, delete broken keys |\n| safe\\_purge            |                               | The same as purge but do not delete broken keys          |\n| repair                 |                               | Repair broken keys, deletes unrepairable                 |\n| check                  |                               | Check keys                                               |\n| info                   |                               | Get database info                                        |\n| server\\_set            | name: String, value: Value    | Modify server options                                    |\n| key\\_exists            | key: String                   | Return boolean True if the key exists; False if does not |\n| key\\_get               | key: String                   | Get key value                                            |\n| key\\_explain           | key: String                   | Get key value and extended info                          |\n| key\\_set               | key: String, value            | Set key value                                            |\n| key\\_decrement         | key: String                   | Decrement values of numeric (integer) keys               |\n| key\\_delete            | key: String                   | Delete the key                                           |\n| key\\_delete\\_recursive | key: String                   | Delete the key and all its subkeys                       |\n| key\\_increment         | key: String                   | Increment value of numeric (integer) keys                |\n| key\\_list              | key: String                   | List the key and all its subkeys Vec\u003cString\u003e             |\n| key\\_list\\_all         | key: String                   | List key and all its subkeys, including hidden           |\n| key\\_get\\_recursive    | key: String                   | Get the key value and all subkeys Vec\\\u003cString, Value\\\u003e   |\n| key\\_copy              | key: String, dst\\_key: String | Copy the key value                                       |\n| key\\_rename            | key: String, dst\\_key: String | Rename the key / key tree                                |\n| key\\_dump              | key: String                   | Get value of the key and all subkeys, ignores broken     |\n| key\\_load              | data: Vec\\\u003cString, Value\\\u003e    | Load dumped keys back                                    |\n\n#### 3.3.1 Purge\n\nThe method MUST return a Generator\\\u003cString\\\u003e or an array/list of deleted broken\nkeys.\n\n#### 3.3.2 Repair\n\nThe method MUST return a Generator\\\u003c(String, bool)\\\u003e or an array/list of\ndeleted broken keys, where the bool value is *true* if the key is repaired and\n*false* if the key has been deleted.\n\n#### 3.3.3 Check\n\nThe method MUST return a Generator\\\u003cString\\\u003e or an array/list of broken keys.\n\n#### 3.3.4 Info\n\nThe method MUST return the following data object:\n\n| Name                | Type             | Description                                            |\n| ------------------- | ---------------- | -----------------------------------------------    --- |\n| auto\\_flush         | bool             | Flush key data to disk immediately                     |\n| checksums           | bool             | Checksums enabled                                      |\n| created             | u64              | Database creation timestamp                            |\n| fmt                 | String           | Current data serialization format                      |\n| path                | bool             | Database path (server local)                           |\n| repair\\_recommended | bool             | Database \"repair recommended\" flag (not auto-repaired) |\n| server              | (String, String) | Server engine ID / Version (custom values)             |\n| version             | u16              | Engine version                                         |\n\nThe object MAY contain additional fields.\n\n#### 3.3.5 Server set\n\nThe following server parameters are allowed to be modified on-the-flow:\n\n| Name                |\n| ------------------- |\n| auto\\_flush         |\n| repair\\_recommended |\n\n#### 3.3.6 Key Explain\n\nThe method MUST return the following data object:\n\n| Name                | Type             | Description                                                |\n| ------------------- | ---------------- | ---------------------------------------------------------- |\n| file                | String           | Key file                                                   |\n| schema              | String           | JSON schema key if schema is defined                       |\n| len                 | u64              | length for strings, objects and arrays, null for others    |\n| mtime               | u64              | Key file modification timestamp                            |\n| stime               | u64              | Key modification timestamp; null if checksums are disabled |\n| sha256              |                  | SHA256-checksum, MUST be serialized to String              |\n| type                |                  | Value type (see 2.4.2), MUST be serialized to String       |\n| value               |                  | Key value                                                  |\n\nThe object MAY contain additional fields.\n\nIf key explain is requested for a schema key, its \"schema\" field MUST be\nstarted with \"!\" symbol to inform clients that the key does not physically\nexist in the database.\n\nIf the database engine has data type schemas (see 2.4.3) implemented, the\n*schema* field for *.schema* keys MUST contain the value \"!JSON Schema VERSION\",\ne.g. \"!JSON Schema Draft-7\".\n\nIf the schema implements custom data types, this MUST be clearly and properly\nexplained. E.g. if a key schema defines that keys must contain valid Python\ncode, the value MUST contain either \"Python\" or the link to\n[python.org](https://www.python.org/).\n\n#### 3.3.7 Key increment and decrement methods\n\n- The methods SHOULD support working with at least 64-bit integer values.\n\n- The methods SHOULD return the new key value after successful\n  increment/decrement.\n\n\n## 4. Engine API\n\n### 4.1 Basics\n\nThe engine MUST implement [JSON RPC 2.0](https://www.jsonrpc.org/specification)\nAPI with the following conditions:\n\n- batch requests: optional\n- requests without a response required (with no \"id\" field): optional\n- all mandatory engine methods MUST be provided\n\nThe engine MAY implement other APIs.\n\n### 4.2 Test\n\nThe engine API MUST implement \"test\" method, which MUST return the following\nstructure:\n\n| Name    | Type   | Description                  |\n| --------| ------ | ---------------------------- |\n| name    | String | MUST have the value = \"yedb\" |\n| version | u16    | Engine version               |\n\n### 4.3 Server types\n\n| Type        | Serialization formats | Notes                                                             |\n| ----------- | --------------------- | ----------------------------------------------------------------- |\n| UNIX socket | msgpack               | The name SHOULD have the suffix \".sock\" or \".socket\"              |\n| TCP socket  | msgpack               | The default port SHOULD be 8870                                   |\n| HTTP        | msgpack, json         | The default port SHOULD be 8878                                   |\n\n### 4.4 Binary packets format\n\nFor binary data exchange (UNIX/TCP sockets), the following format MUST be kept\nfor both client and server:\n\n| Byte range | Size | Value                                 |\n| ---------- | ---- | ------------------------------------- |\n| 0          | 1    | Engine version                        |\n| 1          | 1    | Data format code (2 for msgpack)      |\n| 2-5        | 4    | JSON RPC frame length (little-endian) |\n| 6-         |      | JSON RPC request / response frame     |\n\n### 4.5 HTTP\n\nThe HTTP transport MUST meet the following conditions:\n\n- The API MUST respond to HTTP/POST requests at HTTP ROOT URI (\"/\")\n- The API MUST accept JSON-encoded payloads by default\n- The API MUST accept MessagePack-encoded payloads for requests with content\n  type set to either \"application/msgpack\" or \"application/x-msgpack\".\n\n### 4.6 JSON RPC Error codes\n\n- Error replies SHOULD include correct and clear error messages.\n\n- Schema validation errors MUST return detailed error messages.\n\nError codes, returned by server, MUST match the following:\n\n### 4.6.1 Protocol errors\n\n| Code    | Description               |\n| ------- | --------------------------|\n| -32600  | Invalid request           |\n| -32601  | Method not found          |\n| -32602  | Invalid method parameters |\n\n### 4.6.2 Engine errors\n\n| Code    | Description                                    |\n| ------- | -----------------------------------------------|\n| -32001  | Key not found                                  |\n| -32002  | Data error, checksum error                     |\n| -32003  | Schema validation error                        |\n| -32004  | OS I/O errors: device errors, permissions etc. |\n| -32000  | All other server errors                        |\n\n\n## 5. Dump files\n\nIf the engine or a client creates / loads dump files, these files MUST have\ndata serialized with MessagePack and have the following format:\n\n### 5.1 File header\n\n| Byte range | Size | Value                               |\n| ---------- | ---- | ----------------------------------- |\n| 0          | 1    | Engine version                      |\n| 1          | 1    | Data format code (2 for msgpack)    |\n\n### 5.2 Key data\n\nStored, starting from byte 2, for each key:\n\n| Byte range | Size | Value                               |\n| ---------- | ---- | ----------------------------------- |\n| 0          | 4    | Data length                         |\n| 4-         |      | Data value (msgpack)                |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falttch%2Fyedb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falttch%2Fyedb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falttch%2Fyedb/lists"}