{"id":13812444,"url":"https://github.com/azonenberg/microkvs","last_synced_at":"2025-05-14T22:30:37.126Z","repository":{"id":97698445,"uuid":"415336799","full_name":"azonenberg/microkvs","owner":"azonenberg","description":"Tiny key-value store for persisting configuration data on microcontrollers","archived":false,"fork":false,"pushed_at":"2024-11-22T04:09:49.000Z","size":150,"stargazers_count":17,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-08T20:18:09.773Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/azonenberg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-09T14:47:15.000Z","updated_at":"2024-12-10T19:44:46.000Z","dependencies_parsed_at":"2024-06-04T19:22:19.019Z","dependency_job_id":"92fc0f21-45bc-488b-a7d6-5c4ae4658e66","html_url":"https://github.com/azonenberg/microkvs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azonenberg%2Fmicrokvs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azonenberg%2Fmicrokvs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azonenberg%2Fmicrokvs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azonenberg%2Fmicrokvs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/azonenberg","download_url":"https://codeload.github.com/azonenberg/microkvs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254239441,"owners_count":22037710,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T04:00:51.971Z","updated_at":"2025-05-14T22:30:36.840Z","avatar_url":"https://github.com/azonenberg.png","language":"C++","funding_links":[],"categories":["C++","Storage"],"sub_categories":["Data Bases"],"readme":"# Introduction\n\nmicrokvs is a log structured key-value store for embedded devices intended for storage of configuration data,\nencryption keys, and other relatively small information that needs to be persisted across power cycles. An object\noriented driver layer provides access to the underlying storage media, typically either unused space in code flash or\nan external SPI NOR flash. This enables microkvs to be easily ported to new platforms in the future.\n\nThere are no dependencies on dynamic RAM allocation or any fancy libc APIs that might be unavailable in an embedded\nplatform.\n\nThis is a work in progress and subject to change. Until a formal release, there may be breaking changes to the flash\ndata structure at any time.\n\n# General concepts\n\nFrom a programmer's perspective, microkvs provides access to a series of objects (arbitrary blobs of binary data),\nidentified by keys (16-character ASCII strings, analogous to file names although no hierarchy is supported).\n\nObjects are immutable and copy-on-write due to the nature of the underlying raw flash storage, which defaults to all 1s\nin the erased state and can only be written to logic 0. Writes typically support fine granularity (down to byte level)\nhowever erases have very large granularity (4 to 256 kB is common).\n\nmicrokvs does not support extent based / fragmented object storage. All objects are stored contiguously in the media;\nany change to an object requires completely rewriting it. If portions of an object need to be changed frequently while\nothers do not, consider breaking it up into several objects to reduce write amplification effects.\n\nIf memory mapping is supported by the underlying storage, objects can be directly memory mapped for read-only access.\nMemory mapped writing is not supported due to hardware limitations.\n\n# Architecture details\n\nThe backing store for microkvs consists of two equally sized \"banks\" of flash memory in separate erase blocks. Microkvs\ntreats each bank as one erase block at a logical level even if it physically consists of multiple blocks which must be\nerased in sequence, since the typical use case is to store a small amount of configuration data which easily fits in a\nsingle flash block.\n\nmicrokvs prefers byte writable storage. It can function with storage that has larger write block granularity (for\nexample STM32H7 internal flash memory, with a 256 bit / 32 byte write block size) however overhead is increased as all\ndata structures must be padded to multiples of a write block. Byte writable storage is assumed by default; to enable\npadding set a global preprocessor definition MICROKVS_WRITE_BLOCK_SIZE to the desired block size.\n\nThe store is divided into two regions, log and data. The split must be decided at compile time and cannot be changed\nlater on. The optimal split is application dependent and varies based on average file size weighted by how often each\nfile is modified. A minimum of 28 bytes of storage are required in the log area for each object stored in the data\narea, unless padded by minimum write block sizes.\n\nFor example, with 256K byte storage and 228 byte average file size, a good split would be 28K bytes of log and 228K\nbytes of data. This would allow roughly 1024 objects worth of both log and payload to be stored before both areas are\nexhausted simultaneously and a garbage collection is required.\n\n## Locating the active bank\n\nEach bank has a static header at the start of the log area containing a 32-bit version number. Version numbers start at\n0 and increment; the special value 0xffffffff is used for marking an invalid/blank block. No provision is made for\nhandling overflow as typical flash memory has far less than 2^32 erase cycles of endurance, so the media will wear out\nbefore the counter wraps.\n\nThe bank with the highest non-0xffffffff version number is active.\n\n## Locating an object\n\nTo locate an object, the log in the active bank is scanned from start to end searching for entries with the requested\nkey. The last matching entry in the log points to the current version of the object.\n\nThe CRC-32 checksum of the object is verified before the location is returned. If the checksum fails, earlier versions\nof the object (if present) are scanned, and the most recent version with a valid checksum is returned. If no copy with\na valid checksum can be located, an error is returned.\n\n## Writing an object\n\nTo write an object, the start address and length of the last valid log entry are used to calculate the location of the\nfirst free byte in the data area. If insufficient space is present, or no free log space is available, a garbage\ncollection must be performed.\n\nOnce the space is confirmed available, a new log entry is created and the start address and length are written. The key\nand CRC are left blank since the data is not yet present (the previous version of the object is current), however the\nspace is now reserved and cannot be used by another object.\n\nAfter reading back the log entry to confirm a correct write, the object content is written and the CRC is calculated.\nThe object content is immediately read back to confirm a correct write; if an error is seen the log entry is left with\na blank key and CRC (space reserved but data ignored) and the write is attempted again to a new location.\n\nOnce all object data has been successfully written, the CRC and name in the log entry are written to commit the object.\n\n## Garbage collection\n\nTo garbage collect, the inactive block is erased. The latest entry of each object is then written to the new block, and\nafter verification the block header is written with a new revision number one higher than the current.\n\n# Flash storage format\n\n## Bank header\n\n```\nuint32_t magic = 0xc0def00d\nuint32_t version\nuint32_t logSize\n```\n\n## Log entry\n\n```\nchar     key[16]\nuint32_t start\nuint32_t len\nuint32_t crc\n```\n\n## Data area\n\nData objects consist of raw binary data with a CRC-32 checksum (using the common Ethernet/ZIP polynomial 0x04c11db7)\nappended.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazonenberg%2Fmicrokvs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fazonenberg%2Fmicrokvs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazonenberg%2Fmicrokvs/lists"}