{"id":19528345,"url":"https://github.com/stephane-caron/palimpsest","last_synced_at":"2025-04-26T11:33:14.589Z","repository":{"id":41855337,"uuid":"481116759","full_name":"upkie/palimpsest","owner":"upkie","description":"Fast serializable C++ dictionaries","archived":false,"fork":false,"pushed_at":"2024-08-08T15:36:41.000Z","size":713,"stargazers_count":10,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-08-09T15:34:49.922Z","etag":null,"topics":["cpp","cpp17","dictionary","json","key-value","messagepack","msgpack","serialization"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/upkie.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-13T07:31:04.000Z","updated_at":"2024-08-08T15:36:24.000Z","dependencies_parsed_at":"2023-11-10T15:11:39.976Z","dependency_job_id":"1bf78422-1958-4d69-99aa-20c4ace50dab","html_url":"https://github.com/upkie/palimpsest","commit_stats":null,"previous_names":["tasts-robots/palimpsest","upkie/palimpsest"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upkie%2Fpalimpsest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upkie%2Fpalimpsest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upkie%2Fpalimpsest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upkie%2Fpalimpsest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/upkie","download_url":"https://codeload.github.com/upkie/palimpsest/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224032380,"owners_count":17244420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","cpp17","dictionary","json","key-value","messagepack","msgpack","serialization"],"created_at":"2024-11-11T01:18:31.527Z","updated_at":"2025-04-26T11:33:14.576Z","avatar_url":"https://github.com/upkie.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# palimpsest — Fast serializable C++ dictionaries\n\n[![CI](https://img.shields.io/github/actions/workflow/status/stephane-caron/palimpsest/bazel.yml?branch=main)](https://github.com/stephane-caron/palimpsest/actions)\n[![Documentation](https://img.shields.io/badge/docs-online-brightgreen?style=flat)](https://stephane-caron.github.io/palimpsest/)\n[![Coverage](https://coveralls.io/repos/github/stephane-caron/palimpsest/badge.svg?branch=main)](https://coveralls.io/github/stephane-caron/palimpsest?branch=main)\n![C++ version](https://img.shields.io/badge/C++-17/20-blue.svg?style=flat)\n[![Release](https://img.shields.io/github/v/release/stephane-caron/palimpsest.svg?sort=semver)](https://github.com/stephane-caron/palimpsest/releases)\n\n_palimpsest_ implements a `Dictionary` type for C++ meant for fast value updates and serialization. It is called [palimpsest](https://en.wiktionary.org/wiki/palimpsest#Noun) because dictionaries are designed for frequent rewritings (values change fast) on the same support (keys change slow).\n\n## Example\n\nLet's fill a dictionary and print it to the standard output:\n\n```cpp\n#include \u003ciostream\u003e\n\n#include \u003cpalimpsest/Dictionary.h\u003e\n\nusing palimpsest::Dictionary;\n\nint main() {\n  Dictionary world;\n  world(\"name\") = \"example\";\n  world(\"temperature\") = 28.0;\n\n  auto\u0026 bodies = world(\"bodies\");\n  bodies(\"plane\")(\"orientation\") = Eigen::Quaterniond{0.9239, 0.3827, 0., 0.};\n  bodies(\"plane\")(\"position\") = Eigen::Vector3d{0.0, 0.0, 100.0};\n  bodies(\"truck\")(\"orientation\") = Eigen::Quaterniond::Identity();\n  bodies(\"truck\")(\"position\") = Eigen::Vector3d{42.0, 0.0, 0.0};\n\n  std::cout \u003c\u003c world \u003c\u003c std::endl;\n  return EXIT_SUCCESS;\n}\n```\n\nThis code outputs:\n\n```json\n{\"bodies\": {\"truck\": {\"position\": [42, 0.5, 0], \"orientation\": [1, 0, 0, 0]},\n\"plane\": {\"position\": [0.1, 0, 100], \"orientation\": [0.9239, 0.3827, 0, 0]}},\n\"temperature\": 28, \"name\": \"example\"}\n```\n\nWe can serialize the dictionary to file:\n\n```cpp\nworld.write(\"world.mpack\");\n```\n\nAnd deserialize it likewise:\n\n```cpp\nDictionary world_bis;\nworld_bis.read(\"world.mpack\");\nstd::cout \u003c\u003c world_bis \u003c\u003c std::endl;\n```\n\nDictionaries can also be [serialized to bytes](#serialization-to-bytes) for transmission over TCP, memory-mapped files, telegraph lines, etc. Code in the [examples](https://github.com/stephane-caron/palimpsest/tree/main/examples) directory shows how to save and load dictionaries to and from C++ and Python.\n\n## Link with Python dictionaries\n\n_palimpsest_ will feel familiar if you are used to Python dictionaries, as its API is a subset of Python's `dict`:\n\n| Python `dict` | _palimpsest_ `Dictionary` |\n|---------------|---------------------------|\n| `clear`       | [`clear`](https://stephane-caron.github.io/palimpsest/classpalimpsest_1_1Dictionary.html#ae98a88dd6a1c5e5afa84f719189882d9) |\n| `copy`        | ✖️  |\n| `fromkeys`    | ✖️  |\n| `get`         | [`get`](https://stephane-caron.github.io/palimpsest/classpalimpsest_1_1Dictionary.html#a74bd56b0ec9e4219f54430bcb6f9a084) |\n| `items`       | ✖️  |\n| `keys`        | [`keys`](https://stephane-caron.github.io/palimpsest/classpalimpsest_1_1Dictionary.html#abb1589b67dbeadec8774833921644798)  |\n| `pop`         | ✖️  |\n| `popitem`     | ✖️  |\n| `setdefault`  | ✖️  |\n| `update`      | [`update`](https://stephane-caron.github.io/palimpsest/classpalimpsest_1_1Dictionary.html#a1b5bb02bcf813b05aef280f47b25ce80) |\n| `values`      | ✖️  |\n\nImplementing one of the missing functions is a great way to [contribute](CONTRIBUTING.md) to this project.\n\n## Features and non-features\n\nAll design decisions have their pros and cons. Take a look at the features and non-features below to decide if it is also a fit for _your_ use case.\n\nThe two main assumptions in _palimpsest_ dictionaries are that:\n\n* **Keys** are strings.\n* **Values** hold either a sub-dictionary or a type that can be unambiguously serialized.\n\n### Features\n\n* Prioritizes speed over user-friendliness\n* Returns references to any stored value or sub-dictionaries\n* Built for fast inter-process communication with [Python](https://www.python.org/)\n* Built-in support for [Eigen](https://eigen.tuxfamily.org/)\n* Serialize to and deserialize from [MessagePack](https://msgpack.org/)\n* Print dictionaries to standard output as [JSON](https://www.json.org/json-en.html)\n* [Extensible](#adding-custom-types) to custom types, as long as they deserialize unambiguously\n\n### Non-features\n\n* Prioritizes speed over user-friendliness\n* Array values are mostly limited to Eigen tensors (matrix, quaternion, vector)\n* Copy constructors are disabled\n* Custom types need to deserialize unambiguously\n* Shallow and deep copies are not implemented ([PRs welcome](CONTRIBUTING.md))\n\nCheck out the existing [alternatives](https://github.com/stephane-caron/palimpsest#alternatives) if any of these choices is a no-go for you.\n\n## Installation\n\n### Bazel\n\nAdd the following to your `WORKSPACE` file:\n\n```python\nload(\"@bazel_tools//tools/build_defs/repo:http.bzl\", \"http_archive\")\n\nhttp_archive(\n    name = \"palimpsest\",\n    sha256 = \"d998b4e195ef75e558f0477da85ffd1961fb2a5b9ad1bafa1a378b6fa8931505\",\n    strip_prefix = \"palimpsest-2.3.0\",\n    url = \"https://github.com/stephane-caron/palimpsest/archive/refs/tags/v2.3.0.tar.gz\",\n)\n\nload(\"@palimpsest//tools/workspace:default.bzl\", add_palimpsest_repositories = \"add_default_repositories\")\n\n# This adds dependencies such as @fmt and @mpack for building palimpsest targets\nadd_palimpsest_repositories()\n```\n\nYou can then build C++ targets that depend on ``@palimpsest``:\n\n```python\ncc_binary(\n    name = \"my-target\",\n    srcs = [\"my-target.cpp\"],\n    deps = [\"@palimpsest\"],\n)\n```\n\n### CMake\n\nMake sure Eigen, fmt and spdlog are installed system-wise, for instance on Debian-based distributions:\n\n```console\nsudo apt install libeigen3-dev libfmt-dev libspdlog-dev\n```\n\nThen follow the standard CMake procedure:\n\n```console\nmkdir build \u0026\u0026 cd build\ncmake .. -DCMAKE_BUILD_TYPE=Release\nmake -j4\nmake install\n```\n\nNote that by default [MPack](https://github.com/ludocode/mpack) will be built and installed from the [``third_party``](https://github.com/stephane-caron/palimpsest/tree/main/third_party) folder. Set `-DBUILD_MPACK=OFF` if you already have MPack 1.1 or later installed on your system.\n\n## Usage\n\n### Serialization to bytes\n\nDictionaries can be serialized (``palimpsest::Dictionary::serialize``) to vectors of bytes:\n\n```cpp\nDictionary world;\nstd::vector\u003cchar\u003e buffer;\nsize_t size = world.serialize(buffer);\n```\n\nThe function resizes the buffer automatically if needed, and returns the number of bytes of the serialized message.\n\n### Deserialization from bytes\n\nDictionaries can be updated (``palimpsest::Dictionary::update``) from byte vectors:\n\n```cpp\nDictionary foo;\nfoo(\"bar\") = 1;\nfoo(\"foo\") = 2;\n\nDictionary bar;\nbar(\"bar\") = 3;\nstd::vector\u003cchar\u003e buffer;\nsize_t size = bar.serialize(buffer);\n\nfoo.update(buffer.data(), size);  // OK, now foo(\"bar\") == 3\n```\n\nKeys in the update stream that are not already in the dictionary are ignored:\n\n```cpp\nbar(\"new\") = 4;\nsize_t size = bar.serialize(buffer);\nfoo.update(buffer.data(), size);  // no effect\n```\n\nUpdates therefore behave complementarily to extensions: updating `{\"a\": 12}` with `{\"a\": 42, \"b\": 1}` results in `{\"a\": 42}` rather than `{\"a\": 12, \"b\": 1}`.\n\n### Adding custom types\n\nAdding a new custom type boils down to the following steps:\n\n* Add implicit type conversions to `Dictionary.h`\n* Add a read function specialization to `mpack/read.h`\n* Add a write function specialization to `mpack/Writer.h`\n* Add a write function specialization to `mpack/write.h`\n* Add a write function specialization to `json/write.h`\n\nTake a look at the existing types in these files and in unit tests for inspiration.\n\n## Q and A\n\n\u003e Why isn't _palimpsest_ also distributed as a header-only library?\n\nThe main blocker is that we set a custom flush function `mpack_std_vector_writer_flush` to our internal MPack writers. The [MPack Write API](https://ludocode.github.io/mpack/group__writer.html) requires a function pointer for that, and we define that function in [`Writer.cpp`](src/mpack/Writer.cpp). Open a PR if you have ideas to go around that!\n\n## Alternatives\n\n* [`mc_rtc::Configuration`](https://github.com/jrl-umi3218/mc_rtc/blob/master/include/mc_rtc/Configuration.h) - similar API to palimpsest, based on RapidJSON (see below).\n* [`mc_rtc::DataStore`](https://github.com/jrl-umi3218/mc_rtc/blob/master/include/mc_rtc/DataStore.h) - can hold more general value types, like lambda functions, but does not serialize.\n* [`mjlib::telemetry`](https://github.com/mjbots/mjlib/tree/master/mjlib/telemetry#readme) - if your use case is more specifically telemetry in robotics or embedded systems.\n* [JSON for Modern C++](https://github.com/nlohmann/json) - most user-friendly library of this list, serializes to MessagePack and other binary formats, but not designed for speed.\n* [Protocol Buffers](https://developers.google.com/protocol-buffers/) - good fit if you have a fixed schema (keys don't change at all) that you want to serialize to and from.\n* [RapidJSON](https://github.com/Tencent/rapidjson/) - low memory footprint, can serialize to MessagePack using other [related projects](https://github.com/Tencent/rapidjson/wiki/Related-Projects), but has linear lookup complexity as it stores dictionaries [as lists of key-value pairs](https://github.com/Tencent/rapidjson/issues/102).\n* [simdjson](https://github.com/simdjson/simdjson/) - uses SIMD instructions and microparallel algorithms to parse JSON (reportedly 4x faster than RapidJSON and 25x faster than JSON for Modern C++).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephane-caron%2Fpalimpsest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstephane-caron%2Fpalimpsest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephane-caron%2Fpalimpsest/lists"}