https://github.com/stephane-caron/palimpsest
Fast serializable C++ dictionaries
https://github.com/stephane-caron/palimpsest
cpp cpp17 dictionary json key-value messagepack msgpack serialization
Last synced: about 1 year ago
JSON representation
Fast serializable C++ dictionaries
- Host: GitHub
- URL: https://github.com/stephane-caron/palimpsest
- Owner: upkie
- License: apache-2.0
- Created: 2022-04-13T07:31:04.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2024-08-08T15:36:41.000Z (almost 2 years ago)
- Last Synced: 2024-08-09T15:34:49.922Z (almost 2 years ago)
- Topics: cpp, cpp17, dictionary, json, key-value, messagepack, msgpack, serialization
- Language: C++
- Homepage:
- Size: 696 KB
- Stars: 10
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# palimpsest — Fast serializable C++ dictionaries
[](https://github.com/stephane-caron/palimpsest/actions)
[](https://stephane-caron.github.io/palimpsest/)
[](https://coveralls.io/github/stephane-caron/palimpsest?branch=main)
[](https://github.com/stephane-caron/palimpsest/releases)
_palimpsest_ implements a `Dictionary` type for C++ meant for fast value updates and serialization. It is called [palimpsest](https://en.wiktionary.org/wiki/palimpsest#Noun) because dictionaries are designed for frequent rewritings (values change fast) on the same support (keys change slow).
## Example
Let's fill a dictionary and print it to the standard output:
```cpp
#include
#include
using palimpsest::Dictionary;
int main() {
Dictionary world;
world("name") = "example";
world("temperature") = 28.0;
auto& bodies = world("bodies");
bodies("plane")("orientation") = Eigen::Quaterniond{0.9239, 0.3827, 0., 0.};
bodies("plane")("position") = Eigen::Vector3d{0.0, 0.0, 100.0};
bodies("truck")("orientation") = Eigen::Quaterniond::Identity();
bodies("truck")("position") = Eigen::Vector3d{42.0, 0.0, 0.0};
std::cout << world << std::endl;
return EXIT_SUCCESS;
}
```
This code outputs:
```json
{"bodies": {"truck": {"position": [42, 0.5, 0], "orientation": [1, 0, 0, 0]},
"plane": {"position": [0.1, 0, 100], "orientation": [0.9239, 0.3827, 0, 0]}},
"temperature": 28, "name": "example"}
```
We can serialize the dictionary to file:
```cpp
world.write("world.mpack");
```
And deserialize it likewise:
```cpp
Dictionary world_bis;
world_bis.read("world.mpack");
std::cout << world_bis << std::endl;
```
Dictionaries can also be [serialized to bytes](#serialization-to-bytes) for transmission over TCP, memory-mapped files, telegraph lines, etc. Code in the [examples](https://github.com/stephane-caron/palimpsest/tree/main/examples) directory shows how to save and load dictionaries to and from C++ and Python.
## Link with Python dictionaries
_palimpsest_ will feel familiar if you are used to Python dictionaries, as its API is a subset of Python's `dict`:
| Python `dict` | _palimpsest_ `Dictionary` |
|---------------|---------------------------|
| `clear` | [`clear`](https://stephane-caron.github.io/palimpsest/classpalimpsest_1_1Dictionary.html#ae98a88dd6a1c5e5afa84f719189882d9) |
| `copy` | ✖️ |
| `fromkeys` | ✖️ |
| `get` | [`get`](https://stephane-caron.github.io/palimpsest/classpalimpsest_1_1Dictionary.html#a74bd56b0ec9e4219f54430bcb6f9a084) |
| `items` | ✖️ |
| `keys` | [`keys`](https://stephane-caron.github.io/palimpsest/classpalimpsest_1_1Dictionary.html#abb1589b67dbeadec8774833921644798) |
| `pop` | ✖️ |
| `popitem` | ✖️ |
| `setdefault` | ✖️ |
| `update` | [`update`](https://stephane-caron.github.io/palimpsest/classpalimpsest_1_1Dictionary.html#a1b5bb02bcf813b05aef280f47b25ce80) |
| `values` | ✖️ |
Implementing one of the missing functions is a great way to [contribute](CONTRIBUTING.md) to this project.
## Features and non-features
All design decisions have their pros and cons. Take a look at the features and non-features below to decide if it is also a fit for _your_ use case.
The two main assumptions in _palimpsest_ dictionaries are that:
* **Keys** are strings.
* **Values** hold either a sub-dictionary or a type that can be unambiguously serialized.
### Features
* Prioritizes speed over user-friendliness
* Returns references to any stored value or sub-dictionaries
* Built for fast inter-process communication with [Python](https://www.python.org/)
* Built-in support for [Eigen](https://eigen.tuxfamily.org/)
* Serialize to and deserialize from [MessagePack](https://msgpack.org/)
* Print dictionaries to standard output as [JSON](https://www.json.org/json-en.html)
* [Extensible](#adding-custom-types) to custom types, as long as they deserialize unambiguously
### Non-features
* Prioritizes speed over user-friendliness
* Array values are mostly limited to Eigen tensors (matrix, quaternion, vector)
* Copy constructors are disabled
* Custom types need to deserialize unambiguously
* Shallow and deep copies are not implemented ([PRs welcome](CONTRIBUTING.md))
Check out the existing [alternatives](https://github.com/stephane-caron/palimpsest#alternatives) if any of these choices is a no-go for you.
## Installation
### Bazel
Add the following to your `WORKSPACE` file:
```python
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "palimpsest",
sha256 = "d998b4e195ef75e558f0477da85ffd1961fb2a5b9ad1bafa1a378b6fa8931505",
strip_prefix = "palimpsest-2.3.0",
url = "https://github.com/stephane-caron/palimpsest/archive/refs/tags/v2.3.0.tar.gz",
)
load("@palimpsest//tools/workspace:default.bzl", add_palimpsest_repositories = "add_default_repositories")
# This adds dependencies such as @fmt and @mpack for building palimpsest targets
add_palimpsest_repositories()
```
You can then build C++ targets that depend on ``@palimpsest``:
```python
cc_binary(
name = "my-target",
srcs = ["my-target.cpp"],
deps = ["@palimpsest"],
)
```
### CMake
Make sure Eigen, fmt and spdlog are installed system-wise, for instance on Debian-based distributions:
```console
sudo apt install libeigen3-dev libfmt-dev libspdlog-dev
```
Then follow the standard CMake procedure:
```console
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j4
make install
```
Note that by default [MPack](https://github.com/ludocode/mpack) will be built and installed from the [``third_party``](https://github.com/stephane-caron/palimpsest/tree/main/third_party) folder. Set `-DBUILD_MPACK=OFF` if you already have MPack 1.1 or later installed on your system.
## Usage
### Serialization to bytes
Dictionaries can be serialized (``palimpsest::Dictionary::serialize``) to vectors of bytes:
```cpp
Dictionary world;
std::vector buffer;
size_t size = world.serialize(buffer);
```
The function resizes the buffer automatically if needed, and returns the number of bytes of the serialized message.
### Deserialization from bytes
Dictionaries can be updated (``palimpsest::Dictionary::update``) from byte vectors:
```cpp
Dictionary foo;
foo("bar") = 1;
foo("foo") = 2;
Dictionary bar;
bar("bar") = 3;
std::vector buffer;
size_t size = bar.serialize(buffer);
foo.update(buffer.data(), size); // OK, now foo("bar") == 3
```
Keys in the update stream that are not already in the dictionary are ignored:
```cpp
bar("new") = 4;
size_t size = bar.serialize(buffer);
foo.update(buffer.data(), size); // no effect
```
Updates therefore behave complementarily to extensions: updating `{"a": 12}` with `{"a": 42, "b": 1}` results in `{"a": 42}` rather than `{"a": 12, "b": 1}`.
### Adding custom types
Adding a new custom type boils down to the following steps:
* Add implicit type conversions to `Dictionary.h`
* Add a read function specialization to `mpack/read.h`
* Add a write function specialization to `mpack/Writer.h`
* Add a write function specialization to `mpack/write.h`
* Add a write function specialization to `json/write.h`
Take a look at the existing types in these files and in unit tests for inspiration.
## Q and A
> Why isn't _palimpsest_ also distributed as a header-only library?
The main blocker is that we set a custom flush function `mpack_std_vector_writer_flush` to our internal MPack writers. The [MPack Write API](https://ludocode.github.io/mpack/group__writer.html) requires a function pointer for that, and we define that function in [`Writer.cpp`](src/mpack/Writer.cpp). Open a PR if you have ideas to go around that!
## Alternatives
* [`mc_rtc::Configuration`](https://github.com/jrl-umi3218/mc_rtc/blob/master/include/mc_rtc/Configuration.h) - similar API to palimpsest, based on RapidJSON (see below).
* [`mc_rtc::DataStore`](https://github.com/jrl-umi3218/mc_rtc/blob/master/include/mc_rtc/DataStore.h) - can hold more general value types, like lambda functions, but does not serialize.
* [`mjlib::telemetry`](https://github.com/mjbots/mjlib/tree/master/mjlib/telemetry#readme) - if your use case is more specifically telemetry in robotics or embedded systems.
* [JSON for Modern C++](https://github.com/nlohmann/json) - most user-friendly library of this list, serializes to MessagePack and other binary formats, but not designed for speed.
* [Protocol Buffers](https://developers.google.com/protocol-buffers/) - good fit if you have a fixed schema (keys don't change at all) that you want to serialize to and from.
* [RapidJSON](https://github.com/Tencent/rapidjson/) - low memory footprint, can serialize to MessagePack using other [related projects](https://github.com/Tencent/rapidjson/wiki/Related-Projects), but has linear lookup complexity as it stores dictionaries [as lists of key-value pairs](https://github.com/Tencent/rapidjson/issues/102).
* [simdjson](https://github.com/simdjson/simdjson/) - uses SIMD instructions and microparallel algorithms to parse JSON (reportedly 4x faster than RapidJSON and 25x faster than JSON for Modern C++).