{"id":19850442,"url":"https://github.com/hideakitai/msgpacketizer","last_synced_at":"2025-04-15T00:06:46.820Z","repository":{"id":45910354,"uuid":"95091110","full_name":"hideakitai/MsgPacketizer","owner":"hideakitai","description":"msgpack based serializer / deserializer + packetize for Arduino, ROS, and more","archived":false,"fork":false,"pushed_at":"2024-07-24T17:53:45.000Z","size":4824,"stargazers_count":82,"open_issues_count":0,"forks_count":11,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-15T00:06:18.741Z","etag":null,"topics":["arduino","cpp","messagepack","openframeworks"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hideakitai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-22T08:11:19.000Z","updated_at":"2025-03-27T06:34:21.000Z","dependencies_parsed_at":"2024-07-24T20:15:02.486Z","dependency_job_id":"1bc52acd-abde-45c0-a113-8f109aba11b7","html_url":"https://github.com/hideakitai/MsgPacketizer","commit_stats":null,"previous_names":[],"tags_count":46,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hideakitai%2FMsgPacketizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hideakitai%2FMsgPacketizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hideakitai%2FMsgPacketizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hideakitai%2FMsgPacketizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hideakitai","download_url":"https://codeload.github.com/hideakitai/MsgPacketizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248981266,"owners_count":21193147,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arduino","cpp","messagepack","openframeworks"],"created_at":"2024-11-12T13:26:05.244Z","updated_at":"2025-04-15T00:06:46.784Z","avatar_url":"https://github.com/hideakitai.png","language":"C++","readme":"# MsgPacketizer\n\n[msgpack](https://github.com/msgpack/msgpack-c) based serializer / deserializer + packetize for Arduino, ROS, and more\n\n#### NOTE (\u003e= v0.5.0) : DEPENDENT LIBRARIES REMOVED\n\nIf you have already installed this library, please follow:\n\n- Cloned from GitHub (manually): Please install dependent libraries manually\n- Installed from library manager: re-install this library from library manager\n  - Dependent libraries will be installed automatically\n\n## Feature\n\n- one-line [serialize/deserialize] or [publish/subscribe] + packetize + robust [send/receive]\n- [serializer/deserializer] supports almost all standard type of C++ same as [msgpack-c](https://github.com/msgpack/msgpack-c)\n- support custom class [serialization/deserialization]\n- support one-line manual [serialization/deserialization] to work with any communication interface\n- support working with [ArduinoJSON](https://github.com/bblanchon/ArduinoJson)\n- [serializer/deserializer] based on [MsgPack](https://github.com/hideakitai/MsgPack)\n- packetize based on [Packetizer](https://github.com/hideakitai/Packetizer)\n- working also in ROS with [serial](https://github.com/wjwwood/serial) and [serial-ros2](https://github.com/RoverRobotics-forks/serial-ros2)\n\n## Packet Protocol\n\n| index  | msgpack | crc8   |\n| ------ | ------- | ------ |\n| 1 byte | N bytes | 1 byte |\n\n- 1 byte index (packet index can be used to identify packet)\n- **N byte serialized msgpack data**\n- 1 byte crc8 (for received data check)\n- these bytes are encoded to COBS encoding based on [Packetizer](https://github.com/hideakitai/Packetizer)\n\n## Usage\n\n### Direct Data Receive + Data Publishing\n\n```C++\n#include \u003cMsgPacketizer.h\u003e\n\n// input to msgpack\nint i;\nfloat f;\nMsgPack::str_t s; // std::string or String\nMsgPack::arr_t\u003cint\u003e v; // std::vector or arx::stdx::vector\nMsgPack::map_t\u003cString, float\u003e m; // std::map or arx::stdx::map\n\nuint8_t recv_index = 0x12;\nuint8_t send_index = 0x34;\n\nvoid setup() {\n    Serial.begin(115200);\n\n    // update received data directly\n    MsgPacketizer::subscribe(Serial, recv_index, i, f, s, v, m);\n\n    // publish varibales periodically (default 30[times/sec])\n    MsgPacketizer::publish(Serial, send_index, i, f, s, v, m);\n}\n\nvoid loop() {\n    // must be called to trigger callback and publish data\n    MsgPacketizer::update();\n}\n\n```\n\n### Callback with Received Objects + One-Line Send\n\n```C++\n#include \u003cMsgPacketizer.h\u003e\n\nuint8_t recv_index = 0x12;\nuint8_t send_index = 0x34;\n\nvoid setup() {\n    Serial.begin(115200);\n\n    // handle received data with lambda\n    // which has incoming argument types/data\n\n    MsgPacketizer::subscribe(Serial, recv_index,\n        [](int i, float f, MsgPack::str_t s, MsgPack::arr_t\u003cint\u003e v, MsgPack::map_t\u003cString, float\u003e m)\n        {\n            // send received data back in one-line\n            MsgPacketizer::send(Serial, send_index, i, f, s, v, m);\n        }\n    );\n}\n\nvoid loop() {\n    // must be called to trigger callback\n    MsgPacketizer::parse();\n}\n\n```\n\n### Nested Data with Custom Class\n\nTo serialize / deserialize nested data, defining custom class is recommended. For example, to make `{\"k1\": v, \"k2\":[i, f, s]}`:\n\n```C++\nstruct ArrayData {\n    int i; float f; MsgPack::str_t s;\n    MSGPACK_DEFINE(i, f, s); // [i, f, s]\n};\nstruct NestedData {\n    MsgPack::str_t k1, k2; int v;\n    ArrayData a;\n    MSGPACK_DEFINE_MAP(k1, v, k2, a); // {\"k1\": v, \"k2\":[i, f, s]}\n};\n```\n\nand you can serialize / deserialize your class completely same as other types.\n\n```C++\nNestedData n;\nMsgPacketizer::publish(Serial, send_index, n);\nMsgPacketizer::subscribe(Serial, recv_index, n);\n```\n\nPlease see examples and [MsgPack](https://github.com/hideakitai/MsgPack) for more detail.\n\n### Manual Encode / Decode with Any Communication I/F\n\nYou can just encode / decode data manually to use it with any communication interface.\nPlease note:\n\n- only one unsupoprted interface (serial, udp, tcp, etc.) is available for manual subscription because MsgPacketizer cannot indetify which data is from which device\n- `publish` is not available for unsupported data stream\n\n```C++\n#include \u003cMsgPacketizer.h\u003e\n\nconst uint8_t recv_index = 0x12;\nconst uint8_t send_index = 0x34;\n\nvoid setup() {\n    Serial.begin(115200);\n    delay(2000);\n\n    // subscribe the data for manual operation\n    MsgPacketizer::subscribe(recv_index,\n        [\u0026](const int i, const float f, const String\u0026 s) {\n            // just encode your data manually and get binary packet from MsgPacketizer\n            const auto\u0026 packet = MsgPacketizer::encode(send_index, i, f, s);\n            // send the packet data with your interface\n            Serial.write(packet.data.data(), packet.data.size());\n        }\n    );\n}\n\nvoid loop() {\n    // you should feed the received data manually to MsgPacketizer\n    const size_t size = Serial.available();\n    if (size) {\n        uint8_t* data = new uint8_t[size];\n        // you can get binary data from any communication interface\n        Serial.readBytes(data, size);\n        // feed your binary data to MsgPacketizer manually\n        // if data has successfully received and decoded, callback will be called\n        MsgPacketizer::feed(data, size);\n        delete[] data;\n    }\n}\n```\n\n### UDP and TCP Support\n\n#### TCP Support\n\n- start client first\n- everything else can be used in the same way\n\n```C++\n#include \u003cMsgPacketizer.h\u003e\n#include \u003cWiFi.h\u003e\n\nconst uint8_t index = 0x12;\nint i; float f; String s;\n\n// WiFi stuff\nWiFiClient client;\nconst char* host = \"192.168.0.10\";\nconst int port = 54321;\n\nvoid setup() {\n    WiFi.begin(\"your-ssid\", \"your-password\");\n\n    // start client\n    client.connect(host, port);\n\n    // everything else can be used in the same way\n    MsgPacketizer::publish(client, index, i, f, s)-\u003esetFrameRate(1);\n    MsgPacketizer::subscribe(client, index,\n        [\u0026](const int i, const float f, const String\u0026 s) {\n            // do something with received data\n        }\n    );\n}\n\nvoid loop() {\n    // do something with your variables i, f, s\n\n    // must be called to trigger callback and publish data\n    MsgPacketizer::update();\n}\n```\n\n#### UDP Support\n\n- start client first\n- set ip and port when you publish or send messages\n- everything else can be used in the same way\n\n```C++\n#include \u003cMsgPacketizer.h\u003e\n#include \u003cWiFi.h\u003e\n\nconst uint8_t index = 0x12;\nint i; float f; String s;\n\n// WiFi stuff\nWiFiUDP client;\nconst char* host = \"192.168.0.10\";\nconst int port = 54321;\n\nvoid setup() {\n    WiFi.begin(\"your-ssid\", \"your-password\");\n\n    // start client first\n    client.begin(port);\n\n    // set host and port when publishing or sending messages\n    MsgPacketizer::publish(client, host, port, index, i, f, s)-\u003esetFrameRate(1);\n    MsgPacketizer::subscribe(client, index,\n        [\u0026](const int i, const float f, const String\u0026 s) {\n            // do something with received data\n        }\n    );\n}\n\nvoid loop() {\n    // do something with your variables i, f, s\n\n    // must be called to trigger callback and publish data\n    MsgPacketizer::update();\n}\n```\n\n### ArduinoJson Support\n\n- supports only version \u003e 6.x\n- supports `StaticJsonDocument\u003cN\u003e` and `DynamicJsonDocument`\n- supports only `send`, `decode`, and `subscribe` with callbacks\n  - you can use `publish` and `subscribe` directly by reference, but not reccomended\n  - please see [this official document](https://arduinojson.org/v6/how-to/reuse-a-json-document/) for the detail\n\n```C++\n#include \u003cArduinoJson.h\u003e  // include before MsgPacketizer.h\n#include \u003cMsgPacketizer.h\u003e\n#include \u003cWiFi.h\u003e\n\nconst uint8_t msg_index = 0x12;\nconst char* host = \"192.168.0.10\";\nconst int port = 54321;\nWiFiUDP client;\n\nvoid setup() {\n    WiFi.begin(\"your-ssid\", \"your-password\");\n    client.begin(port);\n\n    MsgPacketizer::subscribe(client, msg_index,\n        [\u0026](const StaticJsonDocument\u003c200\u003e\u0026 doc) {\n            // do something with your json\n        }\n    );\n}\n\nvoid loop() {\n    StaticJsonDocument\u003c200\u003e doc;\n    // make your json here\n    MsgPacketizer::send(client, host, port, msg_index, doc);\n\n    delay(1000);\n\n    // must be called to trigger callback and publish data\n    MsgPacketizer::update();\n}\n```\n\n#### Buffer Size Adjustment when Subscribing `DynamicJsonDocument`\n\nCurrently we cannot calculate the json size from msgpack format before deserialization. Please adjust buffer size by defining following macro before including `MsgPacketizer.h`. Default is `size_of_msgpack_bytes * 3`.\n\n```C++\n#define MSGPACKETIZER_ARDUINOJSON_DESERIALIZE_BUFFER_SCALE 3  // default\n#include \u003cMsgPacketizer.h\u003e\n```\n\n## APIs\n\n### Subscriber\n\n```C++\nnamespace MsgPacketizer {\n\n    // ----- for unsupported communication interface with manual operation -----\n\n    // bind variables directly to specified index packet\n    template \u003ctypename... Args\u003e\n    inline void subscribe_manual(const uint8_t index, Args\u0026\u0026... args);\n    // bind variables directly to specified index packet with array format\n    template \u003ctypename... Args\u003e\n    inline void subscribe_manual_arr(const uint8_t index, Args\u0026\u0026... args);\n    // bind variables directly to specified index packet with map format\n    template \u003ctypename... Args\u003e\n    inline void subscribe_manual_map(const uint8_t index, Args\u0026\u0026... args);\n    // bind callback to specified index packet\n    template \u003ctypename F\u003e\n    inline void subscribe_manual(const uint8_t index, F\u0026\u0026 callback);\n    // bind callback which is always called regardless of index\n    template \u003ctypename F\u003e\n    inline void subscribe_manual(F\u0026\u0026 callback);\n    // unsubscribe\n    inline void unsubscribe_manual(const uint8_t index);\n\n    // feed packet manually: must be called to manual decoding\n    inline void feed(const uint8_t* data, const size_t size);\n\n\n    // ----- for supported communication interface (Arduino, oF, ROS) -----\n\n    template \u003ctypename S, typename... Args\u003e\n    inline void subscribe(S\u0026 stream, const uint8_t index, Args\u0026\u0026... args);\n    template \u003ctypename S, typename... Args\u003e\n    inline void subscribe_arr(S\u0026 stream, const uint8_t index, Args\u0026\u0026... args);\n    template \u003ctypename S, typename... Args\u003e\n    inline void subscribe_map(S\u0026 stream, const uint8_t index, Args\u0026\u0026... args);\n    template \u003ctypename S, typename F\u003e\n    inline void subscribe(S\u0026 stream, const uint8_t index, F\u0026\u0026 callback);\n    template \u003ctypename S, typename F\u003e\n    inline void subscribe(S\u0026 stream, F\u0026\u0026 callback);\n    template \u003ctypename S\u003e\n    inline void unsubscribe(const S\u0026 stream, const uint8_t index);\n    template \u003ctypename S\u003e\n    inline void unsubscribe(const S\u0026 stream);\n    template \u003ctypename S\u003e\n\n    // get UnpackerRef = std::shared_ptr\u003cMsgPack::Unpacker\u003e of stream and handle it manually\n    inline UnpackerRef getUnpackerRef(const S\u0026 stream);\n    // get map of unpackers and handle it manually\n    inline UnpackerMap\u0026 getUnpackerMap();\n\n    // must be called to receive packets\n    inline void parse(bool b_exec_cb = true);\n    inline void update(bool b_exec_cb = true);\n}\n```\n\n### Publisher\n\n```C++\nnamespace MsgPacketizer {\n\n    // ----- for unsupported communication interface with manual operation -----\n\n    // encode arguments directly with variable types\n    template \u003ctypename... Args\u003e\n    inline const Packetizer::Packet\u0026 encode(const uint8_t index, Args\u0026\u0026... args);\n    // encode binary data\n    inline const Packetizer::Packet\u0026 encode(const uint8_t index, const uint8_t* data, const uint8_t size);\n    // encode manually packed data\n    inline const Packetizer::Packet\u0026 encode(const uint8_t index);\n    // encode args as array format\n    template \u003ctypename... Args\u003e\n    inline const Packetizer::Packet\u0026 encode_arr(const uint8_t index, Args\u0026\u0026... args);\n    // encode args as map format\n    template \u003ctypename... Args\u003e\n    inline const Packetizer::Packet\u0026 encode_map(const uint8_t index, Args\u0026\u0026... args);\n\n\n    // ----- for supported communication interface (Arduino, oF, ROS) -----\n\n    // send arguments dilectly with variable types\n    template \u003ctypename S, typename... Args\u003e\n    inline void send(S\u0026 stream, const uint8_t index, Args\u0026\u0026... args);\n    // send binary data\n    template \u003ctypename S\u003e\n    inline void send(S\u0026 stream, const uint8_t index, const uint8_t* data, const uint8_t size);\n    // send manually packed data\n    template \u003ctypename S\u003e\n    inline void send(S\u0026 stream, const uint8_t index);\n    // send args as array format\n    template \u003ctypename S, typename... Args\u003e\n    inline void send_arr(S\u0026 stream, const uint8_t index, Args\u0026\u0026... args);\n    // send args as map format\n    template \u003ctypename S, typename... Args\u003e\n    inline void send_map(S\u0026 stream, const uint8_t index, Args\u0026\u0026... args);\n\n    // UDP version of send\n    template \u003ctypename... Args\u003e\n    inline void send(UDP\u0026 stream, const str_t\u0026 ip, const uint16_t port, const uint8_t index, Args\u0026\u0026... args);\n    inline void send(UDP\u0026 stream, const str_t\u0026 ip, const uint16_t port, const uint8_t index, const uint8_t* data, const uint8_t size);\n    inline void send(UDP\u0026 stream, const str_t\u0026 ip, const uint16_t port, const uint8_t index);\n    template \u003ctypename... Args\u003e\n    inline void send_arr(UDP\u0026 stream, const str_t\u0026 ip, const uint16_t port, const uint8_t index, Args\u0026\u0026... args);\n    template \u003ctypename... Args\u003e\n    inline void send_map(UDP\u0026 stream, const str_t\u0026 ip, const uint16_t port, const uint8_t index, Args\u0026\u0026... args);\n\n    // publish arguments periodically\n    template \u003ctypename S, typename... Args\u003e\n    inline PublishElementRef publish(const S\u0026 stream, const uint8_t index, Args\u0026\u0026... args);\n    // publish arguments periodically as array format\n    template \u003ctypename S, typename... Args\u003e\n    inline PublishElementRef publish_arr(const S\u0026 stream, const uint8_t index, Args\u0026\u0026... args);\n    // publish arguments periodically as map format\n    template \u003ctypename S, typename... Args\u003e\n    inline PublishElementRef publish_map(const S\u0026 stream, const uint8_t index, Args\u0026\u0026... args);\n    // unpublish\n    template \u003ctypename S\u003e\n    inline void unpublish(const S\u0026 stream, const uint8_t index);\n    // get registerd publish element class\n    template \u003ctypename S\u003e\n    inline PublishElementRef getPublishElementRef(const S\u0026 stream, const uint8_t index);\n\n    // UDP version of publish\n    template \u003ctypename... Args\u003e\n    inline PublishElementRef publish(const UDP\u0026 stream, const str_t\u0026 ip, const uint16_t port, const uint8_t index, Args\u0026\u0026... args);\n    template \u003ctypename... Args\u003e\n    inline PublishElementRef publish_arr(const UDP\u0026 stream, const str_t\u0026 ip, const uint16_t port, const uint8_t index, Args\u0026\u0026... args);\n    template \u003ctypename... Args\u003e\n    inline PublishElementRef publish_map(const UDP\u0026 stream, const str_t\u0026 ip, const uint16_t port, const uint8_t index, Args\u0026\u0026... args);\n    inline void unpublish(const UDP\u0026 stream, const str_t\u0026 ip, const uint16_t port, const uint8_t index);\n    inline PublishElementRef getPublishElementRef(const UDP\u0026 stream, const uint8_t index);\n\n    // must be called to publish data\n    inline void post();\n    // get MsgPack::Packer and handle it manually\n    inline const MsgPack::Packer\u0026 getPacker();\n}\n```\n\n## Other Options\n\n```C++\n#define MSGPACKETIZER_DEBUGLOG_ENABLE\n```\n\n## For NO-STL Boards\n\nFor following archtectures, several storage size for packets are limited.\n\n- AVR\n- megaAVR\n\n### Memory Management (only for NO-STL Boards)\n\nAs mentioned above, for such boards like Arduino Uno, the storage sizes are limited.\nAnd of course you can manage them by defining following macros.\nBut these default values are optimized for such boards, please be careful not to excess your boards storage/memory.\nThese macros have no effect for STL enabled boards.\n\n#### MsgPacketizer\n\n```C++\n// max publishing elemtnt size in one destination\n#define MSGPACKETIZER_MAX_PUBLISH_ELEMENT_SIZE 5\n// max destinations to publish\n#define MSGPACKETIZER_MAX_PUBLISH_DESTINATION_SIZE 1\n```\n\n#### MsgPack\n\n```C++\n// msgpack serialized binary size\n#define MSGPACK_MAX_PACKET_BYTE_SIZE 96\n// max size of MsgPack::arr_t\n#define MSGPACK_MAX_ARRAY_SIZE 3\n// max size of MsgPack::map_t\n#define MSGPACK_MAX_MAP_SIZE 3\n// msgpack objects size in one packet\n#define MSGPACK_MAX_OBJECT_SIZE 16\n```\n\n#### Packetizer\n\n```C++\n// max number of decoded packet queues\n#define PACKETIZER_MAX_PACKET_QUEUE_SIZE 1\n// max data bytes in packet\n#define PACKETIZER_MAX_PACKET_BINARY_SIZE 96\n// max number of callback for one stream\n#define PACKETIZER_MAX_CALLBACK_QUEUE_SIZE 3\n// max number of streams\n#define PACKETIZER_MAX_STREAM_MAP_SIZE 1\n```\n\n## Dependent Libraries\n\n- [DebugLog](https://github.com/hideakitai/DebugLog)\n- [MsgPack](https://github.com/hideakitai/MsgPack)\n- [Packetizer](https://github.com/hideakitai/Packetizer)\n\n## License\n\nMIT\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhideakitai%2Fmsgpacketizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhideakitai%2Fmsgpacketizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhideakitai%2Fmsgpacketizer/lists"}