{"id":16836780,"url":"https://github.com/redblobgames/cpp-traverse","last_synced_at":"2025-03-22T04:30:56.361Z","repository":{"id":84749763,"uuid":"44649819","full_name":"redblobgames/cpp-traverse","owner":"redblobgames","description":"C++ Serialization library focusing on extensibility, both of input/output formats and of data types","archived":false,"fork":false,"pushed_at":"2025-01-28T21:42:54.000Z","size":88,"stargazers_count":17,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-18T08:11:20.647Z","etag":null,"topics":["cpp","serialization"],"latest_commit_sha":null,"homepage":"https://gasgame.net/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/redblobgames.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-10-21T03:15:32.000Z","updated_at":"2025-01-28T21:42:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"f8e652dc-5e5c-499a-a7cc-fbc34a9b6069","html_url":"https://github.com/redblobgames/cpp-traverse","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redblobgames%2Fcpp-traverse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redblobgames%2Fcpp-traverse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redblobgames%2Fcpp-traverse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redblobgames%2Fcpp-traverse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/redblobgames","download_url":"https://codeload.github.com/redblobgames/cpp-traverse/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244907420,"owners_count":20529850,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","serialization"],"created_at":"2024-10-13T12:14:49.348Z","updated_at":"2025-03-22T04:30:55.936Z","avatar_url":"https://github.com/redblobgames.png","language":"C++","readme":"* Motivation\n\nWhen using some serialization libraries I have to write code to traverse my data structures and then put each of my struct's fields into the format the serialization library wants, and then I have to do this again in reverse. I have to write this traversal code again when converting to JSON or Lua. I have to write it again when writing a debug GUI that lets me see all the values. I wanted to reduce the boilerplate.\n\n1. *I wanted it to be easy* to turn a struct into a serializable struct, ideally just one line of code per struct.\n1. I didn't want to introduce new build steps or external tools.\n1. I wanted to be able to serialize existing structs /without/ modifying their definitions.\n1. I wanted to add support for containers (vector, set, variant, etc.) /without/ modifying this library.\n1. I wanted to add support for formats (binary, JSON, Lua, etc.), /without/ modifying this library.\n1. I did /not/ need it to work for arbitrary data structures.\n\nThe result is a C++17 header-only serialization library.\n\n* Design\n\nThis isn't a general purpose library. I created it primarily for my own use. I plan to use it for (copyable, assignable, default constructible) structs with public fields, numbers, enums, std::string, and std::vector. For these data structures, the library defines a generic traversal routine that recursively visits each element, calling a serialization/deserialization function for each.\n\nSince there are many different data structures (/nouns/) and many different operations we want to do to them (/verbs/), there are two axes for extension:\n\n| Operation \\\\ Type  | enum | number | string | vector | struct1 | struct2 | ... |\n|--------------------+------+--------+--------+--------+---------+---------+-----|\n| Debug output       |      |        |        |        |         |         |     |\n| Binary serialize   |      |        |        |        |         |         |     |\n| Binary deserialize |      |        |        |        |         |         |     |\n| GUI output         |      |        |        |        |         |         |     |\n| ...                |      |        |        |        |         |         |     |\n\nTwo-axis extension can leave \"holes\" in the matrix. For example, if you add a new extension for Lua that handles enum, number, string, vector, and structs, and you also add a new extension for std::multiset that handles debug output, binary serialize, binary deserialize, and gui output, there's no code to handle the /combination/ of Lua and multiset. The library doesn't have a way to automatically fill those holes, but is designed to be \"open\" so that the end user can add a handle for any missing combinations. See [[file:traverse-picojson-variant.h][traverse-picojson-variant.h]] for an example of filling a hole (picojson extension + variant extension).\n\n* Installation\n\nThis is a header-only library, but has optional extensions for JSON (via [[https://github.com/kazuho/picojson][picojson]] or [[http://rapidjson.org/][rapidjson]]), [[https://www.lua.org/manual/5.2/][Lua 5.2]], and [[https://github.com/mapbox/variant][mapbox::variant]]. To install the dependencies for the extensions:\n\n#+begin_src sh\ngit submodule update --init\n#+end_src\n\n* Usage\n\nTo use any of the visitors, construct one, then call the visit function. The API varies by visitor type. Binary serialization uses a =std::streambuf= and reports no errors; binary deserialization also uses a =std::streambuf= and reports errors as a string:\n\n#+begin_src cpp\nstd::stringbuf buf;\ntraverse::BinarySerialize writer(buf);\nvisit(writer, yourobject);\n// buf.str() contains the serialized string\n\ntraverse::BinaryDeserialize reader(buf);\nvisit(reader, yourobject)\nif (!reader.Errors().empty()) throw reader.Errors();\nif (buf.getc() != std::streambuf::traits_type::eof()) {\n    throw \"not all bytes processed\";\n}\n#+end_src\n\nTo define traversal of a user-defined struct:\n\n#+begin_src cpp\nstruct Point {\n    int32_t x;\n    int32_t y;\n};\n\nTRAVERSE_STRUCT(Point, FIELD(x) FIELD(y))\n#+end_src\n\nAll serializable fields must be public.\n\n* Visitor types\n\nAs described in the Motivation section, I want to be able to extend the set of visitors or the set of data types. These are the visitor types included with the library:\n\n** Debug output\n\nThe =CoutWriter= writes everything to stdout. The =TRAVERSE_STRUCT= macro defines =operator \u003c\u003c(ostream\u0026)= to use =CoutWriter= for output.\n\n*** TODO User defined \u003c\u003c\n\nIf =operator \u003c\u003c= is already defined on your type =T=, the =TRAVERSE_STRUCT= macro fails. I need to provide an alternative.\n\n** Binary serialization\n\nThe =BinarySerialize= and =BinaryDeserialize= classes write/read to a simple binary format. There is no backwards/forwards compatibility, compression, optional fields, data structure sharing, zero-copy, support for multiple programming languages, or other nice features.\n\nIf there are structural errors during deserialization, the =errors= field will contain them. If the string is empty, there were no errors. The library does not perform semantic validation such as numbers being in range or an enum being one of the named items; you will have to write your own code for that.\n\nIntegers are encoded using Google's [[https://developers.google.com/protocol-buffers/docs/encoding][ZigZag format]] (from Google Protocol Buffers). It handles endian changes and also size changes. You can binary serialize a big endian int16 and binary deserialize into a little endian int32. You can't mix signed and unsigned ints.\n\nThe deserialization code is intended to handle malformed data. There is some fuzz testing of int, enum, struct, string, and vector deserialization using the [[http://lcamtuf.coredump.cx/afl/][AFL]] (American Fuzzy Lop); see the =fuzz-tests= rule in the Makefile.\n\nBinary serialization writes to and reads from a =std::streambuf=, which may be a string (=std::stringbuf=), file, stdin/stdout (=*std::cin.rdbuf()=, =*std::cout.rdbuf()=), or a custom streambuf derived class. To read from a block of memory without allocating a =std::stringbuf=:\n\n#+begin_src cpp\nstruct memorybuf : public std::streambuf {\n    memorybuf(char* begin, char* end) {\n        setg(begin, begin, end);\n    }\n};\n#+end_src\n\n** JSON serialization using picojson\n\nFor C++ to JSON, use a writer visitor to convert a C++ data structure into picojson value, then the json library can convert this into a JSON string. Example:\n\n#+begin_src cpp\npicojson::value output;\ntraverse::JsonWriter jsonwriter{output};\nvisit(jsonwriter, yourobject);\nstd::cout \u003c\u003c output.serialize();\n#+end_src\n\nIntegers, enums, and floats are written as JSON numbers. Strings, vectors, and structs are written as JSON strings, arrays, and objects.\n\nFor JSON to C++, use picojson to parse a JSON string into a picojson value, then a reader visitor to convert a picojson value into the C++ data structure. Example:\n\n#+begin_src cpp\npicojson::value input;\nauto err = picojson::parse(input, \"{\\\"a\\\": 3}\");\nif (!err.empty()) { throw \"parse error\"; }\nstd::stringstream errors;\ntraverse::JsonReader jsonreader{input, errors};\nvisit(jsonreader, yourobject);\nif (!errors.empty()) { throw \"type mismatch error\"; }\n#+end_src\n\nWhen deserializing, there may be type mismatches between the JSON data and the C++ data structures. The library leaves data unchanged in the object if it does not have new data to place there. If the JSON object does not contain all the fields in the user struct, or if the types don't match, those fields will be left unchanged. Any errors and warnings during deserialization are written to the =errors= stream. Use a stringstream that captures them; if the string is empty, there were no problems.\n\nIt is expected that you will put a convenience wrapper around this.\n\n** JSON serialization using rapidjson\n\nFor C++ to JSON, use a writer visitor to convert a C++ data structure into a rapidjson document, then the json library can convert this into a JSON string. Example:\n\n#+begin_src cpp\nrapidjson::StringBuffer output;\ntraverse::JsonWriter jsonwriter{output};\nvisit(jsonwriter, yourobject);\nstd::cout \u003c\u003c buffer.GetString();\n#+end_src\n\nIntegers, doubles, enums, and floats are written as JSON numbers. Bools are written as JSON bools. Strings, vectors, and structs are written as JSON strings, arrays, and objects.\n\nFor JSON to C++, use rapidjson to parse a JSON string into a rapidjson document, then a reader visitor to convert that into the C++ data structure. Example:\n\n#+begin_src cpp\nrapidjson::Document input;\ninput.Parse(\"json string\");\nif (input.HasParseError()) { throw \"parse error\"; }\nstd::stringstream errors;\ntraverse::RapidJsonReader jsonreader{input, errors};\nvisit(jsonreader, yourobject);\nif (!errors.empty()) { throw \"read error\"; }\n#+end_src\n\nWhen deserializing, there may be type mismatches between the JSON data and the C++ data structures. The library leaves data unchanged in the object if it does not have new data to place there. If the JSON object does not contain all the fields in the user struct, or if the types don't match, those fields will be left unchanged. Any errors and warnings during deserialization are written to the =errors= stream. Use a stringstream that captures them; if the string is empty, there were no problems.\n\nIt is expected that you will put a convenience wrapper around this.\n\n** Lua serialization\n\nThe Lua extension uses the C-Lua API for Lua 5.2. The writer converts a C++ value into a Lua equivalent and pushes it onto the the Lua stack.\n\n#+begin_src cpp\nlua_State* L;\ntraverse::LuaWriter luawriter{L};\nvisit(luawriter, yourobject);\n// this leaves the object at the top of the lua stack\n#+end_src\n\nIntegers, enums, and floats are written as Lua numbers; the library doesn't handle overflow. Strings are written as Lua strings. Vectors and structs are written as Lua tables.\n\nThe reader pops a value off the Lua stack and writes it to a C++ value.\n\n#+begin_src cpp\n// first put a lua object at the top of the stack\nstd::stringstream errors;\ntraverse::LuaReader luareader{L, errors};\nvisit(luareader, yourobject);\nif (!errors.empty()) { throw \"read error\"; }\n// the value will be popped off the lua stack\n#+end_src\n\nAs Lua is dynamically typed, and tables are used both as arrays and structs, there are several type mismatches that may occur when converting Lua to C++. See the =LuaReader= class in [[file:traverse-lua.h]] to control which type mismatches will be treated as errors and which will be ignored.\n\nIt is expected that you will put a convenience wrapper around this.\n\nI have also included a Lua-to-string function =lua_repr= and a string-to-Lua function =lua_eval= (primarily for unit tests) in [[file:lua-util.h]].\n\n** Other visitors\n\nThe intent of this library is to define data structure traversal separately from the serialization format, so you can write a visitor class to interface to Protocol Buffers, Thrift, Capn Proto, Flatbuffer, MsgPack, XML, YAML, or one of many other formats.  Although serialization is the primary use case, I've also used this library to visit the fields of data structures so that I can construct a debug GUI with the [[https://github.com/ocornut/imgui][dear imgui]] library; I haven't included that code here. Look at the existing visitors in [[file:traverse.h]], [[file:traverse-picojson.h]], [[file:traverse-rapidjson.h]] [[file:traverse-lua.h]] to see how to write a new visitor. You'll have to define how the visitor works with each data type (numbers, strings, vectors, structs).\n\n* Data types\n\nAs described in the Motivation section, I want to be able to extend the set of visitors or the set of data types. Each of the included visitors supports signed/unsigned integers, enum, class enum, std::string, std::vector, and user-defined structs. \n\nUse the =TRAVERSE_STRUCT= macro to define the visitor for a user-defined struct or class. For example: =TRAVERSE_STRUCT(Point, FIELD(x) FIELD(y))= will visit the =x= and =y= fields of the =Point= class.\n\nFor binary serialization, structs are written by serializing each field. For JSON, structs are written as JSON objects. For Lua, structs are converted into Lua tables.\n\n** Variant data types\n\nFor passing messages over a network or through an external message queue, I've used the [[https://github.com/mapbox/variant][mapbox::variant]] library, which is similar to [[http://theboostcpplibraries.com/boost.variant][boost::variant]] and [[http://en.cppreference.com/w/cpp/utility/variant][std::variant]]. Instead of sending /many/ types of messages =A=, =B=, =C= over the network, I send /one/ type, =variant\u003cA,B,C\u003e=. The variant keeps track of which type the message is.\n\nThis keeps the system simpler. I don't need serialization to know about multiple types; it only knows about serializing one type. The variant class knows about multiple types but not about serialization.\n\nThe code in [[file:traverse-variant.h]] will serialize a variant by first serializing the integer type code and then serializing the data. It will deserialize by first deserializaing the the type code, switching to that variant, then deserializing the data.\n\nOne of the downsides of two-axis extension is that there can be \"holes\" in the combinations of extensions. I did not define the variant+json or variant+lua combinations.\n\n** Other data types\n\nYou'll have to define how the data type works with each of the visitors that you want to use (binary serialize, binary deserialize, etc.). Look at [[file:traverse.h]] to see how string and vector work, or [[file:traverse-variant.h]] to see how data type extension works.\n\nI didn't need float/double binary serialization for my project so I didn't implement them, but the JSON and Lua extensions do handle floats/doubles.\n\nYou can override the visitor for a specific type. For example, consider this data structure:\n\n#+begin_src cpp\nstruct Message {\n  enum {A, B, C, D} x;\n  enum {P, Q, R, S} y;\n};\n#+end_src\n\nThe binary serialization will encode =x= and =y= to 1 byte each, for a total of 2 bytes. A more efficient encoding would use 2 bits for each, and could fit both into a total of 1 byte. You can define your own encoding for =Message= by defining =template\u003c\u003e void traverse::visit(BinarySerialize\u0026 writer, const Message\u0026 m)= and =template\u003c\u003e void traverse::visit(BinaryDeserialize\u0026 reader, Message\u0026 m)=.\n\n* Libraries\n\nThe picojson extension uses the [[https://github.com/kazuho/picojson][picojson]] library, licensed 2-clause BSD:\n\n#+begin_quote\nCopyright 2009-2010 Cybozu Labs, Inc.\nCopyright 2011-2014 Kazuho Oku\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice,\n   this list of conditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice,\n   this list of conditions and the following disclaimer in the documentation\n   and/or other materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\nARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\nLIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\nCONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\nSUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\nINTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\nCONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\nARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\nPOSSIBILITY OF SUCH DAMAGE.\n#+end_quote\n\nThe rapidjson library uses the [[http://rapidjson.org/][rapidjson]] library, licensed MIT:\n\n#+begin_quote\nTencent is pleased to support the open source community by making RapidJSON available. \n \nCopyright (C) 2015 THL A29 Limited, a Tencent company, and Milo Yip.  All rights reserved.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n#+end_quote\n\nThe Lua extension links with the C-Lua library (not included).\n\nThe Variant extension uses the [[https://github.com/mapbox/variant][mapbox::variant]] library, licensed 3-clause BSD:\n\n#+begin_quote\nCopyright (c) MapBox\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification,\nare permitted provided that the following conditions are met:\n\n- Redistributions of source code must retain the above copyright notice, this\n  list of conditions and the following disclaimer.\n- Redistributions in binary form must reproduce the above copyright notice, this\n  list of conditions and the following disclaimer in the documentation and/or\n  other materials provided with the distribution.\n- Neither the name \"MapBox\" nor the names of its contributors may be\n  used to endorse or promote products derived from this software without\n  specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR\nANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\nLOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON\nANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#+end_quote\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredblobgames%2Fcpp-traverse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredblobgames%2Fcpp-traverse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredblobgames%2Fcpp-traverse/lists"}