{"id":13736461,"url":"https://github.com/PMunch/protobuf-nim","last_synced_at":"2025-05-08T12:32:50.595Z","repository":{"id":29247279,"uuid":"120334551","full_name":"PMunch/protobuf-nim","owner":"PMunch","description":"Protobuf implementation in pure Nim that leverages the power of the macro system to not depend on any external tools","archived":false,"fork":false,"pushed_at":"2023-10-17T09:08:00.000Z","size":139,"stargazers_count":172,"open_issues_count":12,"forks_count":13,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-05T16:34:29.559Z","etag":null,"topics":["hacktoberfest","library","nim","protobuf","protocol-buffers","serialization"],"latest_commit_sha":null,"homepage":"","language":"Nim","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PMunch.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-02-05T16:59:38.000Z","updated_at":"2025-02-18T11:38:01.000Z","dependencies_parsed_at":"2023-01-14T14:30:14.230Z","dependency_job_id":"ac9e4359-ec41-460d-bbd9-73b622fffa46","html_url":"https://github.com/PMunch/protobuf-nim","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PMunch%2Fprotobuf-nim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PMunch%2Fprotobuf-nim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PMunch%2Fprotobuf-nim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PMunch%2Fprotobuf-nim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PMunch","download_url":"https://codeload.github.com/PMunch/protobuf-nim/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253069029,"owners_count":21848906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","library","nim","protobuf","protocol-buffers","serialization"],"created_at":"2024-08-03T03:01:22.242Z","updated_at":"2025-05-08T12:32:50.265Z","avatar_url":"https://github.com/PMunch.png","language":"Nim","funding_links":[],"categories":["Data","Nim"],"sub_categories":["Serialization"],"readme":"protobuf\n===========\nThis is a pure Nim implementation of protobuf, meaning that it doesn't rely\non the ``protoc`` compiler. The entire implementation is based on a macro\nthat takes in either a string or a file containing the proto3 format as\nspecified at https://developers.google.com/protocol-buffers/docs/proto3. It\nthen produces procedures to read, write, and calculate the length of a\nmessage, along with types to hold the data in your Nim program. The data\ntypes are intended to be as close as possible to what you would normally use\nin Nim, making it feel very natural to use these types in your program in\ncontrast to some protobuf implementations. Protobuf 3 however has all fields\nas optional fields, this means that the types generated have a little bit of\nspecial sauce going on behind the scenes. This will be explained in a later\nsection. The entire read/write structure is built on top of the Stream\ninterface from the ``streams`` module, meaning it can be used directly with\nanything that uses streams.\n\nExample\n-------\nTo whet your appetite the following example shows how this protobuf macro can\nbe used to generate the required code and read and write protobuf messages.\nThis example can also be found in the examples folder. Note that it is also\npossible to read in the protobuf specification from a file.\n\n.. code-block:: nim\n\n  import protobuf, streams\n\n  # Define our protobuf specification and generate Nim code to use it\n  const protoSpec = \"\"\"\n  syntax = \"proto3\";\n\n  message ExampleMessage {\n    int32 number = 1;\n    string text = 2;\n    SubMessage nested = 3;\n    message SubMessage {\n      int32 a_field = 1;\n    }\n  }\n  \"\"\"\n  parseProto(protoSpec)\n\n  # Create our message\n  var msg = new ExampleMessage\n  msg.number = 10\n  msg.text = \"Hello world\"\n  msg.nested = initExampleMessage_SubMessage(aField = 100)\n\n  # Write it to a stream\n  var stream = newStringStream()\n  stream.write msg\n\n  # Read the message from the stream and output the data, if it's all present\n  stream.setPosition(0)\n  var readMsg = stream.readExampleMessage()\n  if readMsg.has(number, text, nested) and readMsg.nested.has(aField):\n    echo readMsg.number\n    echo readMsg.text\n    echo readMsg.nested.aField\n\nGenerated code\n--------------\nSince all the code is generated from the macro on compile-time and not stored\nanywhere the generated code is made to be deterministic and easy to\nunderstand. If you would like to see the code however you can pass\n``-d:echoProtobuf`` switch on compile-time and the macro will output the\ngenerated code.\n\nOptional fields\n^^^^^^^^^^^^^^^\nAs mentioned earlier protobuf 3 makes all fields optional. This means that\neach field can either exist or not exist in a message. In many other protobuf\nimplementations you notice this by having to use special getter or setter\nprocs for field access. In Nim however we have strong meta-programming powers\nwhich can hide much of this complexity for us. As can be seen in the above\nexample it looks just like normal Nim code except from one thing, the call to\n``has``. Whenever a field is set to something it will register its presence\nin the object. Then when you access the field Nim will first check if it is\npresent or not, throwing a runtime ``ValueError`` if it isn't set. If you\nwant to remove a value already set in an object you simply call ``reset``\nwith the name of the field as seen in example 3. To check if a value exists\nor not you can call ``has`` on it as seen in the above example. Since it's a\nvarargs call you can simply add all the fields you require in a single check.\nIn the below sections we will have a look at what the protobuf macro outputs.\nSince the actual field names are hidden behind this abstraction the following\nsections will show what the objects \"feel\" like they are defined as. Notice\nalso that since the fields don't actually have these names a regular object\ninitialiser wouldn't work, therefore you have to use the \"init\" procs created\nas seen in the above example.\n\nMessages\n^^^^^^^^\nThe types generated are named after the path of the message, but with dots\nreplaced by underscores. So if the protobuf specification contains a package\nname it starts with that, then the name of the message. If the message is\nnested then the parent message is put between the package and the message.\nAs an example we can look at a protobuf message defined like this:\n\n.. code-block:: protobuf\n\n  syntax = \"proto3\"; // The only syntax supported\n  package = our.package;\n  message ExampleMessage {\n      int32 simpleField = 1;\n  }\n\nThe type generated for this message would be named\n``our_package_ExampleMessage``. Since Nim is case and underscore insensitive\nyou can of course write this with any style you desire, be it camel-case,\nsnake-case, or a mix as seen above. For this specific instance the type\nwould appear to be:\n\n.. code-block:: nim\n\n  type\n    our_package_ExampleMessage = ref object\n      simpleField: int32\n\nMessages also generate a reader, writer, and length procedure to read,\nwrite, and get the length of a message on the wire respectively. All write\nprocs are simply named ``write`` and are only differentiated by their types.\nThis write procedure takes two arguments plus an optional third parameter,\nthe ``Stream`` to write to, an instance of the message type to write, and a\nboolean telling it to prepend the message with a varint of its length or\nnot. This boolean is used for internal purposes, but might also come in handy\nif you want to stream multiple messages as described in\nhttps://developers.google.com/protocol-buffers/docs/techniques#streaming.\nThe read procedure is named similarily to all the ``streams`` module\nreaders, simply \"read\" appended with the name of the type. So for the above\nmessage the reader would be named ``read_our_package_ExampleMessage``.\nNotice again how you can write it in different styles in Nim if you'd like.\nOne could of course also create an alias for this name should it prove too\nverbose. Analagously to the ``write`` procedure the reader also takes an\noptional ``maxSize`` argument of the maximum size to read for the message\nbefore returning. If the size is set to 0 the stream would be read until\n``atEnd`` returns true. The ``len`` procedure is slightly simpler, it only\ntakes an instance of the message type and returns the size this message would\ntake on the wire, in bytes. This is used internally, but might have some\nother applications elsewhere as well. Notice that this size might vary from\none instance of the type to another as varints can have multiple sizes,\nrepeated fields different amount of elements, and oneofs having different\nchoices to name a few.\n\nEnums\n^^^^^\nEnums are named the same way as messages, and are always declared as pure.\nSo an enum defined like this:\n\n.. code-block:: protobuf\n\n  syntax = \"proto3\"; // The only syntax supported\n  package = our.package;\n  enum Langs {\n    UNIVERSAL = 0;\n    NIM = 1;\n    C = 2;\n  }\n\nWould end up with a type like this:\n\n.. code-block:: nim\n\n  type\n    our_package_Langs {.pure.} = enum\n      UNIVERSAL = 0, NIM = 1, C = 2\n\nFor internal use enums also generate a reader and writer procedure. These\nare basically a wrapper around the reader and writer for a varint, only that\nthey convert to and from the enum type. Using these by themselves is seldom\nuseful.\n\nOneOfs\n^^^^^^\nIn order for oneofs to work with Nims type system they generate their own\ntype. This might change in the future. Oneofs are named the same way as\ntheir parent message, but with the name of the oneof field, and ``_OneOf``\nappended. All oneofs contain a field named ``option`` of a ranged integer\nfrom 0 to the number of options. This type is used to create an object\nvariant for each of the fields in the oneof. So a oneof defined like this:\n\n.. code-block:: protobuf\n\n  syntax = \"proto3\"; // The only syntax supported\n  package our.package;\n  message ExampleMessage {\n    oneof choice {\n      int32 firstField = 1;\n      string secondField = 1;\n    }\n  }\n\nWill generate the following message and oneof type:\n\n.. code-block:: nim\n\n  type\n    our_package_ExampleMessage_choice_OneOf = object\n      case option: range[0 .. 1]\n      of 0: firstField: int32\n      of 1: secondField: string\n    our_package_ExampleMessage = ref object\n      choice: our_package_ExampleMessage_choice_OneOf\n\nExporting message definitions\n-----------------------------\nIf you want to re-use the same message definitions in multiple places in\nyour code it's a good idea to create a module for you definition. This can\nalso be useful if you want to rename some of the fields protobuf declares,\nor if you want to hide particular messages or create extra functionality.\nSince protobuf uses a little bit of magic under the hood a special\n`exportMessage` macro exists that will create the export statements you need\nin order to export a message definition from the module that reads the\nprotobuf specification, to any module that imports it. Note however that it\ndoesn't export sub-messages or any dependent types, so be sure to export\nthose manually. Anything that's not a message (such as an enum) should be\nexported by the normal `export` statement.\n\nLimitations\n-----------\nThis library is still in an early phase and has some limitations over the\nofficial version of protobuf. Noticably it only supports the \"proto3\"\nsyntax, so no optional or required fields. It also doesn't currently support\nmaps but you can use the official workaround found here:\nhttps://developers.google.com/protocol-buffers/docs/proto3#maps. This is\nplanned to be added in the future. It also doesn't support options, meaning\nyou can't set default values for enums and can't control packing options.\nThat being said it follows the proto3 specification and will pack all scalar\nfields. It also doesn't support services.\n\nThese limitations apply to the parser as well, so if you are using an\nexisting protobuf specification you must remove these fields before being\nable to parse them with this library.\n\nIf you find yourself in need of these features then I'd suggest heading over\nto https://github.com/oswjk/nimpb which uses the official protoc compiler\nwith an extension to parse the protobuf file.\n\nRationale\n---------\nSome might be wondering why I've decided to create this library. After all\nthe protobuf compiler is extensible and there are some other attempts at\nusing protobuf within Nim by using this. The reason is three-fold, first off\nno-one likes to add an extra step to their compilation process. Running\n``protoc`` before compiling isn't a big issue, but it's an extra\ncompile-time dependency and it's more work. By using a regular Nim macro\nthis is moved to a simple step in the compilation process. The only\nrequirement is Nim and this library meaning tools can be automatically\ninstalled through nimble and still use protobuf. It also means that all of\nNims targets are supported, and sending data between code compiled to C and\nJavascript should be a breeze and can share the exact same code for\ngenerating the messages. This is not yet tested, but any issues arising\nshould be easy enough to fix. Secondly the programatic protobuf interface\ncreated for some languages are not the best. Python for example has some\nrather awkward and un-natural patterns for their protobuf library. By using\na Nim macro the code can be customised to Nim much better and has the\npotential to create really native-feeling code resulting in a very nice\ninterface. And finally this has been an interesting project in terms of\npushing the macro system to do something most languages would simply be\nincapable of doing. It's not only a showcase of how much work the Nim\ncompiler is able to do for you through its meta-programming, but has also\nbeen highly entertaining to work on.\n\nThis file is automatically generated from the documentation found in\nprotobuf.nim. Use ``nim doc2 protobuf.nim`` to get the full documentation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPMunch%2Fprotobuf-nim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPMunch%2Fprotobuf-nim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPMunch%2Fprotobuf-nim/lists"}