{"id":16287136,"url":"https://github.com/bandysc/libprotobufzerosharp","last_synced_at":"2025-07-03T09:08:48.641Z","repository":{"id":240527313,"uuid":"802807420","full_name":"BAndysc/LibProtobufZeroSharp","owner":"BAndysc","description":"Simplified, but extremely fast protobuf encoder.","archived":false,"fork":false,"pushed_at":"2024-07-02T00:45:46.000Z","size":73,"stargazers_count":17,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-09T05:34:35.389Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BAndysc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-19T10:25:39.000Z","updated_at":"2025-04-01T22:09:53.000Z","dependencies_parsed_at":"2024-05-19T15:27:03.261Z","dependency_job_id":"294906ad-72e7-4a2c-87bb-e47845063202","html_url":"https://github.com/BAndysc/LibProtobufZeroSharp","commit_stats":null,"previous_names":["bandysc/libprotozerosharp","bandysc/libprotobufzerosharp"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BAndysc/LibProtobufZeroSharp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BAndysc%2FLibProtobufZeroSharp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BAndysc%2FLibProtobufZeroSharp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BAndysc%2FLibProtobufZeroSharp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BAndysc%2FLibProtobufZeroSharp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BAndysc","download_url":"https://codeload.github.com/BAndysc/LibProtobufZeroSharp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BAndysc%2FLibProtobufZeroSharp/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263296641,"owners_count":23444498,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T19:44:27.807Z","updated_at":"2025-07-03T09:08:48.615Z","avatar_url":"https://github.com/BAndysc.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LibProtoZeroSharp\n\nThis is an experimental highly efficient, but limited feature-set protobuf encoder/decoder designed for C#. While Protobuf is straightforward to use, it often incurs extra allocations and performance overhead. LibProtoZeroSharp aims to minimize these drawbacks by offering a zero-additional-alloc solution (allocations only for the buffer for the message itself).\n\nInspired by [ProtoZero](https://github.com/mapbox/protozero), a custom C++ protobuf encoder/decoder. LibProtoZeroSharp is a fully managed C# version.\n\nPlease note this is an experimental implementation with limited protobuf feature set.\n\n## Benchmarks\n\nTime for some benchmarks!\n\n### Encoding\n\n* CanonicalProto - The official, object-oriented protobuf implementation for C#\n* ProtoZeroNative - [ProtoZero](https://github.com/mapbox/protozero) compiled with -O3 flags for comparison\n* Protobuf.Net - [Protobuf.Net](https://github.com/protobuf-net/protobuf-net) - alternative protobuf implementation for C#\n* **ProtoZeroSharp** - This project - a struct based, zero-alloc implementation~~~~\n* PerfectSerializer - an \"ideal\" serializer that just copies the data to the output buffer\n\nTested on Apple M2 Pro 12C\n\n```\n| Method            | Mean       | Error     | StdDev    | Ratio | RatioSD | Gen0       | Gen1       | Gen2      | Allocated   | Alloc Ratio |\n|-------------------|-----------:|----------:|----------:|------:|--------:|-----------:|-----------:|----------:|------------:|------------:|\n| CanonicalProto    | 336.232 ms | 1.4800 ms | 1.3844 ms | 1.000 |    0.00 | 21000.0000 | 11000.0000 | 2000.0000 | 168996436 B |       1.000 |\n| Protobuf.Net      | 115.580 ms | 0.2154 ms | 0.1909 ms | 0.344 |    0.00 |          - |          - |         - |      1331 B |       0.000 |\n| ProtoZeroNative   |  99.940 ms | 0.9738 ms | 0.8632 ms | 0.297 |    0.00 |          - |          - |         - |       123 B |       0.000 |\n| PerfectSerializer |   1.295 ms | 0.0045 ms | 0.0042 ms | 0.004 |    0.00 |          - |          - |         - |         1 B |       0.000 |\n| ProtoZeroSharp    |  49.215 ms | 0.5472 ms | 0.5118 ms |     ? |       ? |          - |          - |         - |        67 B |           ? |\n```\n\nThe results are more than promising. Message writing is 7 times faster than the official implementation and allocates zero additional bytes compared to 168 MB in the official implementation! (For reference, the final encoded message is 52 MB). Interestingly, this even outperforms the ProtoZero C++ implementation.\n\nHowever, it is important to add that the most of the cost in canonical implementation comes from the message creation itself. The writing in my tests took around 100 ms (still twice as slow) and did not allocate additional memory. But it doesn't really matter because with official Protobuf you are forced to create the message anyway, so the cost is not avoidable.\n\nTo test native ProtoZero yourself, you need to download the submodule first `git submodule update --init --recursive` and run `native/build.sh` or `native/build.cmd` to build it first. On Windows, you have to execute this batch script from x64 VS Developer Console, as it invokes `cl.exe` command.\n\n\n### Decoding\n\nIt is possible to decode messages on the fly, without allocating any memory at all. This can be useful when you want to deserialize only some data. But that wouldn't be fair comparison against the original implementation which always deserializes the whole message.\n\nLuckily, LibProtoZeroSharp can also deserialize the whole message into generated structures and that's the benchmark below.\n\n```\n| Method              | Mean       | Error    | StdDev   | Median     | Ratio | RatioSD | Gen0        | Gen1       | Gen2      | Allocated  | Alloc Ratio |\n|-------------------- |-----------:|---------:|---------:|-----------:|------:|--------:|------------:|-----------:|----------:|-----------:|------------:|\n| CanonicalProto      | 2,888.2 ms | 57.59 ms | 94.63 ms | 2,944.9 ms |  1.00 |    0.00 | 156000.0000 | 80000.0000 | 5000.0000 | 1238.20 MB |        1.00 |\n| Protobuf.Net        | 2,733.0 ms | 51.37 ms | 40.11 ms | 2,743.3 ms |  0.97 |    0.04 | 132000.0000 | 68000.0000 | 4000.0000 | 1033.85 MB |        0.83 |\n| Protobuf.Net_Struct | 2,163.3 ms | 31.53 ms | 27.95 ms | 2,172.4 ms |  0.77 |    0.03 | 116000.0000 | 60000.0000 | 4000.0000 |  907.35 MB |        0.73 |\n| ProtoZeroSharp      |   408.9 ms |  2.25 ms |  1.88 ms |   408.5 ms |  0.15 |    0.01 |           - |          - |         - |  429.98 MB |        0.35 |\n```\n\nThis is an example 150 MB protobuf message. The official implementation allocates 3 times more memory and is 7 times slower. I've also compared the results with alternative Protobuf.net implementation. By default it is only a bit faster than the official one, however, unlike the official implementation, with Protobuf.net I was able to change some of the classes into structs, which gave some speed boost. It is still far from the performance of LibProtoZeroSharp.\n\n## Usage\n\n### Encoding\n\nCurrently, there are no generators for encoders; you have to construct the message manually:\n\n```csharp\nvar allocator = new ArenaAllocator();\nvar writer = new ProtoWriter(ref allocator);\n\nwriter.AddVarInt(Message.IdFieldNumber, 50);\nwriter.AddVarInt(Message.OtherFieldNumber, 1);\nwriter.AddVarInt(Message.OtherFieldNumber, 1); // we can add the same field, for repeated fields\nwriter.AddBytes(Message.StringFieldNumber, \"Hello world!\"u8); // we can add bytes (string) messages\nwriter.AddString(Message.String2FieldNumber, \"Hello world!\"); // or a string that will be encoded as utf-8, in place\n\n{ // additional scope added only for readability, it is not required\n    writer.StartSub(Message.SubMsgFieldNumber); // start a nested message\n\n    writer.AddVarInt(SubMessage.IdFieldNumber, 1); // now we are writing to a nested submessage\n    {\n        writer.StartSub(SubMessage.DoubleNestedFieldNumber); // nested messages may contain nested messages without any problems\n        writer.CloseSub(); \n    }\n    \n    writer.CloseSub(); // remember to close the submessage! Each Start needs a corresponding Close\n}\n\n{\n    writer.StartSub(Message.SubMsgFieldNumber); // nested messages can be repeated as well\n\n    writer.CloseSub();\n}\n\nusing var file = File.Open(\"output\");\nwriter.WriteTo(file); // save the results\n\nallocator.Dispose(); // Must be called, otherwise unmanaged memory is leaked!\n```\n\n\u003e ⚠️⚠️⚠️ You may **NOT** use `ProtoWriter` after disposing the allocator! ⚠️⚠️⚠️\n\n### Decoding\n\nDecoding can be done manually or using generated classes. The manual way is useful if you want to parse data into your own structures:\n\n**Manually:**\n\n```csharp\nvar reader = new ProtoReader(data); // data is a ReadOnlySpan\u003cbyte\u003e with the encoded message\nwhile (reader.Next())\n{\n    switch (reader.Tag)\n    {\n        case 1:\n            var varint = reader.ReadVarInt(); // for int32, int64, uint32, uint64\n            break;\n        case 2:\n            var boolean = reader.ReadBool();\n            break;\n        case 3:\n            var text = reader.ReadString();\n            break;\n        case 4:\n            var byteSpan = reader.ReadBytes();\n            break;\n        case 5:\n            var single = reader.ReadFloat();\n            break;\n        case 6:\n            var subMessage = reader.ReadMessage();\n            while (subMessage.Next())\n            {\n                Console.WriteLine($\"Tag: {subMessage.Tag} Type: {subMessage.WireType}\");\n                subMessage.Skip();\n            }\n            break;\n        default:\n            reader.Skip();\n            break;\n    }\n}\n```\n\n**Using generated classes:**\n\n1. Reference `ProtoZeroGenerator` and `LibProtoZeroSharp` packages.\n2. Add protobuf files into .csproj as AdditionalFiles tag (files must have .proto extension):\n```xml\n\u003cItemGroup\u003e\n    \u003cAdditionalFiles Include=\"structures.proto\" /\u003e\n\u003c/ItemGroup\u003e\n```\n3. Use the following code to read a message into a generated structure:\n```csharp\nvar allocator = new ArenaAllocator();\nvar reader = new ProtoReader(bytes);\n\nvar generatedStructure = new GeneratedStructure(); // this is the name of your protobuf message you want to deserialize\ngeneratedStructure.Read(ref reader, ref allocator);\n\n// you can do whatever you want with generatedStructure here\n\nallocator.Dispose(); // Must be called, otherwise unmanaged memory is leaked!\n// do not use generatedStructure after disposing the allocator!\n```\n\n\u003e ⚠️⚠️⚠️ You may **NOT** use the structure after disposing the allocator! Including any nested structures! ⚠️⚠️⚠️\n\u003e \n\u003e Decoded structure uses memory allocated by the allocator. When the allocator is disposed, the memory is freed and the structure becomes invalid.\n\u003e You can do whatever you want with the structures as long as the allocator is alive.\n\n\n## A note about the unsafe code\n\nThis library is faster than the official implementation because it uses unsafe code. This is a trade-off between performance and safety. As long as you follow the rules, you should be safe:\n1. Dispose the allocator when you are done with it, to avoid memory leaks. Without calling dispose, the memory will **never** be freed.\n2. **Do not use** the writer after disposing the allocator. When using `LibProtoZeroSharp.Debug` you will get managed exceptions, but in release mode, you will get segmentation faults!\n3. **Do not use** the generated structures after disposing the allocator. No matter if you use debug or release mode, you will get segmentation faults!\n4. You may copy the generated structures, but remember that they are just a shallow copy. And you may not use them after disposing the allocator.\n5. **Do not copy** the allocator! It is a struct and copying it will lead to bugs. Always pass it by `ref`.\n\n## Nuget\n\nThis library is available on Nuget:\n\n```\n\u003cPackageReference Include=\"ProtobufZeroSharp\" Version=\"0.5.0\" /\u003e\n```\n\nThis is a Release-mode library with commented out safety checks, so that it is faster, but also potential misusage (i.e. ReadMessage when the current tag is a varint) may lead to segmentation faults. Include `ProtobufZeroSharp.Debug` package for additional safety checks.\n\n```\n\u003cPackageReference Include=\"ProtobufZeroSharp.Debug\" Version=\"0.5.0\" /\u003e\n```\n\nYou can also include those packages depending on your configuration:\n\n```\n  \u003cItemGroup Condition=\"'$(Configuration)' == 'Debug'\"\u003e\n    \u003cPackageReference Include=\"ProtobufZeroSharp.Debug\" Version=\"0.5.0\" /\u003e\n  \u003c/ItemGroup\u003e\n\n  \u003cItemGroup Condition=\"'$(Configuration)' != 'Debug'\"\u003e\n    \u003cPackageReference Include=\"ProtobufZeroSharp\" Version=\"0.5.0\" /\u003e\n  \u003c/ItemGroup\u003e\n```\n\nFor the source generator, include \n```\n\u003cPackageReference Include=\"ProtobufZero.Generator\" Version=\"0.5.0.1\" /\u003e\n```\n\nAnd add your .proto files to the project as AdditionalFiles.\n```xml\n\u003cItemGroup\u003e\n    \u003cAdditionalFiles Include=\"structures.proto\" /\u003e\n\u003c/ItemGroup\u003e\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbandysc%2Flibprotobufzerosharp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbandysc%2Flibprotobufzerosharp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbandysc%2Flibprotobufzerosharp/lists"}