{"id":13622995,"url":"https://github.com/inkeliz/karmem","last_synced_at":"2025-04-04T15:09:35.373Z","repository":{"id":40405546,"uuid":"491185405","full_name":"inkeliz/karmem","owner":"inkeliz","description":"Karmem is a fast binary serialization format, faster than Google Flatbuffers and optimized for TinyGo and WASM.","archived":false,"fork":false,"pushed_at":"2023-03-03T14:05:01.000Z","size":8232,"stargazers_count":660,"open_issues_count":38,"forks_count":29,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-28T14:08:04.001Z","etag":null,"topics":["assemblyscript","c","csharp","dotnet","dotnet7","go","golang","odin-lang","random-access","serialization","swift","webassembly","zig"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/inkeliz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-05-11T16:13:34.000Z","updated_at":"2025-03-21T13:06:55.000Z","dependencies_parsed_at":"2024-01-14T05:07:54.602Z","dependency_job_id":"f9e1e9e8-427d-421e-a3ed-48ba6864786d","html_url":"https://github.com/inkeliz/karmem","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inkeliz%2Fkarmem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inkeliz%2Fkarmem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inkeliz%2Fkarmem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inkeliz%2Fkarmem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/inkeliz","download_url":"https://codeload.github.com/inkeliz/karmem/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247198463,"owners_count":20900080,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["assemblyscript","c","csharp","dotnet","dotnet7","go","golang","odin-lang","random-access","serialization","swift","webassembly","zig"],"created_at":"2024-08-01T21:01:26.754Z","updated_at":"2025-04-04T15:09:35.353Z","avatar_url":"https://github.com/inkeliz.png","language":"Go","readme":"# KARMEM \n[![builds.sr.ht status](https://builds.sr.ht/~inkeliz/karmem/commits.svg)](https://builds.sr.ht/~inkeliz/karmem/commits?)  [![Coverage Status](https://coveralls.io/repos/github/inkeliz/karmem/badge.svg?branch=master)](https://coveralls.io/github/inkeliz/karmem?branch=master) [![Go Report Card](https://goreportcard.com/badge/karmem.org)](https://goreportcard.com/report/karmem.org) ![GitHub license](https://badgen.net/github/license/inkeliz/karmem)\n\nKarmem is a fast binary serialization format. The priority of Karmem is to be\neasy to use while been fast as possible. It's optimized to take Golang and\nTinyGo's maximum performance and is efficient for repeatable reads, reading\ndifferent content of the same type. Karmem demonstrates to be ten times faster\nthan Google Flatbuffers, with the additional overhead of bounds-checking\nincluded.\n\n\u003e ⚠️ Karmem still under development, the API is not stable. However, serialization-format itself is unlike to change and\n\u003e should remain backward compatible with older versions.\n\n# Contents\n\n- [🧐 Motivation](#Motivation)\n- [🧠 Usage](#Usage)\n- [🏃 Benchmark](#Benchmark)\n- [🌎 Languages](#Languages)\n- [📙 Schema](#Schema)\n    - Example\n    - Types\n    - Structs\n    - Enum\n- [🛠️ Generator](#Generator)\n- [🔒 Security](#Security)\n\n# Motivation\n\nKarmem was create to solve one single issue: make easy to transfer data between WebAssembly host and guest. While still\nportable for non-WebAssembly languages. We are experimenting with an \"event-command pattern\" between wasm-host and\nwasm-guest in one project, but sharing data is very expensive, and FFI calls are not cheap either. Karmem encodes once\nand shares the same content with multiple guests, regardless of the language, making it very efficient. Also, even using\nObject-API to decode, it's fast enough, and Karmem was designed to take advantage of that pattern, avoid allocations,\nand re-use the same struct for multiple data.\n\nWhy not use [Witx](https://github.com/jedisct1/witx-codegen)? It is good project and aimed to WASM, however it seems\nmore complex and defines not just data-structure, but functions, which I'm trying to avoid. Also, it is not intended to\nbe portable to non-wasm. Why not use [Flatbuffers](https://google.github.io/flatbuffers/)? We tried, but it's not fast\nenough and also causes panics due to the lack of bound-checking. Why not use [Cap'n'Proto](https://capnproto.org/)? It's\na good alternative but lacks implementation for Zig and AssemblyScript, which is top-priority, it also has more\nallocations and the generated API is harder to use, compared than Karmem.\n\n# Usage\n\nThat is a small example of how use Karmem.\n\n### Schema\n\n```go\nkarmem app @packed(true) @golang.package(`app`);  \n  \nenum SocialNetwork uint8 { Unknown; Facebook; Instagram; Twitter; TikTok; }  \n  \nstruct ProfileData table {  \n    Network  SocialNetwork;  \n    Username []char;  \n    ID       uint64;  \n}  \n  \nstruct Profile inline {  \n    Data ProfileData;  \n}  \n  \nstruct AccountData table {  \n    ID       uint64;  \n    Email    []char;  \n    Profiles []Profile;  \n}\n```\n\nGenerate the code using `go run karmem.org/cmd/karmem build --golang -o \"km\" app.km`.\n\n### Encoding\n\nIn order to encode, use should create an native struct and then encode it.\n\n```go\nvar writerPool = sync.Pool{New: func() any { return karmem.NewWriter(1024) }}\n\nfunc main() {\n\twriter := writerPool.Get().(*karmem.Writer)\n\n\tcontent := app.AccountData{\n\t\tID:    42,\n\t\tEmail: \"example@email.com\",\n\t\tProfiles: []app.Profile{\n\t\t\t{Data: app.ProfileData{\n\t\t\t\tNetwork:  app.SocialNetworkFacebook,\n\t\t\t\tUsername: \"inkeliz\",\n\t\t\t\tID:       123,\n\t\t\t}},\n\t\t\t{Data: app.ProfileData{\n\t\t\t\tNetwork:  app.SocialNetworkFacebook,\n\t\t\t\tUsername: \"karmem\",\n\t\t\t\tID:       231,\n\t\t\t}},\n\t\t\t{Data: app.ProfileData{\n\t\t\t\tNetwork:  app.SocialNetworkInstagram,\n\t\t\t\tUsername: \"inkeliz\",\n\t\t\t\tID:       312,\n\t\t\t}},\n\t\t},\n\t}\n\n\tif _, err := content.WriteAsRoot(writer); err != nil {\n\t\tpanic(err)\n\t}\n\n\tencoded := writer.Bytes()\n\t_ = encoded // Do something with encoded data\n\n\twriter.Reset()\n\twriterPool.Put(writer)\n}\n```\n\n### Reading\n\nInstead of decoding it to another struct, you can read some fields directly, without any additional decoding. In this\nexample, we only need the username of each profile.\n\n```go\nfunc decodes(encoded []byte) {\n\treader := karmem.NewReader(encoded)\n\taccount := app.NewAccountDataViewer(reader, 0)\n\n\tprofiles := account.Profiles(reader)\n\tfor i := range profiles {\n\t\tfmt.Println(profiles[i].Data(reader).Username(reader))\n\t}\n}\n```\n\nNotice: we use `NewAccountDataViewer`, any `Viewer` is just a Viewer, and doesn't copy the backend data. Some \nlanguages (C#, AssemblyScript) uses UTF-16, while Karmem uses UTF-8, in those cases you have some performance \npenalty.\n\n### Decoding\n\nYou can also decode it to an existent struct. In some cases, it's better if you re-use the same struct for multiples\nreads.\n\n```go\nvar accountPool = sync.Pool{New: func() any { return new(app.AccountData) }}\n\nfunc decodes(encoded []byte) {\n\taccount := accountPool.Get().(*app.AccountData)\n\taccount.ReadAsRoot(karmem.NewReader(encoded))\n\n\tprofiles := account.Profiles\n\tfor i := range profiles {\n\t\tfmt.Println(profiles[i].Data.Username)\n\t}\n\n\taccountPool.Put(account)\n}\n```\n\n# Benchmark\n\n### Flatbuffers vs Karmem\n\nUsing similar schema with Flatbuffers and Karmem. Karmem is almost 10 times faster than Google Flatbuffers.\n\n**Native (MacOS/ARM64 - M1):**\n\n```\nname               old time/op    new time/op    delta\nEncodeObjectAPI-8    2.54ms ± 0%    0.51ms ± 0%   -79.85%  (p=0.008 n=5+5)\nDecodeObjectAPI-8    3.57ms ± 0%    0.20ms ± 0%   -94.30%  (p=0.008 n=5+5)\nDecodeSumVec3-8      1.44ms ± 0%    0.16ms ± 0%   -88.86%  (p=0.008 n=5+5)\n\nname               old alloc/op   new alloc/op   delta\nEncodeObjectAPI-8    12.1kB ± 0%     0.0kB       -100.00%  (p=0.008 n=5+5)\nDecodeObjectAPI-8    2.87MB ± 0%    0.00MB       -100.00%  (p=0.008 n=5+5)\nDecodeSumVec3-8       0.00B          0.00B           ~     (all equal)\n\nname               old allocs/op  new allocs/op  delta\nEncodeObjectAPI-8     1.00k ± 0%     0.00k       -100.00%  (p=0.008 n=5+5)\nDecodeObjectAPI-8      110k ± 0%        0k       -100.00%  (p=0.008 n=5+5)\nDecodeSumVec3-8        0.00           0.00           ~     (all equal)\n```\n\n**WebAssembly on Wazero (MacOS/ARM64 - M1):**\n\n```\nname               old time/op    new time/op    delta\nEncodeObjectAPI-8    17.2ms ± 0%     4.0ms ± 0%  -76.51%  (p=0.008 n=5+5)\nDecodeObjectAPI-8    50.7ms ± 2%     1.9ms ± 0%  -96.18%  (p=0.008 n=5+5)\nDecodeSumVec3-8      5.74ms ± 0%    0.75ms ± 0%  -86.87%  (p=0.008 n=5+5)\n\nname               old alloc/op   new alloc/op   delta\nEncodeObjectAPI-8    3.28kB ± 0%    3.02kB ± 0%   -7.80%  (p=0.008 n=5+5)\nDecodeObjectAPI-8    3.47MB ± 2%    0.02MB ± 0%  -99.56%  (p=0.008 n=5+5)\nDecodeSumVec3-8      1.25kB ± 0%    1.25kB ± 0%     ~     (all equal)\n\nname               old allocs/op  new allocs/op  delta\nEncodeObjectAPI-8      4.00 ± 0%      4.00 ± 0%     ~     (all equal)\nDecodeObjectAPI-8      5.00 ± 0%      4.00 ± 0%  -20.00%  (p=0.008 n=5+5)\nDecodeSumVec3-8        5.00 ± 0%      5.00 ± 0%     ~     (all equal)\n```\n\n### Raw-Struct vs Karmem\n\nThe performance is nearly the same when comparing reading non-serialized data from a native struct and reading it from a\nkarmem-serialized data.\n\n**Native (MacOS/ARM64 - M1):**\n\n```\nname             old time/op    new time/op    delta\nDecodeSumVec3-8     154µs ± 0%     160µs ± 0%  +4.36%  (p=0.008 n=5+5)\n\nname             old alloc/op   new alloc/op   delta\nDecodeSumVec3-8     0.00B          0.00B         ~     (all equal)\n\nname             old allocs/op  new allocs/op  delta\nDecodeSumVec3-8      0.00           0.00         ~     (all equal)\n```\n\n### Karmem vs Karmem\n\nThat is an comparison with all supported languages.\n\n**WebAssembly on Wazero (MacOS/ARM64 - M1):**\n\n```\nname \\ time/op     result/wasi-go-km.out  result/wasi-as-km.out  result/wasi-zig-km.out  result/wasi-swift-km.out  result/wasi-c-km.out  result/wasi-odin-km.out  result/wasi-dotnet-km.out\nDecodeSumVec3-8               757µs ± 0%            1651µs ± 0%              369µs ± 0%               9145µs ± 6%            368µs ± 0%              1330µs ± 0%               75671µs ± 0% \nDecodeObjectAPI-8            1.59ms ± 0%            6.13ms ± 0%             1.04ms ± 0%              30.59ms ±34%           0.90ms ± 1%              4.06ms ± 0%              231.72ms ± 0% \nEncodeObjectAPI-8            3.96ms ± 0%            4.51ms ± 1%             1.20ms ± 0%               8.26ms ± 0%           1.03ms ± 0%              5.19ms ± 0%              237.99ms ± 0% \n\nname \\ alloc/op    result/wasi-go-km.out  result/wasi-as-km.out  result/wasi-zig-km.out  result/wasi-swift-km.out  result/wasi-c-km.out  result/wasi-odin-km.out  result/wasi-dotnet-km.out \nDecodeSumVec3-8              1.25kB ± 0%           21.75kB ± 0%             1.25kB ± 0%               1.82kB ± 0%           1.25kB ± 0%              5.34kB ± 0%              321.65kB ± 0% \nDecodeObjectAPI-8            15.0kB ± 0%           122.3kB ± 1%            280.8kB ± 1%              108.6kB ± 3%            1.2kB ± 0%              23.8kB ± 0%               386.5kB ± 0% \nEncodeObjectAPI-8            3.02kB ± 0%           58.00kB ± 1%             1.23kB ± 0%               1.82kB ± 0%           1.23kB ± 0%              8.91kB ± 0%              375.82kB ± 0% \n\nname \\ allocs/op   result/wasi-go-km.out  result/wasi-as-km.out  result/wasi-zig-km.out  result/wasi-swift-km.out  result/wasi-c-km.out  result/wasi-odin-km.out  result/wasi-dotnet-km.out \nDecodeSumVec3-8                5.00 ± 0%              5.00 ± 0%               5.00 ± 0%                32.00 ± 0%             5.00 ± 0%                6.00 ± 0%                 11.00 ± 0% \nDecodeObjectAPI-8              5.00 ± 0%              4.00 ± 0%               4.00 ± 0%                32.00 ± 0%             4.00 ± 0%                6.00 ± 0%                340.00 ± 0% \nEncodeObjectAPI-8              4.00 ± 0%              3.00 ± 0%               3.00 ± 0%                30.00 ± 0%             3.00 ± 0%                5.00 ± 0%                 40.00 ± 0% \n```\n\n# Languages\n\nCurrently, we have focus on WebAssembly, and because of that those are the languages supported:\n\n- AssemblyScript 0.20.16\n- C/Emscripten\n- C#/.NET 7\n- Golang 1.19/TinyGo 0.25.0\n- Odin\n- Swift 5.7/SwiftWasm 5.7\n- Zig 0.10\n\nSome languages still under development, and doesn't have any backward compatibility promise. We will try \nto keep up with the latest version. Currently, the API generated and the libraries should not consider stable.\n\n### Features\n\n| Features | Go/TinyGo | Zig | AssemblyScript | Swift | C | C#/.NET | Odin |\n| -- | -- | -- | -- | -- | -- | -- | -- |\n| Performance | Good | Excellent | Good | Poor | Excellent | Horrible | Good |\n| Priority | High | High | High | Low | High | Medium | Low |\n| **Encoding** | | | | | | | |\n| Object Encoding | ✔️ |✔️ |✔️ | ✔️ | ✔️| ✔️ | ✔️ |\n| Raw Encoding | ❌ |❌ | ❌| ❌ | ❌ | ❌ | ❌ |\n| Zero-Copy |❌ | ❌ |❌ | ❌ | ❌ | ❌ | ❌ |\n| **Decoding** | | | | | | | |\n| Object Decoding |✔️ |✔️ |✔️ | ✔️ |✔️ | ✔️ | ✔️|\n| Object Re-Use |✔️ |✔️ |✔️ | ❌ |✔️ |  ✔️|  ✔️ |\n| Random-Access |✔️ |✔️ |✔️ | ✔️ |✔️ | ✔️ |  ✔️|\n| Zero-Copy |✔️ | ✔️ |✔️ | ❌ |✔️ |  ✔️| ✔️|\n| Zero-Copy-String |✔️ | ✔️ |❌ | ✔️ |✔️ |  ❌| ✔️|\n| Native Array | ✔️ |✔️ |❌ | ❌ |✔️ |  ❌️| ✔️|\n\n# Schema\n\nKarmem uses a custom schema language, which defines structs, enums and types.\n\n### Example\n\nThe schema is very simple to understand and define:\n\n```go\nkarmem game @packed(true) @golang.package(`km`) @assemblyscript.import(`../../assemblyscript/karmem`);\n\nenum Team uint8 {Humans;Orcs;Zombies;Robots;Aliens;}\n\nstruct Vec3 inline {\n    X float32;\n    Y float32;\n    Z float32;\n}\n\nstruct MonsterData table {\n    Pos       Vec3;\n    Mana      int16;\n    Health    int16;\n    Name      []char;\n    Team      Team;\n    Inventory [\u003c128]byte;\n    Hitbox    [4]float64;\n    Status    []int32;\n    Path      [\u003c128]Vec3;\n}\n\nstruct Monster inline {\n    Data MonsterData;\n}\n\nstruct State table {\n    Monsters [\u003c2000]Monster;\n}\n```\n\n### Header:\n\nEvery file must begin with: `karmem {name} [@tag()];`. Other optional tags can be defined, as shown above, it's\nrecommended to use the `@packed(true)` option.\n\n### Types:\n\n**Primitives**:\n\n- Unsigned Integers:\n  `uint8`, `uint16`, `uint32`, `uint64`\n- Signed Integers:\n  `int8`, `int16`, `int32,` `int64`\n- Floats:\n  `float32`, `float64`\n- Boolean:\n  `bool`\n- Byte:\n  `byte`, `char`\n\nIt's not possible to define optional or nullable types.\n\n**Arrays**:\n\n- Fixed:\n  `[{Length}]{Type}` (example: `[123]uint16`, `[3]float32`)\n- Dynamic:\n  `[]{Type}` (example: `[]char`, `[]uint64`)\n- Limited:\n  `[\u003c{Length}]{Type}` (example: `[\u003c512]float64`, `[\u003c42]byte`)\n\nIt's not possible to have slice of tables or slices of enums or slice of slices. However, it's possible to wrap those\ntypes inside one inline-struct.\n\n### Struct:\n\nCurrently, Karmem has two structs types: inline and table.\n\n**Inline:**\nInline structs, as the name suggests, are inlined when used. That reduces the size and may improve the performance.\nHowever, it can't have their definition changed. In order words: you can't edit the field of one inline struct\nwithout breaking compatibility.\n\n```go\nstruct Vec3 inline {\n    X float32;\n    Y float32;\n    Z float32;\n}\n```\n\n*That struct is exactly the same of `[3]float32` and will have the same serialization result. Because of that, any\nchange of this struct (for instance, change it to `float64` or adding new fields) will break the compatibility.*\n\n**Tables:**\nTables can be used when backward compatibility matters. For example, tables can have new fields append at the bottom\nwithout breaking compatibility.\n\n```go\nstruct User table {\n    Name     []char;\n    Email    []char;\n    Password []char;\n}\n```\n\nLet's consider that you need another field... For tables, it's not an issue:\n\n```go\nstruct User table {\n    Name      []char;\n    Email     []char;\n    Password  []char;\n    Telephone []char;\n}\n```\n\nSince it's a table, you can add new fields at the bottom of the struct, and both versions are compatible between them. If\nthe message is sent to a client that doesn't understand the new field, it will be ignored. If one outdated client sends a\nmessage to a newer client, the new field will have the default value (0, false, empty string, etc).\n\n### Enums:\n\nEnums can be used as an alias to Integers type, such as `uint8`.\n\n```go\nenum Team uint8 {\n    Unknown;\n    Humans;\n    Orcs;\n    Zombies = 255;\n}\n```\n\nEnums must start with a zero value, the default value in all cases. If the value of any enum is omitted, it will use the\norder of enum as value.\n\n# Generator\n\nOnce you have a schema defined, you can generate the code. First, you need to `karmem` installed, get it from the\nreleases page or run it with go.\n\n```\nkarmem build --assemblyscript -o \"output-folder\" your-schema.km\n```\n\n*If you already have Golang installed, you can use `go karmem.org/cmd/karmem build --zig -o \"output-folder\" your-schema.km`\ninstead.*\n\n**Commands:**\n\n**`build`**\n\n- `--zig`: Enable generation for Zig\n- `--swift`: Enable generation for Swift/SwiftWasm\n- `--odin`: Enable generation for Odin\n- `--golang`: Enable generation for Golang/TinyGo\n- `--dotnet`: Enable generation for .NET\n- `--c`: Enable generation for C\n- `--assemblyscript`: Enable generation for AssemblyScript\n- `-o \u003cdir\u003e`: Defines the output folder\n- `\u003cinput-file\u003e`: Defines the input schema\n\n# Security\n\nKarmem is fast and is also aimed to be secure and stable for general usage.\n\n**Out Of Bounds**\n\nKarmem includes bounds-checking to prevent out-of-bounds reading and avoid crashes and panics. That is something that\nGoogle Flatbuffers doesn't have, and malformed content will cause panic. However, it doesn't fix all possible\nvulnerabilities.\n\n**Resource Exhaustion**\n\nKarmem allows one pointer/offset can be re-used multiple times in the same message. Unfortunately, that behaviour makes\nit possible for a short message to generate more extensive arrays than the message size. Currently, the only mitigation\nfor that issue is using Limited-Arrays instead of Arrays and avoiding Object-API decode.\n\n**Data Leak**\n\nKarmem doesn't clear the memory before encoding, that may leak information from the previous message or from system\nmemory itself. That can be solved using `@packed(true)` tag, as previously described. The `packed` tag will remove\npadding from the message, which will prevent the leak. Alternatively, you can clear the memory before encoding,\nmanually.\n\n# Limitations\n\nKarmem have some limitations compared to other serialization libraries, such as:\n\n**Maximum Size**\n\nSimilar to Google Protobuf and Google Flatbuffers, Karmem has a maximum size of 2GB. That is the maximum size of the entire\nmessage, not the maximum size of each array. This limitation is due to the fact that WASM is designed to be 32-bit,\nand the maximum size of 2GB seems adequate for the current needs. The current Writer doesn't enforce this limitation,\nbut reading a message that is bigger than 2GB will cause undefined behaviour.\n\n**Arrays of Arrays/Tables**\n\nKarmem doesn't support arrays of arrays or arrays of tables. However, it's possible to wrap those types inside one\ninline-struct, as mentioned above. That limitation was imposed to take advantage of native arrays/slice from the\nlanguage. Most languages encapsulates the pointer and the size of the array inside a struct-like, that requires \nthe size of each element to be known, consequently preventing arrays of items with variable size/strides.\n\n**UTF-8**\n\nKarmem only supports UTF-8 and doesn't support other encodings.\n\n","funding_links":[],"categories":["Packages","Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finkeliz%2Fkarmem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finkeliz%2Fkarmem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finkeliz%2Fkarmem/lists"}