{"id":22369895,"url":"https://github.com/firmanmm/go-smc","last_synced_at":"2026-04-29T08:02:11.817Z","repository":{"id":57546089,"uuid":"287033166","full_name":"firmanmm/go-smc","owner":"firmanmm","description":"Simple Message Codec Golang Implementation","archived":false,"fork":false,"pushed_at":"2021-02-07T14:41:10.000Z","size":175,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-31T19:40:09.604Z","etag":null,"topics":["data-structures","encoder-decoder","golang","message","message-codec"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/firmanmm.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-12T14:18:53.000Z","updated_at":"2021-02-07T14:31:06.000Z","dependencies_parsed_at":"2022-09-05T10:50:40.915Z","dependency_job_id":null,"html_url":"https://github.com/firmanmm/go-smc","commit_stats":null,"previous_names":["firmanmm/gosmc"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firmanmm%2Fgo-smc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firmanmm%2Fgo-smc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firmanmm%2Fgo-smc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firmanmm%2Fgo-smc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/firmanmm","download_url":"https://codeload.github.com/firmanmm/go-smc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245690250,"owners_count":20656556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-structures","encoder-decoder","golang","message","message-codec"],"created_at":"2024-12-04T19:29:39.360Z","updated_at":"2026-04-29T08:02:06.781Z","avatar_url":"https://github.com/firmanmm.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Go Simple Message Codec (SMC)\n\n### Preface\nThis is a simple message codec `rewritten` based on version that is used on my `sync-mq` library. I decided to share it with you guys since i think it's ready to be used by the public. Feel free to improve this repository. I don't really like using solution that requires me to generate code in each change so here it goes.\n\n## About\nWell this is a message codec. It's mainly used to pack and unpack together a bunch of data structure into an array of byte to be transported via network. You can think of it like json marshal and unmarshal but friendlier to machine instead of human. This is a work in progress version that may change in the future. However, if you want to use it right now, i suggest you to fork this project so there is no breaking change in the future. (PS: I do it often as i see fit, so I highly recommend that). \n\n## Example\nPlease see example folder and test file to know more about how to use it\n```go\npackage main\n\nimport (\n\t\"crypto/sha512\"\n\t\"fmt\"\n\t\"log\"\n\n\t\"github.com/firmanmm/gosmc\"\n)\n\ntype Organism struct {\n\tName    string\n\tAge     uint\n\tSpecies string\n}\n\ntype ParentOrganism struct {\n\tName         string\n\tAge          uint\n\tSpecies      string\n\tActive       bool\n\tPassive      bool\n\tWeight       float64\n\tFingerprint  []byte\n\tChild        Organism\n\tPointerChild *Organism\n\tPayload      map[interface{}]interface{}\n}\n\nfunc main() {\n\tfingerprint := sha512.Sum512([]byte(\"A Fingerprint\"))\n\n\tdata := ParentOrganism{\n\t\tName:        \"Rendoru\",\n\t\tAge:         22,\n\t\tSpecies:     \"Human\",\n\t\tActive:      true,\n\t\tPassive:     false,\n\t\tWeight:      172.2,\n\t\tFingerprint: fingerprint[:],\n\t\tChild: Organism{\n\t\t\tName:    \"Doru\",\n\t\t\tAge:     1,\n\t\t\tSpecies: \"Digital Or Unknown\",\n\t\t},\n\t\tPointerChild: \u0026Organism{\n\t\t\tName:    \"Ren\",\n\t\t\tAge:     1,\n\t\t\tSpecies: \"Digital\",\n\t\t},\n\t\tPayload: map[interface{}]interface{}{\n\t\t\t\"A Key\":    \"This is a key\",\n\t\t\t\"Number\":   int32(12345),\n\t\t\t1000:       1234.56789012345678901234567890,\n\t\t\tuint(1000): 8765, //Look the same with previous key, but beware, its a different data type\n\t\t\t\"1000\":     \"THis is a string\",\n\t\t},\n\t}\n\t//Lets just use our standard implementation\n\tcodec := gosmc.NewSimpleMessageCodec()\n\tencoded, err := codec.Encode(data)\n\tif err != nil {\n\t\tlog.Fatalln(err.Error())\n\t}\n\t//Do Some random stuff until finish\n\n\t//Lets try to decode the encoded data\n\tdecodedData, err := codec.Decode(encoded)\n\tif err != nil {\n\t\tlog.Fatalln(err.Error())\n\t}\n\t//Since the underlying data is struct it is need to be accessed as map[string]interface{}, see conversion table\n\tcastedDecodedData := decodedData.(map[string]interface{})\n\tfmt.Println(\"This is original data\")\n\tfmt.Println(data) //OUTPUT : {Rendoru 22 Human true false 172.2 [152 77 121 122 155 37 109 58 12 79 188 10 231 17 219 248 232 113 251 119 237 205 193 131 23 73 134 70 240 208 100 84 245 35 9 23 43 145 216 219 102 98 216 246 250 44 157 223 130 103 222 194 7 72 19 0 244 218 107 123 93 199 9 120] {Doru 1 Digital Or Unknown} 0xc00006dc20 map[1000:1234.5678 1000:THis is a string A Key:This is a key Number:12345 1000:8765]}\n\tfmt.Println(\"This is decoded data\")\n\tfmt.Println(decodedData) //OUTPUT : map[Active:true Age:22 Child:map[Age:1 Name:Doru Species:Digital Or Unknown] Fingerprint:[152 77 121 122 155 37 109 58 12 79 188 10 231 17 219 248 232 113 251 119 237 205 193 131 23 73 134 70 240 208 100 84 245 35 9 23 43 145 216 219 102 98 216 246 250 44 157 223 130 103 222 194 7 72 19 0 244 218 107 123 93 199 9 120] Name:Rendoru Passive:false Payload:map[1000:1234.5678 1000:THis is a string A Key:This is a key Number:12345 1000:8765] PointerChild:map[Age:1 Name:Ren Species:Digital] Species:Human Weight:172.2]\n\tfmt.Println(\"This is example of accessing decoded data\")\n\tfmt.Printf(\"Field: %s content: %s\\n\", \"Name\", castedDecodedData[\"Name\"].(string)) //OUTPUT : Field: Name content: Rendoru\n\t//Since the underlying data is struct it is need to be accessed as map[string]interface{}, see conversion table\n\tchild := castedDecodedData[\"Child\"].(map[string]interface{})\n\tfmt.Printf(\"Child Field: %s content: %s\\n\", \"Name\", child[\"Name\"].(string)) //OUTPUT : Child Field: Name content: Doru\n\t//Since the underlying data is map it is need to be accessed as map[interface{}]interface{}, see conversion table\n\n\tpayload := castedDecodedData[\"Payload\"].(map[interface{}]interface{})\n\t//Lets access the \"1000\" int key in the payload, beware of precision lost when encoding float64\n\tfmt.Printf(\"Payload Key: %d content: %.40f, original %.40f\\n\", 1000, payload[1000].(float64), 1234.56789012345678901234567890) //OUTPUT : Payload Key: 1000 content: 1234.5678901234568911604583263397216796875000, original 1234.5678901234568911604583263397216796875000\n\t//BEWARE : This behaviour is only available when using Pure implementation and not the Jsoniter one,\n\t//Its becuase SMC will store their original data type in the encoded form\n\t//Lets now access the \"1000\" uint key in the payload and see the magic stuff\n\tfmt.Printf(\"Payload Key: %d content: %d\\n\", uint(1000), payload[uint(1000)].(int)) //OUTPUT : Payload Key: 1000 content: 8765\n\t//Lets now access the \"1000\" string key in the payload and see the magic stuff\n\tfmt.Printf(\"Payload Key: %s content: %s\\n\", \"1000\", payload[\"1000\"].(string)) //OUTPUT : Payload Key: 1000 content: THis is a string\n}\n\n```\n\nOr if you prefer manual write and read structure then you can find it here [example/manual](example/manual/readme.md)\n\n## Conversion Table\nSMC will convert certain data type to another data type in the process. If you decode something, please use the highest available data type that is compatible with that type.\n| Type | Implementation | Converted |\n| :------------- | :------------- | :---------- |\n| bool | Any | bool |\n| []byte | Pure, Universal | []byte |\n| []byte | Jsoniter | string |\n| string | Any | string |\n| int, int8, int16, int32, int64 | Any | int |\n| uint, uint8, uint16, uint32, uint64 | Pure | uint |\n| uint, uint8, uint16, uint32, uint64 | Universal, Jsoniter | int |\n| float32, float64 | Any | float64 |\n| []Any (Except []byte) | Any | []interface{} |\n| map[Any]Any | Pure, Universal | map[interface{}]interface{} |\n| map[Any]Any | Jsoniter | map[string]interface{} |\n| Struct | Any | map[string]interface{} |\n| *Any | Any | Dereferenced and follow conversion table |\n\nAlso, since `Jsoniter` convert all map keys to string, you simply can't do what the example shows you. That means `int(1000)`, `uint(1000)`, and \"1000\" are actually the same, and the behaviour is unknown. This same behaviour also happen when using `universal` encoder but it will only happen for `int(1000)` and `uint(1000)`, this because the `universal` encoder attempt to encode to the most commonly supported data type. An example of unsupported data type in other proggramming langguage is `uint` where it doesn't exist in `java` or `python`. If you want safety, then you can use the pure implementation if you need to deal with that kind of key.\n\n### Conversion Optimization\nPlease refer to [example/optimization](example/optimization/readme.md) for conversion optimization.\n\n## Benchmark\nWell this is the comparison of `smc` against `json`, `jsoniter` and `smc backed with jsoniter` :\n\n```\ngoos: windows\ngoarch: amd64\npkg: github.com/firmanmm/gosmc\nBenchmarkArrayOfByteJson-8                    \t   10000\t    108515 ns/op\t   28791 B/op\t       6 allocs/op\nBenchmarkArrayOfByteJsoniter-8                \t   27824\t     43265 ns/op\t   40897 B/op\t       5 allocs/op\nBenchmarkArrayOfByteSMC-8                     \t  402747\t      2849 ns/op\t   10368 B/op\t       6 allocs/op\nBenchmarkArrayOfByteSMCWithJsoniter-8         \t  414871\t      2897 ns/op\t   10368 B/op\t       6 allocs/op\nBenchmarkArrayOfByteMsgpack-8                 \t  376017\t      3466 ns/op\t   10704 B/op\t       9 allocs/op\nBenchmarkNestedArrayOfByteJson-8              \t    1102\t   1089359 ns/op\t  309311 B/op\t      43 allocs/op\nBenchmarkNestedArrayOfByteJsoniter-8          \t    2558\t    465842 ns/op\t  452246 B/op\t     142 allocs/op\nBenchmarkNestedArrayOfByteSMC-8               \t   19771\t     60885 ns/op\t  121610 B/op\t     316 allocs/op\nBenchmarkNestedArrayOfByteSMCWithJsoniter-8   \t   19378\t     60738 ns/op\t  122194 B/op\t     316 allocs/op\nBenchmarkNestedArrayOfByteMsgpack-8           \t   12019\t     99748 ns/op\t  272889 B/op\t      13 allocs/op\nBenchmarkInterfaceMapJsoniter-8               \t  261379\t      4883 ns/op\t     775 B/op\t      25 allocs/op\nBenchmarkInterfaceMapSMC-8                    \t  445605\t      2890 ns/op\t     680 B/op\t      29 allocs/op\nBenchmarkInterfaceMapSMCWithJsoniter-8        \t  249328\t      4963 ns/op\t     905 B/op\t      28 allocs/op\nBenchmarkInterfaceMapMsgPack-8                \t  141552\t      8603 ns/op\t    1251 B/op\t      43 allocs/op\nBenchmarkDeepInterfaceMapJsoniter-8           \t    2614\t    543717 ns/op\t   80994 B/op\t    2625 allocs/op\nBenchmarkDeepInterfaceMapSMC-8                \t    4010\t    308331 ns/op\t   70561 B/op\t    3231 allocs/op\nBenchmarkDeepInterfaceMapSMCWithJsoniter-8    \t    2504\t    478350 ns/op\t   89413 B/op\t    2630 allocs/op\nBenchmarkStringJson-8                         \t  132218\t      9043 ns/op\t    2385 B/op\t       5 allocs/op\nBenchmarkStringJsoniter-8                     \t  279594\t      4529 ns/op\t    2210 B/op\t       4 allocs/op\nBenchmarkStringSMC-8                          \t 1000000\t      1062 ns/op\t    2257 B/op\t       7 allocs/op\nBenchmarkStringSMCWithJsoniter-8              \t 1000000\t      1063 ns/op\t    2257 B/op\t       7 allocs/op\nBenchmarkStringMsgpack-8                      \t  707775\t      1729 ns/op\t    3648 B/op\t      11 allocs/op\nBenchmarkListStringJson-8                     \t    1398\t    856033 ns/op\t  225638 B/op\t     213 allocs/op\nBenchmarkListStringJsoniter-8                 \t    2934\t    418458 ns/op\t  238283 B/op\t     312 allocs/op\nBenchmarkListStringSMC-8                      \t   12960\t     92543 ns/op\t  230898 B/op\t     506 allocs/op\nBenchmarkListStringSMCWithJsoniter-8          \t   13046\t     93593 ns/op\t  231428 B/op\t     506 allocs/op\nBenchmarkListStringMsgpack-8                  \t    9254\t    133565 ns/op\t  387025 B/op\t     116 allocs/op\nBenchmarkListOfMapJsoniter-8                  \t    2733\t    453001 ns/op\t   82771 B/op\t    2511 allocs/op\nBenchmarkListOfMapSMC-8                       \t    5012\t    275604 ns/op\t   67338 B/op\t    2906 allocs/op\nBenchmarkListOfMapSMCWithJsoniter-8           \t    2557\t    481477 ns/op\t   90300 B/op\t    2706 allocs/op\nBenchmarkListOfMapMsgpack-8                   \t    1502\t    813546 ns/op\t   87564 B/op\t    3516 allocs/op\nBenchmarkStructJson-8                         \t  138300\t      8895 ns/op\t    2754 B/op\t      48 allocs/op\nBenchmarkStructJsoniter-8                     \t  156184\t      7935 ns/op\t    2787 B/op\t      58 allocs/op\nBenchmarkStructSMC-8                          \t  130779\t      8825 ns/op\t    3010 B/op\t     101 allocs/op\nBenchmarkStructSMCWithJsoniter-8              \t  141243\t      8487 ns/op\t    3173 B/op\t      63 allocs/op\nBenchmarkStructMsgpack-8                      \t  240477\t      5183 ns/op\t    1216 B/op\t      31 allocs/op\nPASS\nok  \tgithub.com/firmanmm/gosmc\t51.327s\n```\n\nComparing the output generated and we get :\n```\n=== RUN   TestSizeComparison\n--- PASS: TestSizeComparison (0.02s)\n    message_codec_test.go:298: Size Comparison :\n    message_codec_test.go:299: Jsoniter : 418001\n    message_codec_test.go:300: SMC : 365004\n    message_codec_test.go:301: Msgpack : 349003\nPASS\nok      github.com/firmanmm/gosmc       0.539s\n```\nAnd if we add slice of byte in to the mix then we get : \n```\n=== RUN   TestSizeComparisonWithArrayOfByte\n--- PASS: TestSizeComparisonWithArrayOfByte (0.02s)\n    message_codec_test.go:333: Size Comparison : \n    message_codec_test.go:334: Jsoniter : 523001  \n    message_codec_test.go:335: SMC : 446004\n    message_codec_test.go:336: Msgpack : 427003\nPASS\nok      github.com/firmanmm/gosmc       0.524s\n```\nWell, it clearly gives lower output size compared to `jsoniter` with or without having `slice of byte` in the map entry. However `Msgpack` provides much better size compared to `smc`\n\nAs you can see. This Simple Message Codec provides higher throughput in certain usecase compared to `jsoniter`, `json` and `msgpack`. However, you can also see that this message codec may also take higher memory compared to `jsoniter`. So pick your choice between speed and memory.\n\n## Behavioural Note\nWhen using `jsoniter`, the decoded `map` version will be in the `map[string]interface{}` format where all keys will be converted into `string`. It also apply to codec instantiated with `NewSimpleMessageCodecWithJsoniter`. \n\nWhen using `NewSimpleMessageCodec` or `pure` implementation, the decoded `map` will become `map[interface{}]interface{}` and will maintain their original data type. So `int` key will remain `int` and not converted to `string` like `jsoniter`.\n\n## Todo\n- Improve Struct performance (Need to be better than jsoniter)\n- Make more example\n- Need to try using Unsafe Operation (prefer not to)\n- Make code generator since there are manual encode and decode support\n\n## Note\nI highly recommend that you use `jsoniter` or `msgpack` as that is more battle tested than this. Also `jsoniter` is easier to read than this (Good luck if you want to read the output of SMC).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirmanmm%2Fgo-smc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffirmanmm%2Fgo-smc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirmanmm%2Fgo-smc/lists"}