{"id":34514095,"url":"https://github.com/koykov/decoder","last_synced_at":"2026-06-01T11:33:15.078Z","repository":{"id":57571867,"uuid":"283234456","full_name":"koykov/decoder","owner":"koykov","description":"Dynamic decoder of arbitrary data to Go structs.","archived":false,"fork":false,"pushed_at":"2025-12-11T19:39:07.000Z","size":404,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-25T15:30:08.407Z","etag":null,"topics":["decoders","dynamic","highload"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koykov.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-07-28T14:20:39.000Z","updated_at":"2025-12-11T19:39:12.000Z","dependencies_parsed_at":"2023-10-29T23:27:29.551Z","dependency_job_id":"7909207c-5ee0-407b-b722-2197ee780faf","html_url":"https://github.com/koykov/decoder","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/koykov/decoder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koykov%2Fdecoder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koykov%2Fdecoder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koykov%2Fdecoder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koykov%2Fdecoder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koykov","download_url":"https://codeload.github.com/koykov/decoder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koykov%2Fdecoder/sbom","scorecard":{"id":568748,"data":{"date":"2025-08-11","repo":{"name":"github.com/koykov/decoder","commit":"70acd012323caa812119b7e2ba8d4688998deb0d"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.3,"checks":[{"name":"Maintained","score":10,"reason":"26 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":0,"reason":"Found 0/7 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: license.md:0","Info: FSF or OSI recognized license: MIT License: license.md:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-20T15:44:10.592Z","repository_id":57571867,"created_at":"2025-08-20T15:44:10.592Z","updated_at":"2025-08-20T15:44:10.592Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33773774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-01T02:00:06.963Z","response_time":115,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["decoders","dynamic","highload"],"created_at":"2025-12-24T04:18:09.259Z","updated_at":"2026-06-01T11:33:15.041Z","avatar_url":"https://github.com/koykov.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Decoder\n\nDynamic decoders based on [inspector](https://github.com/koykov/inspector) framework\nand [vector parsers](https://github.com/koykov/vector).\n\n## Retrospective\n\nOne of the major problems we ran into was a necessity to convert tons of different response formats from external\nservices into an internal response format. The problem became harder due to new external services with their own response\nformats may appear at any time. Due to highload conditions, there is no way to use standard dynamic approaches like\nreflection - the convertation must work very fast, make zero allocations and support dynamic to avoid application deploys.\n\nThis package was developed as an answer to this challenge. It provides a possibility to describe decoding rules in\nGo-like meta-language with full dynamic support - registering new decoders (or edit an existing) may on the fly.\n\n## How it works\n\nDecoders are similar to [dyntpl](https://github.com/koykov/dyntpl) package in opposite - dyntpl makes a text from\nstructures but decoders parses text and assign data to structures.\n\nSimilar to dyntpl, decoding divides into two phases - parsing and decoding. The parsing phase builds from decoder's body\na tree (like AST) and registers it in decoders registry by unique name afterward. This phase isn't intended to be used in\nhighload conditions due to high pressure to cpu/mem. The second phase - decoding, against intended to use in highload.\n\nDecoding phase required a preparation to pass data to the decoder. There is a special object [Ctx](ctx.go), that collects\nvariables to use in decoder. Each variable must have three params:\n* unique name\n* data - anything you need to use in decoder\n* inspector type\n\nWhat is the inspector describes [here](https://github.com/koykov/inspector), but need an extra explanation of how it works\ntogether with decoders. In general, decoding problem sounds like \"grab an arbitrary data from one struct and write it\nto another struct as fast as it possible and with zero allocations\". The first part of the problem was solved in\n[dyntpl using inspectors](https://github.com/koykov/inspector/tree/master?tab=readme-ov-file#intro), and it was a good\ndecision to extend inspectors with possibility to write data to destination structs. Thus, the problem became like\n\"using one inspector, read data from the source struct and, using another inspector, write it to the destination struct\".\n\n## Usage\n\nThe typical usage of decoders looks like this:\n```go\npackage main\n\nimport (\n\t\"github.com/koykov/decoder\"\n\t\"github.com/koykov/inspector/testobj\"\n\t\"github.com/koykov/inspector/testobj_ins\"\n\t\"github.com/koykov/jsonvector\"\n)\n\nvar (\n\tdata     testobj.TestObject\n\tresponse = []byte(`{\"identifier\":\"xf44e\",\"person\":{\"full_name\":\"Marquis Warren\",\"status\":67},\"finance\":{\"balance\":\"164.5962\"\",\"is_active\":true}}`)\n\tdecBody  = []byte(`data.Id = resp.identifier\ndata.Name = resp.person.full_name\ndata.Status = resp.person.status|default(-1)\ndata.Finance.Balance = atof(resp.finance.balance)`)\n)\n\nfunc init() {\n\t// Parse decoder body and register it.\n\tdec, _ := decoder.Parse(decBody)\n\tdecoder.RegisterDecoderKey(\"myDecoder\", dec)\n}\n\nfunc main() {\n\t// Prepare response as vector object.\n\tvec := jsonvector.Acquire()\n\tdefer jsonvector.Release(vec)\n\t_ = vec.Parse(response)\n\n\tctx := decoder.AcquireCtx()\n\tdefer decoder.ReleaseCtx(ctx)\n\t\n\t// Prepare context.\n\tctx.SetVector(\"resp\", vec)\n\tctx.Set(\"data\", \u0026data, testobj_ins.TestObjectInspector{})\n\t// Execute the decoder.\n\terr := decoder.Decode(\"myDecoder\", ctx)\n\tprintln(err)                  // nil\n\tprintln(data.Id)              // xf44e\n\tprintln(data.Name)            // []byte(\"Marquis Warren\")\n\tprintln(data.Status)          // 67\n\tprintln(data.Finance.Balance) // 164.5962\n}\n```\n\nContent of init() function should be executed once (or periodically on the fly from some source, eg DB).\n\nContent of main() function is how to use decoders in a general way in highload.\n\n## Syntax\n\nDecoders inherits Go syntax, but provides an extra features like modifiers and coalesce operator (see below).\n\n### Assigning\n\nThe base decoding operation is assigning the data from source variable to destination variable. The syntax is typical\n`lvalue.Field1 = rvalue.Field2`. From [example](#usage):\n```\ndata.Id = resp.identifier\ndata.Name = resp.person.full_name\n```\nwhere `data` represents `lvalue` (source variable) and `resp` - `rvalue` (destination variable).\n\n### Coalesce operator\n\nDecoders provide a possibility to read one-of-many fields when read nested fields from struct:\n```\ndst.Field = src.Nested.{Field1|Field2|Field3|...}\n```\nThe first non-empty field between curly brackets will be read as data to assign. This syntax sugar allows to avoid tons\nof comparisons or build chain of `default` modifiers. Example of usage see [here](testdata/decoder/decoder4.dec).\n\n### Modifiers\n\nDecoders supports user-defined modifiers, which applies additional logic to data before assigning. It may be helpful for\nedge cases (no data, conditional assignment, etc.). Modifiers usage syntax is typical - after source of data, using `|`\nsymbol, modifier calls as function call:\n```\ndst.Field = src.Field|modifier0(arg0, arg1, ...)|modifier1(arg0, arg1, ...)|...\n```\n\nExample:\n```\ndata.Status = src.Nested.Blocked|ifThenElse(src.Nested.State, -1)\n                                ^ simple modifier\ndata.Name = src.FullName|default(\"N\\D\")|toUpper()\n                        ^ first mod    ^ second modifier\n```\n\nModifiers may collect in chain with variadic length. In that case, each modifier will take to input the result of\nprevious modifier. Each modifier may take an arbitrary count of arguments.\n\nModifier is a Go function with a special signature:\n```go\ntype ModFn func(ctx *Ctx, buf *any, val any, args []any) error\n```\nwhere:\n* ctx - context of the decoder\n* buf - pointer to return the result\n* val - value to pass to the modifier (value of `varName` in example `varName|modifier()`) \n* args - list of all arguments\n\nYou should register your modifier using one of the functions:\n* `RegisterModFn(name, alias string, mod ModFn)`\n* `RegisterModFnNS(namespace, name, alias string, mod ModFn)`\n\nThey are the same, but NS version allows to specify the namespace of the function. In that case, you should specify namespace\nin modifiers call:\n```\ndst.Field = src.Field|namespaceName::modifier()\n```\n\n### Conditions\n\nDecoders supports classic syntax of conditions:\n```\nif leftVar [==|!=|\u003e|\u003e=|\u003c|\u003c=] rightVar {\n    true branch\n} else {\n    false branch\n}\n```\n\nExamples: [1](testdata/parser/cond.dec), [2](testdata/parser/cond_else.dec), [3](testdata/parser/condOK.dec).\n\nDecoders can't handle complicated conditions containing more than one comparison, like:\n```\nif user.Id == 0 || user.Finance.Balance == 0 {...}\n```\nIn the future this problem will be solved, but now you can make nested conditions or use conditions helpers - functions\nwith signature:\n```go\ntype CondFn func(ctx *Ctx, args []any) bool\n```\n, where you may pass an arbitrary amount of arguments and these functions will return bool to choose the right execution branch.\nThese functions are user-defined, like modifiers, and you may write your own and then register it using one of the functions:\n```go\nfunc RegisterCondFn(name string, cond CondFn)\nfunc RegisterCondFnNS(namespace, name string, cond CondFn) // namespace version\n```\n\nThen condition helper will be accessible inside decoders and you may use it using the name:\n```\nif helperName(user.Id, user.Finance.Balance) {...}\n```\n\nDecoders supports ternary operator for most primitive cases of assigning. Conditions like this:\n```\nif x.a == 123 {\n    dst.Field1 = src.Field2\n} else {\n    dst.Field1 = \"N/D\n}\n```\nmay be shortener using ternary operator:\n```\ndst.Field1 = x.a == 123 ? src.Field2 : \"N/D\"\n```\n\nCondition helpers also supported:\n```\nobj.Id = testns::check(obj.Id, 15.123, \"foobar\", false) ? 225 : src.{status|state}\n```\n\n#### switch\n\nFor multiple conditions, you can use `switch` statement, examples:\n* [classic switch](testdata/parser/switch.dec)\n* [no-condition switch](testdata/parser/switch_no_cond.dec)\n* [no-condition switch with helpers](testdata/parser/switch_no_cond_helper.dec)\n\n### Loops\n\nDecoders supports both types of loops:\n* counter loops, like `for i:=0; i\u003c5; i++ {...}`\n* range-loop, like `for k, v := range obj.Items {...}`\n\nEdge cases like `for k \u003c 2000 {...}` or `for ; i \u003c 10 ; {...}` isn't supported.\nAlso, you can't make an infinite loop by using `for {...}`.\n\n#### Loop breaking\n\nDecoders supports default instructions `break` and `continue` to break loop/iteration, example:\n```\nfor _, v := list\n  if v.ID == 0 {\n    continue\n  }\n  if v.Status == -1 {\n    break\n  }\n}\n```\n\nThese instructions works as intended, but they required condition a wrapper and that's bulky. Therefore, decoders provide\ncombined `break if` and `continue if` that works the same:\n```\nfor _, v := list {\n  continue if v.ID == 0\n  break if v.Status == -1\n}\n```\n\nBoth examples are equal, but the second is more compact.\n\n#### Lazy breaks\n\nImagine the case - you've decided in the middle of iteration that loop requires a break, but the iteration must finish its\nwork the end. For that case, decoders supports special instruction `lazybreak`. It breaks the loop but allows current\niteration works till the end.\n\n### Extensions\n\nDecoders may be extended by including modules in the project. Currently supported modules:\n* [decoder_vector](https://github.com/koykov/decoder_vector) provide support of vector parsers.\n* [decoder_i18n](https://github.com/koykov/decoder_legacy) allows legacy features in the project.\n\nTo enable necessary module just import it to the project, eg:\n```go\nimport (\n\t_ \"https://github.com/koykov/decoder_vector\"\n)\n```\nand vector's [features](https://github.com/koykov/decoder_vector) will be available inside decoders.\n\nFeel free to develop your own extensions. Strongly recommend to register new modifiers using namespaces, like\n[this](https://github.com/koykov/decoder_vector/blob/master/init.go#L15).\n\n### Conclusion\n\nDue to two phases (parsing and decoding) in using decoders it isn't handy to use in simple cases, especially outside\nhighload. The good condition to use it is a highload project and dynamic support requirement. Use decoders in proper\nconditions and wish you happy decoding.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoykov%2Fdecoder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoykov%2Fdecoder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoykov%2Fdecoder/lists"}