{"id":50307211,"url":"https://github.com/opencost/bingen","last_synced_at":"2026-05-28T17:30:34.141Z","repository":{"id":352636343,"uuid":"1215965346","full_name":"opencost/bingen","owner":"opencost","description":"Bingen is OpenCost's data storage format","archived":false,"fork":false,"pushed_at":"2026-04-28T00:24:49.000Z","size":244,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-28T02:28:34.149Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opencost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-20T12:35:29.000Z","updated_at":"2026-04-24T18:23:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/opencost/bingen","commit_stats":null,"previous_names":["opencost/bingen"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/opencost/bingen","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencost%2Fbingen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencost%2Fbingen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencost%2Fbingen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencost%2Fbingen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opencost","download_url":"https://codeload.github.com/opencost/bingen/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencost%2Fbingen/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33619964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-28T17:30:30.965Z","updated_at":"2026-05-28T17:30:34.126Z","avatar_url":"https://github.com/opencost.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bingen\nBinary Codec Generator for annotated structs in go.\n\n### Install\nUsing an ssh-agent and git, issue a global config update:\n```bash\n$ git config --global url.git@github.com:.insteadOf https://github.com/\n```\n\nThen get `bingen`:\n```bash\n$ go get github.com/opencost/bingen/cmd/bingen\n```\n\nThen install `bingen`:\n```bash\n$ go install github.com/opencost/bingen/cmd/bingen\n```\n\n### Usage\n```\nUsage of bingen:\n        bingen [flags] -package P [directory]\nFlags:\n  -buffer string\n        qualified package for the Buffer type (default \"github.com/opencost/bingen/pkg/util\")\n  -package string\n        package name to generate binary codecs for\n  -version uint8\n        the versioning to use for the default version set (default 1)\n```\n\n##### Buffer\nThe buffer flag should point to the location of the `util.Buffer` type. Since this is currently a private repository, it's best to just copy/paste [https://github.com/opencost/bingen/blob/main/pkg/util/buffer.go](https://github.com/opencost/bingen/blob/78b6ec35c5fc1050e2eb44f0d78ff658985413eb/pkg/util/buffer.go) into a `pkg/util` within your project. For instance, let's say you copy `buffer.go` to `pkg/util` in your project `github.com/bruh/gen-test`, then the buffer flag would be passed as `-buffer=github.com/bruh/gen-test/pkg/util`\n\n##### Example\nThe easiest way to use `bingen` is via `go:generate`. In a project that contains custom struct types you wish to generate `MarshalBinary` and `UnmarshalBinary` methods for, navigate to the target package. Assuming that the package `pkg/stuff` has 3 types you want to generate binary marshal/unmarshal for: `Foo`, `Bar`, and `Widget`, create a new source file in `pkg/stuff` with the following:\n```go\npackage stuff\n\n// @bingen:generate:Foo\n// @bingen:generate:Bar\n// @bingen:generate:Widget\n\n//go:generate bingen -package=stuff -version=1 -buffer=github.com/bruh/gen-test/pkg/util\n```\n\nIf you're using VSCode, a link will appear above `//go:generate ...`. Click the `run go generate ./...` option. This should run and create a `stuff_codecs.go` source file in the `pkg/stuff` directory. \n\n##### Non Standard Library Types\nIf you're using a non-standard library type as a field on a type targetted for generation, you'll need to ensure that the type implements both `encoding.BinaryMarshaler` and `encoding.BinaryUnmarshaler`. In addition, the type's import path will need to be output in the generated output. To accomplish this, you can use the `// @bingen:import:\u003cpackage import\u003e` directive. For example, if you're using the package: \"github.com/acme/widgets/pkg/widget\", you would need to add the following to anywhere in the target package:\n\n```go\n// @bingen:import:github.com/acme/widgets/pkg/widget\n```\n\n##### External Alias Types\nA more advanced version of the *import* command is the *define* command. This is used when you have an alias type that you want to be treated as a first class citizen in the generated code. For example, let's say you have `type WidgetID string` in a `shared` package, and you want to use `shared.WidgetID` on a field within your bingen package. If you were to only use the `// @bingen:import:github.com/acme/widgets/pkg/shared` directive, then the generated code would treat `shared.WidgetID` as a `encoding.BinaryMarshaler` and `encoding.BinaryUnmarshaler` implementation, not as a simple string. In order to have the generated code treat `shared.WidgetID` as a string, you need to define the external alias type with the `define` directive:\n\nLet's say we have the following `WidgetID` alias type in the `github.com/acme/widgets/pkg/shared` package:\n```go \npackage shared\n\ntype WidgetID string \n```\n\nNow, in our target bingen package, we have the following type that uses `shared.WidgetID`:\n```go \npackage stuff \n\nimport \"github.com/acme/widgets/pkg/shared\"\n\ntype Widget struct {\n      ID shared.WidgetID \n}\n```\n\nNow our bingen syntax will need to include the `define` directive for `shared.WidgetID`:\n```go\npackage stuff\n\n// @bingen:define[string]:github.com/acme/widgets/pkg/shared.WidgetID\n\n// @bingen:generate:Widget\n\n//go:generate bingen -package=stuff -version=1 -buffer=github.com/acme/widgets/pkg/util \n```\n\nNote that the `define` directive also implicitly imports the package, so you do not need to also include an `import` directive for the same package. To summarize, the `import` directive is used when you have a non-standard library type that implements `encoding.BinaryMarshaler` and `encoding.BinaryUnmarshaler`, and you want to use it as a field on a generated type. The `define` directive is used when you have an alias type that does not implement `encoding.BinaryMarshaler` and `encoding.BinaryUnmarshaler`, but you want to treat it as a first class citizen in the generated code with the underlying type's encoding/decoding behavior.\n\n\n##### Interface Implementations\nIt's important that if you type any fields as an `interface`, you'll need to annotate both the interface type as well as any implementations of that type for generation. Continuing the example above, assume we have a `type Thing interface` which both `Bar`, `Widget`, and a new type `Gear` implement. We'll need to add the annotation for both the new type and the interface:\n\n```go\npackage stuff\n\n// @bingen:generate:Foo\n// @bingen:generate:Bar\n// @bingen:generate:Widget\n// @bingen:generate:Gear\n// @bingen:generate:Thing\n\n\n//go:generate bingen -package=stuff -version=2 -buffer=github.com/bruh/gen-test/pkg/util\n```\n\n##### String Table \nThere exists a string table implementation which allows nested objects to leverage a shared set of strings in the encodings. For larger resources that contain many repetitive strings, this will be the most optimal option. However, the impact will be less noticeable on smaller resources. To apply a string table encoding/decoding, use the `generate` command options via `[]`:\n\n```go\npackage stuff\n\n// @bingen:generate:Foo\n// @bingen:generate[stringtable]:Bar\n// @bingen:generate:Widget\n// @bingen:generate:Gear\n// @bingen:generate:Thing\n\n\n//go:generate bingen -package=stuff -version=3 -buffer=github.com/bruh/gen-test/pkg/util\n```\n\nThis option must be applied to a concrete type. \n\n##### Version Sets\nIn the event that you have a resources that should be versioned separately, you can apply a `VersionSet` to generation. This is an ordered operation that needs to be annotated prior to `generate` annotations you wish to be added to that set. The syntax for a version set:\n\n```go\n// @bingen:set[name=\u003cversion-set-name\u003e,version=\u003cset-version\u003e]\n// @bingen:generate:ResourceInVersionSet\n// ...\n// @bingen:end\n```\n\nNote that the `end` command pops the scope of the version set. The following applies version sets for the previous example:\n\n```go\npackage stuff\n\n// Any generates that appear outside of a version set scope will be applied in the default version set,\n// which uses the value passed via -version flag (3 in this example)\n// @bingen:generate:User\n\n// @bingen:set[name=FooBar,version=3]\n// @bingen:generate:Foo\n// @bingen:generate[stringtable]:Bar\n// @bingen:end\n\n// @bingen:set[name=Widgets,version=4]\n// @bingen:generate:Widget\n// @bingen:generate:Gear\n// @bingen:generate:Thing\n// @bingen:end\n\n\n//go:generate bingen -package=stuff -version=3 -buffer=github.com/bruh/gen-test/pkg/util\n```\n\n##### Ignoring Fields\nYou can also ignore fields in the generation process. This is useful if you have a transient field you wish to be ignored during marshal and unmarshal. This is done by adding the `@bingen:field[ignore]` annotation directly after the field:\n\n```go\ntype Person struct {\n      FirstName string \n      LastName string\n      FullName string //@bingen:field[ignore]\n}\n```\n\nIn this example, `FirstName` and `LastName` will both be marshalled and unmarshalled. `FullName` will be ignored.\n\n##### Pre and Post Processing Types\nYou can apply pre and post processing hooks to any generated type. These hooks often work well with ignored/transient fields to ensure that any data being encoded can be populated from the transient data. Likewise, you can also ensure that any transient fields are populated on decode as well. These hooks are enabled via the `generate` options:\n\n```go\n// @bingen:generate[stringtable,preprocess,postprocess]:Person\n```\n\nUsing the `Person` from the previous example, let's have the pre and post processor functions manage the `FullName` field. It's important to note that when you mark a type with `preprocess` and/or `postprocess`, you must also ensure the existence of two functions in the package you have targetting for generation: \n\nFor the `preprocess`, the function name will be `preProcess\u003cType\u003e(myType *\u003cType\u003e)`. For our `Person` example, it would look like this:\n```go\nfunc preProcessPerson(p *Person) {\n      // If the FullName field is set, update the FirstName and LastName fields\n      if p.FullName != \"\" {\n            firstAndLast := strings.Split(p.FullName, \" \")\n            // Make FullName contains a first and last name separated by space\n            if len(firstAndLast) != 2 {\n                  return\n            }\n\n            p.FirstName = firstAndLast[0]\n            p.LastName = firstAndLast[1]\n      }\n}\n```\n\nFor the `postprocess`, the function name will be `postProcess\u003cType\u003e(myType *\u003cType\u003e)`. For our `Person` example, it would look like this:\n```go\nfunc postProcessPerson(p *Person) {\n      // Set the FullName field to the concatenation of FirstName and LastName separated by a space\n      p.FullName = fmt.Sprintf(\"%s %s\", p.FirstName, p.LastName)\n}\n```\n\n##### Migration of Types\nSimilar to the pre and post processing hooks for generated types, you can also specify a migration hook. A migration hook is used when a higher versioned struct unmarshals from a lesser versioned encoding. The most common use of this feature would be to load older data, make a one time change, then store out the new result data. This hook is enabled via the `generate` options:\n\n```go\n// @bingen:generate[stringtable,preprocess,postprocess,migrate]:Person\n```\n\nFor the `migrate` option, the function name will be `migrate\u003cType\u003e(myType *\u003cType\u003e, fromVersion uint8, toVersion uint8)`. For our `Person` example, it would look like this:\n```go\nfunc migratePerson(p *Person, fromVersion uint8, toVersion uint8) {\n      if fromVersion == 1 \u0026\u0026 toVersion == 3 {\n            // special handling for a new field added in v3, loaded from v1 \n      }\n      if fromVersion == 2 \u0026\u0026 toVersion == 3 {\n            // special handling for a new field added in v3, loaded from v2 \n      }\n}\n```\n\n\n##### Streamable Types\nThe `streamable` option generates an `io.Reader`-based streaming iterator for a type, allowing its fields to be decoded one at a time without unmarshalling the entire object into memory. Slices and maps are flattened one level deep, with each element yielded individually. This is well suited for large types where only a subset of fields need to be inspected, or where memory pressure makes loading the full object at once undesirable.\n\nTo enable streaming for a type, add `streamable` to the `generate` annotation options:\n\n```go\npackage stuff\n\n// @bingen:generate[streamable]:Foo\n\n//go:generate bingen -package=stuff -version=1 -buffer=github.com/bruh/gen-test/pkg/util\n```\n\nFor each `streamable` type, bingen generates a `\u003cType\u003eStream` struct implementing the `BingenStream` interface. The interface exposes three methods: `Stream()`, which returns an `iter.Seq2[BingenFieldInfo, *BingenValue]` iterator; `Close()`, which releases the underlying `io.Reader`; and `Error()`, which returns any error that occurred during streaming and should be checked after iteration completes.\n\n`NewStreamFor[T]` is a generated generic function that creates the appropriate stream from a given `io.Reader` for type `T`. It returns an error if `T` was not annotated with `streamable`. Each iteration of the stream yields a `BingenFieldInfo` with the field's `Name string` and `Type reflect.Type`, along with a `*BingenValue` carrying the decoded `Value`. A `nil` `*BingenValue` (detected via `IsNil()`) indicates that a nilable field was encoded as `nil`. For slice and map fields, `BingenValue.Index` holds the element's integer index or map key respectively.\n\nUsing the `Foo` type from the previous example, the following reads its fields from an `io.Reader`:\n\n```go\nstream, err := stuff.NewStreamFor[stuff.Foo](reader)\nif err != nil {\n    fmt.Printf(\"Error creating stream: %s\\n\", err)\n    return\n}\ndefer stream.Close()\n\nfor fi, bv := range stream.Stream() {\n    if bv.IsNil() {\n        fmt.Printf(\"Field: %s (nil)\\n\", fi.Name)\n        continue\n    }\n    // For slice or map fields, bv.Index holds the element's index or key.\n    if bv.Index != nil {\n        fmt.Printf(\"Field: %s[%v] = %v\\n\", fi.Name, bv.Index, bv.Value)\n        continue\n    }\n    fmt.Printf(\"Field: %s = %v\\n\", fi.Name, bv.Value)\n}\n\nif err := stream.Error(); err != nil {\n    fmt.Printf(\"Stream error: %s\\n\", err)\n}\n```\n\nThe `streamable` option can be combined with other options such as `stringtable`:\n\n```go\n// @bingen:generate[streamable,stringtable]:Foo\n```\n\nWhen streaming a type that was encoded with a string table, the stream automatically reads and resolves the string table from the `io.Reader` before yielding any fields. For memory-constrained environments, bingen can spill the string table to a temporary file rather than holding it entirely in memory. This is configured once per package via `ConfigureBingen`:\n\n```go\nstuff.ConfigureBingen(\u0026stuff.BingenConfiguration{\n    FileBackedStringTableEnabled: true,\n    FileBackedStringTableDir:     \"/tmp\",\n})\n```\n\nWhen `FileBackedStringTableEnabled` is `true`, the string table is written to a temporary file in `FileBackedStringTableDir` and individual strings are read from disk on demand. This reduces peak memory usage at the cost of additional file I/O, and pairs well with streaming reads for high-throughput processing of large binary datasets.\n\n#### Backwards Compatibility and Field Versioning\nBingen supports backwards compatibility, but it depends on the user to annotate new fields with the first version the field appears and any specific default values (optional). \n\nFor example, let's say we have a struct `Container`:\n```go\ntype Container struct {\n      Name     string \n      Children []string \n}\n```\n\nand in our `bingen.go` file, we setup a version set:\n```go\n// @bingen:set[name=ContainerExample,version=1]\n// @bingen:generate:Container\n// @bingen:end\n\n//go:generate bingen -package=container -version=1 -buffer=github.com/container-example/pkg/util\n```\n\nNow, if we generate our codec, we can write code that marshals a `Container` instance:\n```go\nc := \u0026Container{\n      Name: \"TestContainer\",\n      Children: []string{\n            \"Child1\",\n            \"Child2\",\n            \"Child3\",\n      },\n}\n\nb, err := c.MarshalBinary()\nif err != nil {\n      fmt.Printf(\"Error: %s\\n\", err)\n      return \n}\n\n// Write Encoded Binary out to a File\nerr = os.WriteFile(\"container.bin\", b, 0644)\nif err != nil {\n      fmt.Printf(\"Failed to save container.bin: %s\\n\", err)\n      return \n}\n```\n\nNow, some time later, we want to update our `Container` struct with a new property: `Parent string`. If we simply add the field and update the version, then our old saved binary file `container.bin` will fail to unmarshal as the new `Container` type. \n\nHowever, we can ensure that bingen knows that the new field is specific to the next version by annotating the specific _new_ fields on the `Container`. These annotations are different than other `//@bingen` tags in that they are specific to the field in which they're applied. \n\nTo annotate our new `Parent string` field, we'll add it to `Container` and use the field versioning annotation:\n```go\ntype Container struct {\n      Name     string \n      Children []string \n      Parent   string    // @bingen:field[version=2]\n}\n```\n\nWe then need to ensure our version set version is also updated:\n```go\n// @bingen:set[name=ContainerExample,version=2]\n// @bingen:generate:Container\n// @bingen:end\n\n//go:generate bingen -package=container -version=2 -buffer=github.com/container-example/pkg/util\n```\n\nNow if we were to load our file `container.bin`:\n```go\nfile, err := os.ReadFile(\"container.bin\")\nif err != nil {\n      fmt.Printf(\"Error reading file container.bin: %s\\n\", err)\n      return\n}\n\nc := \u0026Container{}\nerr = c.UnmarshalBinary(file)\nif err != nil {\n      fmt.Printf(\"Failed to unmarshal binary: %s\\n\", err)\n      return\n}\n\nfmt.Printf(\"Name: %s\\n\", c.Name)\nfor _, child := range c.Children {\n      fmt.Printf(\"Child: %s\\n\", child)\n}\n// Now we check the new property, Parent:\nfmt.Printf(\"Parent: %s\\n\", c.Parent)\n\n// outputs: \n// Name: TestContainer\n// Child: Child1\n// Child: Child2\n// Child: Child3\n// Parent: \n```\n\nNote that the `Parent` was set to the empty string when we unmarshalled the old binary format. This is the default for a `string`, but if you need a specific default to be set when loading older versions, you may specify in the `//@bingen:field` annotation:\n\n```go\ntype Container struct {\n      Name     string \n      Children []string \n      Parent   string    // @bingen:field[version=2, default=momma-container]\n}\n```\n\nNow if we were to re-run the `ReadFile` example, the output would be:\n```go\n// outputs: \n// Name: TestContainer\n// Child: Child1\n// Child: Child2\n// Child: Child3\n// Parent: momma-container\n```\n\n##### Defaults \nWhile the `default` is optional in the field version annotation, there are a few special cases to be aware of:\n* Any nilable type will default to `nil` and ignore the `default` value set. This include `map`, `slice`, `interface` types, and pointer types. \n* String fields default value does not need to include `\"` characters (see above example)\n* Alias types may require casting and more explicit defaults. For example, if you alias `type Foo int`, then use a property `F Foo`, your tag may need to cast the default: \n```go\nF Foo //@bingen:field[version=2, default=Foo(15)]\n```\n\nNow, let's assume we want to add another field to our `Container` type:\n```go\ntype Container struct {\n      Name     string \n      Children []string \n      Parent   string    // @bingen:field[version=2, default=none]\n      Size     int       // @bingen:field[version=3, default=1]\n}\n```\n\nWe advance the version set as well:\n\n```go\n// @bingen:set[name=ContainerExample,version=3]\n// @bingen:generate:Container\n// @bingen:end\n\n//go:generate bingen -package=container -version=3 -buffer=github.com/container-example/pkg/util\n```\n\nThis does _NOT_ prevent us from loading the `container.bin` file that was generated using the v1 shema. Version 3 can unmarshal Version 2 and Version 1. \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencost%2Fbingen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopencost%2Fbingen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencost%2Fbingen/lists"}