{"id":36549894,"url":"https://github.com/encodingx/binary","last_synced_at":"2026-01-12T06:32:23.320Z","repository":{"id":45629733,"uuid":"427589777","full_name":"encodingx/binary","owner":"encodingx","description":"Marshal and unmarshal binary message and file formats in Go","archived":false,"fork":false,"pushed_at":"2022-02-10T02:12:35.000Z","size":262,"stargazers_count":17,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-06-19T18:12:13.102Z","etag":null,"topics":["binary","bitfields","bits","bitwise","bytes","deserialize","encoding","file-format","formats","go","golang","low-level","marshal","messages","packet","serialize","struct","struct-tags","structures","unmarshal"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/encodingx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-13T06:41:43.000Z","updated_at":"2024-03-07T11:59:27.000Z","dependencies_parsed_at":"2022-09-16T22:00:21.791Z","dependency_job_id":null,"html_url":"https://github.com/encodingx/binary","commit_stats":null,"previous_names":["joel-ling/go-bitfields"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/encodingx/binary","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/encodingx%2Fbinary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/encodingx%2Fbinary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/encodingx%2Fbinary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/encodingx%2Fbinary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/encodingx","download_url":"https://codeload.github.com/encodingx/binary/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/encodingx%2Fbinary/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28336316,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T06:09:07.588Z","status":"ssl_error","status_checked_at":"2026-01-12T06:05:18.301Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary","bitfields","bits","bitwise","bytes","deserialize","encoding","file-format","formats","go","golang","low-level","marshal","messages","packet","serialize","struct","struct-tags","structures","unmarshal"],"created_at":"2026-01-12T06:32:23.237Z","updated_at":"2026-01-12T06:32:23.312Z","avatar_url":"https://github.com/encodingx.png","language":"Go","readme":"# Marshal and Unmarshal Binary Formats in Go\nThe Go standard library features packages for converting between structs\nand various message and file formats.\nPackages `encoding/json` and `encoding/xml` are well known\nfor providing convenient functions `Marshal()` and `Unmarshal()`\nthat leverage the declarative nature of struct tags.\nThese functions are however missing from\n[`encoding/binary`](https://pkg.go.dev/encoding/binary),\nleaving developers without an accessible and intuitive way\nto work with binary formats.\n\nThis module supplies the ubiquitous `Marshal()` and `Unmarshal()`\nfunction signatures\nso that developers can define custom binary formats using only struct tags and\navoid custom bit manipulation.\n\n### User Story\n```\nAs a Go developer implementing a binary message or file format,\nI want a pair of functions \"Marshal/Unmarshal\" like those in \"encoding/json\"\nthat convert a struct into a series of bits in a byte slice and vice versa,\nso that I can avoid the complexities of custom bit manipulation.\n```\n\nSee the rest of the behaviour-driven specifications of this module\n[below](#behaviour-driven-specifications).\n\n## Binary Formats\nMessage and file formats specify how bits are arranged to encode information.\nControl over individual bits or groups smaller than a byte is often required\nto put together and take apart these binary structures.\n\n### Message Format Examples\nDescribing the anatomy of TCP/IP headers\nat the beginning of every internet datagram (\"packet\")\nare\n* [Section 3.1](https://datatracker.ietf.org/doc/html/rfc791#section-3.1)\nof RFC 791 Internet Protocol, and\n* [Section 3.1](https://datatracker.ietf.org/doc/html/rfc793#section-3.1)\nof RFC 793 Transmission Control Protocol.\n\n### File Format Examples\nBinary file formats are not significantly different from message formats\nfrom an application developer's perspective.\n[RFC 1952](https://datatracker.ietf.org/doc/html/rfc1952#section-2)\ndescribes the GZIP File Format Specification.\n\n## Working with Binary Formats in Go\nThe smallest data structures Go provides are\nthe basic type `byte` (alias of `uint8`, an unsigned 8-bit integer), and `bool`,\nboth eight bits long.\nTo manipulate data at a scale smaller than eight bits\nwould require the use of bitwise logical and shift [operators](https://go.dev/ref/spec#Arithmetic_operators) such as\n[AND](https://en.wikipedia.org/wiki/Bitwise_operation#AND) (`\u0026`),\n[OR](https://en.wikipedia.org/wiki/Bitwise_operation#OR) (`|`),\nleft [shift](https://en.wikipedia.org/wiki/Bitwise_operation#Bit_shifts) (`\u003c\u003c`),\nand right shift (`\u003e\u003e`).\n\n### Relevant Questions Posted on StackOverflow\nSuggestions on StackOverflow are limited to the use of bitwise operators.\n\n* [Golang: Parse bit values from a byte](https://stackoverflow.com/questions/54809254/golang-parse-bit-values-from-a-byte)\n* [Creating 8 bit binary data from 4,3, and 1 bit data in Golang](https://stackoverflow.com/questions/61885659/creating-8-bit-binary-data-from-4-3-and-1-bit-data-in-golang)\n* [How to pack the C bit field struct via encoding package in GO?](https://stackoverflow.com/questions/60180827/how-to-pack-the-c-bit-field-struct-via-encoding-package-in-go)\n\n## Behaviour-Driven Specifications\n```gherkin\nFeature: Marshal and Unmarshal\n\n    As a Go developer implementing a binary message or file format,\n    I want a pair of functions \"Marshal/Unmarshal\" like those in \"encoding/json\"\n    that convert a struct into a series of bits in a byte slice and vice versa,\n    so that I can avoid the complexities of custom bit manipulation.\n\n    Background:\n        # Ubiquitous language\n        Given a message or file \"format\"\n            \"\"\"\n            A format specifies how bits are arranged to encode information.\n            \"\"\"\n        And the format is a series of \"bit fields\"\n            \"\"\"\n            A bit field is one or more adjacent bits representing a value,\n            and should not be confused with struct fields.\n            \"\"\"\n        And adjacent bit fields are grouped into \"words\"\n            \"\"\"\n            A word is a series of bits that can be simultaneously processed\n            by a given computer architecture and programming language.\n            \"\"\"\n\n        # Define format-structs\n        And a format is represented by a type definition of a \"format-struct\"\n        And the format-struct nests one or more exported \"word-structs\"\n        And the words are tagged to indicate their lengths in number of bits\n```\n```go\n            type RFC791InternetHeaderFormatWithoutOptions struct {\n                RFC791InternetHeaderFormatWord0 `word:\"32\"`\n                RFC791InternetHeaderFormatWord1 `word:\"32\"`\n                RFC791InternetHeaderFormatWord2 `word:\"32\"`\n                RFC791InternetHeaderFormatWord3 `word:\"32\"`\n                RFC791InternetHeaderFormatWord4 `word:\"32\"`\n            }\n```\n```gherkin\n        And the length of each word is a multiple of eight in the range [8, 64]\n\n        # Define word-structs\n        And each word-struct has exported field(s) corresponding to bit field(s)\n        And the fields are of unsigned integer or boolean types\n        And the fields are tagged to indicate the lengths of those bit fields\n```\n```go\n            type RFC791InternetHeaderFormatWord0 struct {\n                Version     uint8  `bitfield:\"4\"`\n                IHL         uint8  `bitfield:\"4\"`\n                Precedence  uint8  `bitfield:\"3\"`\n                Delay       bool   `bitfield:\"1\"`\n                Throughput  bool   `bitfield:\"1\"`\n                Reliability bool   `bitfield:\"1\"`\n                Reserved    uint8  `bitfield:\"2\"`\n                TotalLength uint16 `bitfield:\"16\"`\n            }\n```\n```gherkin\n        And the length of each bit field does not overflow the type of the field\n            \"\"\"\n            A bit field overflows a type\n            when it is long enough to represent values\n            outside the set of values of the type.\n            \"\"\"\n        And the sum of lengths of all fields is equal to the length of that word\n```\n\n### Marshal\n```gherkin\n    Scenario: Marshal a struct into a byte slice\n        Given a format-struct variable representing a binary message or file\n```\n```go\n            internetHeader = RFC791InternetHeaderFormatWithoutOptions{\n                RFC791InternetHeaderFormatWord0{\n                    Version: 4,\n                    IHL:     5,\n                    // ...\n                },\n                // ...\n            }\n```\n```gherkin\n        And the struct field values do not overflow corresponding bit fields\n            \"\"\"\n            A struct field value overflows its corresponding bit field\n            when it falls outside the range of values\n            that can be represented by that bit field given its length.\n            \"\"\"\n        When I pass to function Marshal() a pointer to that struct variable\n```\n```go\n            var (\n                bytes []byte\n                e     error\n            )\n\n            bytes, e = binary.Marshal(\u0026internetHeader)\n```\n```gherkin\n        Then Marshal() should return a slice of bytes and a nil error\n        And I should see struct field values reflected as bits in those bytes\n```\n```go\n            log.Printf(\"%08b\", bytes)\n            // [01000101 ...]\n\n            log.Println(e == nil)\n            // true\n```\n```gherkin\n        And I should see that the lengths of the slice and the format are equal\n            \"\"\"\n            The length of a format is the sum of lengths of the words in it.\n            The length of a word is the sum of lengths of the bit fields in it.\n            \"\"\"\n```\n\n### Unmarshal\n```gherkin\n    Scenario: Unmarshal a byte slice into a struct\n        Given a format-struct type representing a binary message or file format\n```\n```go\n            var internetHeader RFC791InternetHeaderFormatWithoutOptions\n```\n```gherkin\n        And a slice of bytes containing a binary message or file\n```\n```go\n            var bytes []byte\n\n            // ...\n\n            log.Printf(\"%08b\", bytes)\n            // [01000101 ...]\n```\n```gherkin\n        And the lengths of the slice and the format (measured in bits) are equal\n        When I pass to function Unmarshal() the slice of bytes as an argument\n        And I pass to the function a pointer to the struct as a second argument\n```\n```go\n            e = binary.Unmarshal(bytes, \u0026internetHeader)\n```\n```gherkin\n        Then Unmarshal() should return a nil error\n        And I should see struct field values matching the bits in those bytes\n```\n```go\n            log.Println(e == nil)\n            // true\n\n            log.Println(internetHeader.RFC791InternetHeaderFormatWord0.Version)\n            // 4\n\n            log.Println(internetHeader.RFC791InternetHeaderFormatWord0.IHL)\n            // 5\n```\n\n## Performance and Optimisation\nThis module is optimised for performance.\n\n```bash\n$ go test -cpuprofile cpu.prof -memprofile mem.prof -bench . -benchmem\n```\n```\ngoos: linux\ngoarch: arm64\npkg: github.com/encodingx/binary\nBenchmarkMarshal-2     \t 3435181\t       349.9 ns/op\t      64 B/op\t       6 allocs/op\nBenchmarkUnmarshal-2   \t 3004425\t       396.4 ns/op\t      64 B/op\t       8 allocs/op\nPASS\nok  \tgithub.com/encodingx/binary\t3.288s\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fencodingx%2Fbinary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fencodingx%2Fbinary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fencodingx%2Fbinary/lists"}