{"id":24314950,"url":"https://github.com/karrick/goavro","last_synced_at":"2025-10-25T21:12:22.755Z","repository":{"id":57481671,"uuid":"87674201","full_name":"karrick/goavro","owner":"karrick","description":"Goavro translates between Go native types and binary and textual Avro data","archived":false,"fork":false,"pushed_at":"2020-03-28T05:00:44.000Z","size":726,"stargazers_count":32,"open_issues_count":1,"forks_count":5,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-01-11T05:24:25.350Z","etag":null,"topics":["avro","go","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/karrick.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-04-09T01:14:13.000Z","updated_at":"2021-03-19T16:51:28.000Z","dependencies_parsed_at":"2022-09-26T17:50:34.147Z","dependency_job_id":null,"html_url":"https://github.com/karrick/goavro","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karrick%2Fgoavro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karrick%2Fgoavro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karrick%2Fgoavro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karrick%2Fgoavro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/karrick","download_url":"https://codeload.github.com/karrick/goavro/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234345365,"owners_count":18817558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","go","golang"],"created_at":"2025-01-17T10:16:16.150Z","updated_at":"2025-09-26T21:30:43.370Z","avatar_url":"https://github.com/karrick.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# goavro\n\n## Deprecation Notice\n\nThis library has been merged into https://github.com/linkedin/goavro,\nwhere all further development will continue.\n\n## Description\n\nGoavro is a library written in Go that supports translating binary and\ntextual Avro data to Go native data types, and conversely translating\nGo native data types to binary or textual Avro data. It encodes by\nappending to an existing or empty Go byte slice, and decodes by\nconsuming bytes from an existing Go byte slice.\n\nA goavro `Codec` is created as a stateless structure that can be\nsafely used in multiple go routines simultaneously.\n\nWith the exeption of features not yet supported, goavro attempts to be\nfully compliant with the most recent version of the\n[Avro specification](http://avro.apache.org/docs/1.8.1/spec.html).\n\n## Resources\n\n* [Avro CLI Examples](https://github.com/miguno/avro-cli-examples)\n* [Avro](http://avro.apache.org/)\n* [Google Snappy](https://code.google.com/p/snappy/)\n* [JavaScript Object Notation, JSON](http://www.json.org/)\n\n## Contrast With Code Generation Tools\n\nIf you have the ability to rebuild and redeploy your software whenever\ndata schemas change, code generation tools might be the best solution\nfor your application.\n\nThere are numerous excellent tools for generating source code to\ntranslate data between native and Avro binary or textual data. One\nsuch tool is linkedin below. If a particular application is designed\nto work with a rarely changing schema, programs that use code\ngenerated functions can potentially be more performant than a program\nthat uses goavro to create a `Codec` at run time.\n\n* [gogen-avro](https://github.com/alanctgardner/gogen-avro)\n\nI recommend benchmarking the resultant programs using typical data\nusing both the code generated functions and using goavro to see which\nperforms better. Not all code generated functions will out perform\ngoavro for all data corpuses.\n\nIf you don't have the ability to rebuild and redeploy software updates\nwhenever a data schema change occurs, goavro could be a great fit for\nyour needs. With goavro at runtime your program can be given a new\nschema, compile it into a `Codec`, and immediately start encoding or\ndecoding data using that `Codec`. Because Avro encoding specifies that\nencoded data always be accompanied by a schema this is not usually a\nproblem. If the schema change is backwards compatible, and the portion\nof your program that handles the decoded data is still able to\nreference the decoded fields, there is nothing that needs to be done\nwhen the schema change is detected by your program when using goavro\n`Codec` instances to encode or decode data.\n\n## Usage\n\nDocumentation is available via\n[![GoDoc](https://godoc.org/github.com/karrick/goavro?status.svg)](https://godoc.org/github.com/karrick/goavro).\n\n```Go\npackage main\n\nimport (\n    \"fmt\"\n\n    \"github.com/karrick/goavro\"\n)\n\nfunc main() {\n    codec, err := goavro.NewCodec(`\n        {\n          \"type\": \"record\",\n          \"name\": \"LongList\",\n          \"fields\" : [\n            {\"name\": \"next\", \"type\": [\"null\", \"LongList\"], \"default\": null}\n          ]\n        }`)\n    if err != nil {\n        fmt.Println(err)\n    }\n\n    // NOTE: May omit fields when using default value\n    textual := []byte(`{\"next\":{\"LongList\":{}}}`)\n\n    // Convert textual Avro data (in Avro JSON format) to native Go form\n    native, _, err := codec.NativeFromTextual(textual)\n    if err != nil {\n        fmt.Println(err)\n    }\n\n    // Convert native Go form to binary Avro data\n    binary, err := codec.BinaryFromNative(nil, native)\n    if err != nil {\n        fmt.Println(err)\n    }\n\n    // Convert binary Avro data back to native Go form\n    native, _, err = codec.NativeFromBinary(binary)\n    if err != nil {\n        fmt.Println(err)\n    }\n\n    // Convert native Go form to textual Avro data\n    textual, err = codec.TextualFromNative(nil, native)\n    if err != nil {\n        fmt.Println(err)\n    }\n\n    // NOTE: Textual encoding will show all fields, even those with values that\n    // match their default values\n    fmt.Println(string(textual))\n    // Output: {\"next\":{\"LongList\":{\"next\":null}}}\n}\n```\n\nAlso please see the example programs in the `examples` directory for\nreference. The `ab2t` program is similar to the reference standard\n`avrocat` program and converts Avro OCF files to Avro JSON\nencoding. The Avro-ReWrite program, `arw`, can be used to rewrite an\nAvro OCF file while optionally changing the block counts, the\ncompression algorithm. `arw` can also upgrade the schema provided the\nexisting datum values can be encoded with the newly provided schema.\n\n### Translating Data\n\nA `Codec` provides four methods for translating between a byte slice\nof either binary or textual Avro data and native Go data.\n\nThe following methods convert data between native Go data and byte\nslices of the binary Avro representation:\n\n    BinaryFromNative\n    NativeFromBinary\n\nThe following methods convert data between native Go data and byte\nslices of the textual Avro representation:\n\n    NativeFromTextual\n    TextualFromNative\n\nEach `Codec` also exposes the `Schema` method to return a simplified\nversion of the JSON schema string used to create the `Codec`.\n\n#### Translating From Avro to Go Data\n\nGoavro does not use Go's structure tags to translate data between\nnative Go types and Avro encoded data.\n\nWhen translating from either binary or textual Avro to native Go data,\ngoavro returns primitive Go data values for corresponding Avro data\nvalues. That is, a Go `nil` is returned for an Avro `null`; a Go\n`bool` for an Avro `boolean`; a Go `[]byte` for an Avro `bytes`; a Go\n`float32` for an Avro `float`, a Go `float64` for an Avro `double`; a\nGo `int64` for an Avro `long`; a Go `int32` for an Avro `int`; and a\nGo `string` for an Avro `string`.\n\nFor complex Avro data types, a Go `[]interface{}` is returned for an\nAvro `array`; a Go `string` for an Avro `enum`; a Go `[]byte` for an\nAvro `fixed`; a Go `map[string]interface{}` for an Avro `map` and\n`record`.\n\nBecause of encoding rules for Avro unions, when an union's value is\n`null`, a simple Go `nil` is returned. However when an union's value\nis non-`nil`, a Go `map[string]interface{}` with a single key is\nreturned for the union. The map's single key is the Avro type name and\nits value is the datum's value.\n\n#### Translating From Go to Avro Data\n\nGoavro does not use Go's structure tags to translate data between\nnative Go types and Avro encoded data.\n\nWhen translating from native Go to either binary or textual Avro data,\ngoavro generally requires the same native Go data types as the decoder\nwould provide, with some exceptions for programmer convenience. Goavro\nwill accept any numerical data type provided there is no precision\nlost when encoding the value. For instance, providing `float64(3.0)`\nto an encoder expecting an Avro `int` would succeed, while sending\n`float64(3.5)` to the same encoder would return an error.\n\nWhen providing a slice of items for an encoder, the encoder will\naccept either `[]interface{}`, or any slice of the required type. For\ninstance, when the Avro schema specifies:\n`{\"type\":\"array\",\"items\":\"string\"}`, the encoder will accept either\n`[]interface{}`, or `[]string`. If given `[]int`, the encoder will\nreturn an error when it attempts to encode the first non-string array\nvalue using the string encoder.\n\nWhen providing a value for an Avro union, the encoder will accept\n`nil` for a `null` value. If the value is non-`nil`, it must be a\n`map[string]interface{}` with a single key-value pair, where the key\nis the Avro type name and the value is the datum's value. As a\nconvenience, the `Union` function wraps any datum value in a map as\nspecified above.\n\n```Go\nfunc ExampleUnion() {\n    codec, err := goavro.NewCodec(`[\"null\",\"string\",\"int\"]`)\n    if err != nil {\n        fmt.Println(err)\n    }\n    buf, err := codec.TextFromNative(nil, goavro.Union(\"string\", \"some string\"))\n    if err != nil {\n        fmt.Println(err)\n    }\n    fmt.Println(string(buf))\n    // Output: {\"string\":\"some string\"}\n}\n```\n\n## Implementation Notes\n\n### API\n\nIn general it is poor form to define a library API which shares the\nsame function or method names but provides a different method\nsignature to an accepted standard. Go has particular strong emphasis\non what a Reader and Writer are, and they conflict with what the Avro\nspecification describes as a reader and a writer. Thus goavro shys\naway from using the terms _reader_ and _writer_ as most Avro tools and\nlibraries would normally use.\n\nIn Go, an `io.Reader` reads data from the stream specified at object\ninstantiation time into a preallocated slice of bytes and returns both\nthe number of bytes read along with an error. In the Avro\nspecification, a reader is a function that decodes Avro data and\nreturns data in native form.\n\nA Go `io.Writer` writes bytes from a slice of bytes to a stream\nspecified at its instantiation time and returns the number of bytes\nwritten along with an error. In the Avro specification, a writer is a\nfunction that encodes data from native form to either binary or text\nAvro bytes.\n\n### Record Field Default Values\n\nThe Avro specification allows for providing default values for each\nAvro Record field. The default value is to be used when reading\ninstances that lack the respective field.\n\nWhen reading binary Avro data, a Record is decoded by reading bytes\nfor the first Record field, immediately followed by the second Record\nfield, and so on. No fields may be skipped in a Record's binary\nencoding, so a default value is deemed unusable. If this assessment is\nwrong, please open a Bug, and provide one or more suitable examples,\nand the developers will be happy to revisit the issue.\n\nWhen decoding from textual Avro data that is missing a particular\nrecord field name, if the record field has a default value, it will be\nused in place of the missing value.\n\nWhen encoding from native Go data that is missing a particular record\nfield name, if the record field has a default value, it will be used\nin place of the missing value.\n\n## Limitations\n\nWith the exeption of features not yet supported, goavro attempts to be\nfully compliant with the most recent version of the\n[Avro specification](http://avro.apache.org/docs/1.8.1/spec.html). The\nfollowing limitations may change as future releases of goavro may\ninclude support for some of these features.\n\n### Aliases\n\nThe Avro specification allows an implementation to optionally map a\nwriter's schema to a reader's schema using aliases. Although goavro\ncan compile schemas with aliases, it does not implement this feature.\n\n### Default maximum block count and block size\n\nTo prevent over allocation of memory when decoding Avro arrays, bytes,\nmaps, strings, and OCF data, goavro returns an error whenever a block\ncount exceeds `MaxBlockCount`, or a block size exceeds\n`MaxBlockSize`. Both of these tokens are set to `math.MaxInt32`, or\n~2.2 GiB, but are declared as variables so a user can change the limit\nif deemed necessary.\n\n### Kafka Streams\n\n[Kafka](http://kafka.apache.org) is the reason goavro was\nwritten. Similar to Avro Object Container Files being a layer of\nabstraction above Avro Data Serialization format, Kafka's use of Avro\nis a layer of abstraction that also sits above Avro Data Serialization\nformat, but has its own schema. Goavro itself is not a Kafka\nlibrary. Goavro coupled with a Kafka library is used everyday to\nprocess hundreds of billions of datum values everyday where goavro was\ndeveloped.\n\n### Logical Types\n\nGoavro does not implement Logical Types as required by the Avro\nspecification.\n\n### RPC Support\n\nGoavro does not implement any high level RPC mechanics required by the\nAvro specification. Avro protocol declarations, messages, message\ntransports, message framing, handshakes, and call format are all\nunsupported by this library.\n\n### Record Field Aliases\n\nThe Avro specification allows for providing a JSON array of strings as\nalternate names for a Record field. While goavro can create `Codec`\ninstances that specify `aliases`, that list is ignored.\n\n### Record Field Order\n\nThe Avro specification allows for providing a sory order string,\neither `ascending`, `descending`, or `ignore`, for use when sorting\nrecords. While goavro can create `Codec` instances that specify\n`order`, those values are not used.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarrick%2Fgoavro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkarrick%2Fgoavro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarrick%2Fgoavro/lists"}