{"id":22281132,"url":"https://github.com/hamba/avro","last_synced_at":"2025-05-13T23:08:10.297Z","repository":{"id":37935154,"uuid":"172952427","full_name":"hamba/avro","owner":"hamba","description":"A fast Go Avro codec","archived":false,"fork":false,"pushed_at":"2025-04-07T10:36:15.000Z","size":563,"stargazers_count":414,"open_issues_count":15,"forks_count":105,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-10T01:03:53.031Z","etag":null,"topics":["avro","avro-codec","encoder-decoder","golang"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hamba.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-27T16:40:50.000Z","updated_at":"2025-04-07T10:36:19.000Z","dependencies_parsed_at":"2023-10-02T06:23:23.686Z","dependency_job_id":"d9881280-76b2-42d7-84fe-dbcd127ba900","html_url":"https://github.com/hamba/avro","commit_stats":null,"previous_names":[],"tags_count":84,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hamba%2Favro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hamba%2Favro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hamba%2Favro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hamba%2Favro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hamba","download_url":"https://codeload.github.com/hamba/avro/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248137891,"owners_count":21053775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","avro-codec","encoder-decoder","golang"],"created_at":"2024-12-03T16:14:48.278Z","updated_at":"2025-04-10T01:04:10.859Z","avatar_url":"https://github.com/hamba.png","language":"Go","readme":"\u003cpicture\u003e\n  \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"http://svg.wiersma.co.za/hamba/project?title=avro\u0026tag=A%20fast%20Go%20avro%20codec\u0026mode=dark\"\u003e\n  \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"http://svg.wiersma.co.za/hamba/project?title=avro\u0026tag=A%20fast%20Go%20avro%20codec\"\u003e\n  \u003cimg alt=\"Logo\" src=\"http://svg.wiersma.co.za/hamba/project?title=avro\u0026tag=A%20fast%20Go%20avro%20codec\"\u003e\n\u003c/picture\u003e\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/hamba/avro/v2)](https://goreportcard.com/report/github.com/hamba/avro/v2)\n[![Build Status](https://github.com/hamba/avro/actions/workflows/test.yml/badge.svg)](https://github.com/hamba/avro/actions)\n[![Coverage Status](https://coveralls.io/repos/github/hamba/avro/badge.svg?branch=main)](https://coveralls.io/github/hamba/avro?branch=main)\n[![Go Reference](https://pkg.go.dev/badge/github.com/hamba/avro/v2.svg)](https://pkg.go.dev/github.com/hamba/avro/v2)\n[![GitHub release](https://img.shields.io/github/release/hamba/avro.svg)](https://github.com/hamba/avro/releases)\n[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/hamba/avro/master/LICENSE)\n\nA fast Go avro codec\n\n## Overview\n\nInstall with:\n\n```shell\ngo get github.com/hamba/avro/v2\n```\n\n**Note:** This project has renamed the default branch from `master` to `main`. You will need to update your local environment.\n\n## Usage\n\n```go\ntype SimpleRecord struct {\n\tA int64  `avro:\"a\"`\n\tB string `avro:\"b\"`\n}\n\nschema, err := avro.Parse(`{\n    \"type\": \"record\",\n    \"name\": \"simple\",\n    \"namespace\": \"org.hamba.avro\",\n    \"fields\" : [\n        {\"name\": \"a\", \"type\": \"long\"},\n        {\"name\": \"b\", \"type\": \"string\"}\n    ]\n}`)\nif err != nil {\n\tlog.Fatal(err)\n}\n\nin := SimpleRecord{A: 27, B: \"foo\"}\n\ndata, err := avro.Marshal(schema, in)\nif err != nil {\n\tlog.Fatal(err)\n}\n\nfmt.Println(data)\n// Outputs: [54 6 102 111 111]\n\nout := SimpleRecord{}\nerr = avro.Unmarshal(schema, data, \u0026out)\nif err != nil {\n\tlog.Fatal(err)\n}\n\nfmt.Println(out)\n// Outputs: {27 foo}\n```\n\nMore examples in the [godoc](https://pkg.go.dev/github.com/hamba/avro/v2).\n\n#### Types Conversions\n\n| Avro                          | Go Struct                                                  | Go Interface             |\n|-------------------------------|------------------------------------------------------------|--------------------------|\n| `null`                        | `nil`                                                      | `nil`                    |\n| `boolean`                     | `bool`                                                     | `bool`                   |\n| `bytes`                       | `[]byte`                                                   | `[]byte`                 |\n| `float`                       | `float32`                                                  | `float32`                |\n| `double`                      | `float64`                                                  | `float64`                |\n| `long`                        | `int`\\*, `int64`, `uint32`\\**                              | `int`, `int64`, `uint32` |\n| `int`                         | `int`\\*, `int32`, `int16`, `int8`, `uint8`\\**, `uint16`\\** | `int`, `uint8`, `uint16` |\n| `fixed`                       | `uint64`                                                   | `uint64`                 |\n| `string`                      | `string`                                                   | `string`                 |\n| `array`                       | `[]T`                                                      | `[]any`                  |\n| `enum`                        | `string`                                                   | `string`                 |\n| `fixed`                       | `[n]byte`                                                  | `[n]byte`                |\n| `map`                         | `map[string]T{}`                                           | `map[string]any`         |\n| `record`                      | `struct`                                                   | `map[string]any`         |\n| `union`                       | *see below*                                                | *see below*              |\n| `int.date`                    | `time.Time`                                                | `time.Time`              |\n| `int.time-millis`             | `time.Duration`                                            | `time.Duration`          |\n| `long.time-micros`            | `time.Duration`                                            | `time.Duration`          |\n| `long.timestamp-millis`       | `time.Time`                                                | `time.Time`              |\n| `long.timestamp-micros`       | `time.Time`                                                | `time.Time`              |\n| `long.local-timestamp-millis` | `time.Time`                                                | `time.Time`              |\n| `long.local-timestamp-micros` | `time.Time`                                                | `time.Time`              |\n| `bytes.decimal`               | `*big.Rat`                                                 | `*big.Rat`               |\n| `fixed.decimal`               | `*big.Rat`                                                 | `*big.Rat`               |\n| `string.uuid`                 | `string`                                                   | `string`                 |\n\n\\* Please note that the size of the Go type `int` is platform dependent. Decoding an Avro `long` into a Go `int` is\nonly allowed on 64-bit platforms and will result in an error on 32-bit platforms. Similarly, be careful when encoding a\nGo `int` using Avro `int` on a 64-bit platform, as that can result in an integer overflow causing misinterpretation of\nthe data.\n\n\\** Please note that when the Go type is an unsigned integer care must be taken to ensure that information is not lost\nwhen converting between the Avro type and Go type. For example, storing a *negative* number in Avro of `int = -100`\nwould be interpreted as `uint16 = 65,436` in Go. Another example would be storing numbers in Avro `int = 256` that\nare larger than the Go type `uint8 = 0`.\n\n##### Unions\n\nThe following union types are accepted: `map[string]any`, `*T` and `any`.\n\n* **map[string]any:** If the union value is `nil`, a `nil` map will be en/decoded.\nWhen a non-`nil` union value is encountered, a single key is en/decoded. The key is the avro\ntype name, or schema full name in the case of a named schema (enum, fixed or record).\n* ***T:** This is allowed in a \"nullable\" union. A nullable union is defined as a two schema union,\nwith one of the types being `null` (ie. `[\"null\", \"string\"]` or `[\"string\", \"null\"]`), in this case\na `*T` is allowed, with `T` matching the conversion table above. In the case of a slice, the slice can be used\ndirectly.\n* **any:** An `interface` can be provided and the type or name resolved. Primitive types\nare pre-registered, but named types, maps and slices will need to be registered with the `Register` function.\nIn the case of arrays and maps the enclosed schema type or name is postfix to the type with a `:` separator,\ne.g `\"map:string\"`. Behavior when a type cannot be resolved will depend on your chosen configuation options:\n\t* !Config.UnionResolutionError \u0026\u0026 !Config.PartialUnionTypeResolution: the map type above is used\n\t* Config.UnionResolutionError \u0026\u0026 !Config.PartialUnionTypeResolution: an error is returned\n\t* !Config.UnionResolutionError \u0026\u0026 Config.PartialUnionTypeResolution: any registered type will get resolved while any unregistered type will fallback to the map type above.\n\t* Config.UnionResolutionError \u0026\u0026 !Config.PartialUnionTypeResolution: any registered type will get resolved while any unregistered type will return an error.\n\n##### TextMarshaler and TextUnmarshaler\n\nThe interfaces `TextMarshaler` and `TextUnmarshaler` are supported for a `string` schema type. The object will\nbe tested first for implementation of these interfaces, in the case of a `string` schema, before trying regular\nencoding and decoding.\n\nEnums may also implement `TextMarshaler` and `TextUnmarshaler`, and must resolve to valid symbols in the given enum schema.\n\n##### Identical Underlying Types\n\nOne type can be [ConvertibleTo](https://go.dev/ref/spec#Conversions) another type if they have identical underlying types.\nA non-native type is allowed to be used if it can be convertible to *time.Time*, *big.Rat* or *avro.LogicalDuration* for the particular of *LogicalTypes*.\n\nEx.: `type Timestamp time.Time`\n\n##### Custom Type Conversion\n\nIn case of incompatible types, custom type conversion functions can be registered with the `RegisterTypeConverters` function.\nThis requires the use of `map[string]any` or `[]any`.\nThe type conversion for encoding will receive the original value that is to be encoded, and must return a data type that is compatible with the schema, as specified in the table above.\nThe type conversion for decoding will receive the decoded value with a data type that is compatible with the schema, and its return value will be used as the final decoded value.\n\n##### Untrusted Input With Bytes and Strings\n\nFor security reasons, the configuration `Config.MaxByteSliceSize` restricts the maximum size of `bytes` and `string` types created\nby the `Reader`. The default maximum size is `1MiB` and is configurable. This is required to stop untrusted input from consuming all memory and\ncrashing the application. Should this not be need, setting a negative number will disable the behaviour.\n\n## Benchmark\n\nBenchmark source code can be found at: [https://github.com/nrwiersma/avro-benchmarks](https://github.com/nrwiersma/avro-benchmarks)\n\n```\nBenchmarkGoAvroDecode-8      \t  788455\t      1505 ns/op\t     418 B/op\t      27 allocs/op\nBenchmarkGoAvroEncode-8      \t  624343\t      1908 ns/op\t     806 B/op\t      63 allocs/op\nBenchmarkGoGenAvroDecode-8   \t 1360375\t       876.4 ns/op\t     320 B/op\t      11 allocs/op\nBenchmarkGoGenAvroEncode-8   \t 2801583\t       425.9 ns/op\t     240 B/op\t       3 allocs/op\nBenchmarkHambaDecode-8       \t 5046832\t       238.7 ns/op\t      47 B/op\t       0 allocs/op\nBenchmarkHambaEncode-8       \t 6017635\t       196.2 ns/op\t     112 B/op\t       1 allocs/op\nBenchmarkLinkedinDecode-8    \t 1000000\t      1003 ns/op\t    1688 B/op\t      35 allocs/op\nBenchmarkLinkedinEncode-8    \t 3170553\t       381.5 ns/op\t     248 B/op\t       5 allocs/op\n```\n\nAlways benchmark with your own workload. The result depends heavily on the data input.\n\n## Go structs generation\n\nGo structs can be generated for you from the schema. The types generated follow the same logic in [types conversions](#types-conversions)\n\nInstall the struct generator with:\n\n```shell\ngo install github.com/hamba/avro/v2/cmd/avrogen@\u003cversion\u003e\n```\n\nExample usage assuming there's a valid schema in `in.avsc`:\n\n```shell\navrogen -pkg avro -o bla.go -tags json:snake,yaml:upper-camel in.avsc\n```\n\n**Tip:** Omit `-o FILE` to dump the generated Go structs to stdout instead of a file.\n\nCheck the options and usage with `-h`:\n\n```shell\navrogen -h\n```\n\nOr use it as a lib in internal commands, it's the `gen` package\n\n## Avro schema validation\n\n### avrosv\n\nA small Avro schema validation command-line utility is also available. This simple tool leverages the\nschema parsing functionality of the library, showing validation errors or optionally dumping parsed\nschemas to the console. It can be used in CI/CD pipelines to validate schema changes in a repository.\n\nInstall the Avro schema validator with:\n\n```shell\ngo install github.com/hamba/avro/v2/cmd/avrosv@\u003cversion\u003e\n```\n\nExample usage assuming there's a valid schema in `in.avsc` (exit status code is `0`):\n\n```shell\navrosv in.avsc\n```\n\nAn invalid schema will result in a diagnostic output and a non-zero exit status code:\n\n```shell\navrosv bad-default-schema.avsc; echo $?\nError: avro: invalid default for field someString. \u003cnil\u003e not a string\n2\n```\n\nSchemas referencing other schemas can also be validated by providing all of them (schemas are parsed in order):\n\n```shell\navrosv base-schema.avsc schema-withref.avsc\n```\n\nCheck the options and usage with `-h`:\n\n```shell\navrosv -h\n```\n\n### Name Validation\n\nAvro names are validated according to the\n[Avro specification](https://avro.apache.org/docs/1.11.1/specification/#names).\n\nHowever, the official Java library does not validate said names accordingly, resulting to some files out in the wild\nto have invalid names. Thus, this library has a configuration option to allow for these invalid names to be parsed.\n\n```go\navro.SkipNameValidation = true\n```\n\nNote that this variable is global, so ideally you'd need to unset it after you're done with the invalid schema.\n\n## Go Version Support\n\nThis library supports the last two versions of Go. While the minimum Go version is\nnot guaranteed to increase along side Go, it may jump from time to time to support\nadditional features. This will be not be considered a breaking change.\n\n## Who uses hamba/avro?\n\n- [Apache Arrow for Go](https://github.com/apache/arrow-go)\n- [confluent-kafka-go](https://github.com/confluentinc/confluent-kafka-go)\n- [pulsar-client-go](https://github.com/apache/pulsar-client-go)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhamba%2Favro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhamba%2Favro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhamba%2Favro/lists"}