{"id":21056961,"url":"https://github.com/fluent-plugins-nursery/fluent-plugin-parser-avro","last_synced_at":"2025-07-10T07:05:48.386Z","repository":{"id":54585532,"uuid":"298506589","full_name":"fluent-plugins-nursery/fluent-plugin-parser-avro","owner":"fluent-plugins-nursery","description":"Avro parser plugin for Fluentd.","archived":false,"fork":false,"pushed_at":"2025-07-08T08:56:41.000Z","size":43,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-07-08T09:51:55.817Z","etag":null,"topics":["avro","avro-parser-plugin","avro-schema","fluentd","fluentd-plugin"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fluent-plugins-nursery.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-25T07:55:39.000Z","updated_at":"2025-07-08T08:56:46.000Z","dependencies_parsed_at":"2022-08-13T20:31:24.225Z","dependency_job_id":null,"html_url":"https://github.com/fluent-plugins-nursery/fluent-plugin-parser-avro","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/fluent-plugins-nursery/fluent-plugin-parser-avro","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-parser-avro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-parser-avro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-parser-avro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-parser-avro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fluent-plugins-nursery","download_url":"https://codeload.github.com/fluent-plugins-nursery/fluent-plugin-parser-avro/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-parser-avro/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264545018,"owners_count":23625387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","avro-parser-plugin","avro-schema","fluentd","fluentd-plugin"],"created_at":"2024-11-19T16:55:22.495Z","updated_at":"2025-07-10T07:05:48.365Z","avatar_url":"https://github.com/fluent-plugins-nursery.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent-plugin-parser-avro\n\n![Testing on Ubuntu](https://github.com/fluent-plugins-nursery/fluent-plugin-parser-avro/workflows/Testing%20on%20Ubuntu/badge.svg?branch=master)\n![Testing on Windows](https://github.com/fluent-plugins-nursery/fluent-plugin-parser-avro/workflows/Testing%20on%20Windows/badge.svg?branch=master)\n\n[Fluentd](https://fluentd.org/) parser plugin to parse avro formatted data.\n\n## Installation\n\n### RubyGems\n\n```\n$ gem install fluent-plugin-avro\n```\n\n### Bundler\n\nAdd following line to your Gemfile:\n\n```ruby\ngem \"fluent-plugin-avro\"\n```\n\nAnd then execute:\n\n```\n$ bundle\n```\n\n## Configuration\n\n* **schema_file** (string) (optional): avro schema file path.\n* **schema_json** (string) (optional): avro schema definition hash.\n* **schema_url** (string) (optional): avro schema remote URL.\n* **schema_url_key** (string) (optional): avro schema registry or something's response schema key.\n* **writers_schema_file** (string) (optional): avro schema file path for writers definition.\n* **writers_schema_json** (string) (optional): avro schema definition hash for writers definition.\n* **readers_schema_file** (string) (optional): avro schema file path for readers definition.\n* **readers_schema_json** (string) (optional): avro schema definition hash for readers definition.\n* **use_confluent_schema** (bool) (optional): Assume to use confluent schema. Confluent avro schema uses the first 5-bytes for magic byte (1 byte) and schema_id (4 bytes). This parameter specifies to skip reading the first 5-bytes or not.\n  * Default value: `true`.\n* **api_key** (string) (optional): Set key for Basic authentication.\n* **api_secret** (string) (optional): Set secret for Basic authentication.\n\n### \\\u003cconfluent_registry\\\u003e section (optional) (single)\n\n* **url** (string) (required): confluent schema registry URL.\n* **subject** (string) (required): Specify schema subject.\n* **schema_key** (string) (optional): Specify schema key on confluent registry REST API response.\n  * Default value: `schema`.\n* **schema_version** (string) (optional): Specify schema version for the specified subject.\n  * Default value: `latest`.\n\n### Configuration Example\n\n```aconf\n\u003cparse\u003e\n  @type avro\n  # schema_file /path/to/file\n  # schema_json { \"namespace\": \"org.fluentd.parser.avro\", \"type\": \"record\", \"name\": \"User\", \"fields\" : [{\"name\": \"username\", \"type\": \"string\"}, {\"name\": \"age\", \"type\": \"int\"}, {\"name\": \"verified\", \"type\": [\"boolean\", \"null\"], \"default\": false}]}\n  # schema_url http(s)://[server fqdn]:[port]/subjects/[a great user's subject]/[the latest schema version]\n  # schema_key schema\n  # When using with confluent registry without \u003cconfluent_registry\u003e, this parameter must be true.\n  # use_confluent_schema true\n  #\u003cconfluent_registry\u003e\n  #  url http://localhost:8081/\n  #  subject your-awesome-subject\n  #  # schema_key schema\n  #  # schema_version 1\n  #\u003c/confluent_registry\u003e\n\u003c/parse\u003e\n```\n\n## AVRO schema registry support\n\nConfluent AVRO schema registry should respond with REST API.\n\nThis plugin uses the following API:\n\n* [`GET /subjects/(string: subject)/versions/(versionId: version)`](https://docs.confluent.io/current/schema-registry/develop/api.html#get--subjects-(string-%20subject)-versions)\n* [`GET /schemas/ids/(int: id)`](https://docs.confluent.io/current/schema-registry/develop/api.html#get--schemas-ids-int-%20id)\n\nUsers can specify a URL for retrieving the latest schemna information with `\u003cconfluent_registry\u003e`:\n\ne.g.)\n```\n  \u003cconfluent_registry\u003e\n    url http://[confluent registry server ip]:[port]/\n    subject your-awesome-subject\n    # schema_key schema\n    # schema_version 1\n  \u003c/confluent_registry\u003e\n```\n\nFor example, when specifying the following configuration:\n\n```\n\u003cparse\u003e\n  @type avro\n  \u003cconfluent_registry\u003e\n    url http://localhost:8081/\n    subject persons-avro-value\n    # schema_key schema\n    # schema_version 1\n  \u003c/confluent_registry\u003e\n```\n\nThen the parser plugin calls `GET http://localhost:8081/subjects/persons-avro-value/versions/latest` to retrive the registered schema versions. And when parsing failure occurred, this plugin will call `GET http://localhost:8081/schemas/ids/\u003cschema id which is obtained from the second record on avro schema\u003e`.\n\nIf you use this plugin to parse confluent schema, please specify `use_confluent_schema` as `true`.\n\nThis is because, confluent avro schema uses the following structure:\n\nMAGIC_BYTE | schema_id | record\n----------:|:---------:|:---------------\n 1byte     |  4bytes   | record contents\n\nWhen specifying `\u003cconfluent_registry\u003e` section on configuration, this plugin will skip to read the first 5-bytes automatically and parse `schema_id` from there.\n\n## Copyright\n\n* Copyright(c) 2020- Hiroshi Hatake\n* License\n  * Apache License, Version 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-parser-avro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-parser-avro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-parser-avro/lists"}