{"id":18489712,"url":"https://github.com/abicky/typed_data","last_synced_at":"2025-05-13T23:34:52.217Z","repository":{"id":59158406,"uuid":"257127412","full_name":"abicky/typed_data","owner":"abicky","description":"A library that converts hash objects managed by an Avro schema","archived":false,"fork":false,"pushed_at":"2021-08-15T11:46:51.000Z","size":58,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-12T06:37:58.734Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abicky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-19T23:57:21.000Z","updated_at":"2021-08-15T11:46:54.000Z","dependencies_parsed_at":"2022-09-13T17:50:48.750Z","dependency_job_id":null,"html_url":"https://github.com/abicky/typed_data","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abicky%2Ftyped_data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abicky%2Ftyped_data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abicky%2Ftyped_data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abicky%2Ftyped_data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abicky","download_url":"https://codeload.github.com/abicky/typed_data/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254043217,"owners_count":22004912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T12:57:39.889Z","updated_at":"2025-05-13T23:34:51.874Z","avatar_url":"https://github.com/abicky.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TypedData\n\n![](https://github.com/abicky/typed_data/workflows/CI/badge.svg?branch=master)\n\nTypedData is a library that converts hash objects managed by an Avro schema so that the objects can be loaded into BigQuery.\n\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'typed_data'\n```\n\nAnd then execute:\n\n    $ bundle install\n\nOr install it yourself as:\n\n    $ gem install typed_data\n\n## Usage\n\n### Use as Ruby library\n\n```ruby\nrequire \"typed_data\"\n\nschema = {\n  \"name\" =\u003e \"Record\",\n  \"type\" =\u003e \"record\",\n  \"fields\" =\u003e [\n    {\n      \"name\" =\u003e \"int_field\",\n      \"type\" =\u003e \"int\",\n    },\n    {\n      \"name\" =\u003e \"int_or_string_field\",\n      \"type\" =\u003e [\"int\", \"string\"],\n    },\n    {\n      \"name\" =\u003e \"array_field\",\n      \"type\" =\u003e {\n        \"type\" =\u003e \"array\",\n        \"items\" =\u003e \"int\",\n      },\n    },\n    {\n      \"name\" =\u003e \"union_type_array_field\",\n      \"type\" =\u003e {\n        \"type\" =\u003e \"array\",\n        \"items\" =\u003e [\"int\", \"string\"],\n      },\n    },\n    {\n      \"name\" =\u003e \"nested_map_field\",\n      \"type\" =\u003e {\n        \"type\" =\u003e \"map\",\n        \"values\" =\u003e {\n          \"type\" =\u003e \"map\",\n          \"values\" =\u003e [\"int\", \"string\"],\n        },\n      },\n    },\n  ],\n}\n\nconverter = TypedData::Converter.new(schema)\nconverter.convert({\n  \"int_field\" =\u003e 1,\n  \"int_or_string_field\" =\u003e \"string\",\n  \"array_field\" =\u003e [1, 2],\n  \"union_type_array_field\" =\u003e [1, \"2\"],\n  \"nested_map_field\" =\u003e {\n    \"nested_map\" =\u003e {\n      \"key1\" =\u003e 1,\n      \"key2\" =\u003e \"2\",\n    },\n  },\n})\n#=\u003e {\"int_field\"=\u003e1,\n#    \"int_or_string_field\"=\u003e{\"string_value\"=\u003e\"string\"},\n#    \"array_field\"=\u003e[1, 2],\n#    \"union_type_array_field\"=\u003e[{\"int_value\"=\u003e1}, {\"string_value\"=\u003e\"2\"}],\n#    \"nested_map_field\"=\u003e\n#     [{\"key\"=\u003e\"nested_map\",\n#       \"value\"=\u003e\n#        [{\"key\"=\u003e\"key1\", \"value\"=\u003e{\"int_value\"=\u003e1}},\n#         {\"key\"=\u003e\"key2\", \"value\"=\u003e{\"string_value\"=\u003e\"2\"}}]}]}\n```\n\nYou can specify the formatter for union type keys. The default formatter is `:bigquery`, which is used for BigQuery tables created by loading Avro data for the first time.\nThe other formatter is `:avro`, the formatter for the Avro JSON encoding, which is used in tables managed by [Google BigQuery Sink Connector](https://docs.confluent.io/current/connect/kafka-connect-bigquery/index.html):\n\n\n```ruby\nconverter = TypedData::Converter.new(schema, key_formatter: :avro)\nconverter.convert({\n  \"int_field\" =\u003e 1,\n  \"int_or_string_field\" =\u003e \"string\",\n  \"array_field\" =\u003e [1, 2],\n  \"union_type_array_field\" =\u003e [1, \"2\"],\n  \"nested_map_field\" =\u003e {\n    \"nested_map\" =\u003e {\n      \"key1\" =\u003e 1,\n      \"key2\" =\u003e \"2\",\n    },\n  },\n})\n#=\u003e {\"int_field\"=\u003e1,\n#    \"int_or_string_field\"=\u003e{\"string\"=\u003e\"string\"},\n#    \"array_field\"=\u003e[1, 2],\n#    \"union_type_array_field\"=\u003e[{\"int\"=\u003e1}, {\"string\"=\u003e\"2\"}],\n#    \"nested_map_field\"=\u003e\n#     [{\"key\"=\u003e\"nested_map\",\n#       \"value\"=\u003e\n#        [{\"key\"=\u003e\"key1\", \"value\"=\u003e{\"int\"=\u003e1}},\n#         {\"key\"=\u003e\"key2\", \"value\"=\u003e{\"string\"=\u003e\"2\"}}]}]}\n```\n\n`TypedData::Restorer` enables you to restore the converted data:\n\n```ruby\nrestorer = TypedData::Restorer.new(schema)\nrestorer.restore({\n  \"int_field\" =\u003e 1,\n  \"int_or_string_field\" =\u003e { \"string_value\" =\u003e \"string\" },\n  \"array_field\" =\u003e [1, 2],\n  \"union_type_array_field\" =\u003e [\n    { \"int_value\" =\u003e 1 },\n    { \"string_value\" =\u003e \"2\" },\n  ],\n  \"nested_map_field\" =\u003e [\n    {\n      \"key\" =\u003e \"nested_map\",\n      \"value\" =\u003e[\n        {\n          \"key\" =\u003e \"key1\",\n          \"value\" =\u003e { \"int_value\" =\u003e 1 }\n        },\n        {\n          \"key\" =\u003e \"key2\",\n          \"value\" =\u003e { \"string_value\" =\u003e \"2\"}\n        },\n      ],\n    },\n  ],\n})\n#=\u003e {\"int_field\"=\u003e1,\n#    \"int_or_string_field\"=\u003e\"string\",\n#    \"array_field\"=\u003e[1, 2],\n#    \"union_type_array_field\"=\u003e[1, \"2\"],\n#    \"nested_map_field\"=\u003e{\"nested_map\"=\u003e{\"key1\"=\u003e1, \"key2\"=\u003e\"2\"}}}\n```\n\n### Use as CLI\n\n```\n$ typed-data help\nCommands:\n  typed-data convert [file] --schema=SCHEMA  # Convert data in an encoding similar to Avro JSON encoding\n  typed-data help [COMMAND]                  # Describe available commands or one specific command\n  typed-data restore [file] --schema=SCHEMA  # Restore converted data\n\n$ typed-data help convert\nUsage:\n  typed-data convert [file] --schema=SCHEMA\n\nOptions:\n  --schema=SCHEMA        # Path to Avro schema file\n  [--key-format=FORMAT]  # Format for union type key\n                         # Default: bigquery\n                         # Possible values: bigquery, avro\n\nDescription:\n  This command converts data in an encoding similar to Avro JSON encoding. You can specify the file in\n  JSON format or JSON Lines format. If the file option is ommited, the command read data from stdin.\n$ typed-data help restore\nUsage:\n  typed-data restore [file] --schema=SCHEMA\n\nOptions:\n  --schema=SCHEMA        # Path to Avro schema file\n  [--key-format=FORMAT]  # Format for union type key\n                         # Default: bigquery\n                         # Possible values: bigquery, avro\n\nDescription:\n  This command restores converted data. You can specify the file in JSON format or JSON Lines format. If\n  the file option is ommited, the command read data from stdin.\n```\n\nFor example, you can restore the data loaded into a BigQuery table like below:\n\n```\n$ bq query --format json 'SELECT * FROM \u003ctable\u003e' | typed-data restore --schema /path/to/avsc\n```\n\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/abicky/typed_data.\n\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabicky%2Ftyped_data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabicky%2Ftyped_data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabicky%2Ftyped_data/lists"}