{"id":13879392,"url":"https://github.com/salsify/avro-schema-registry","last_synced_at":"2025-06-10T15:39:14.174Z","repository":{"id":40008139,"uuid":"54473002","full_name":"salsify/avro-schema-registry","owner":"salsify","description":"Implementation of the Confluent Schema Registry API as a Rails application","archived":false,"fork":false,"pushed_at":"2024-10-31T13:32:42.000Z","size":391,"stargazers_count":91,"open_issues_count":5,"forks_count":21,"subscribers_count":38,"default_branch":"master","last_synced_at":"2024-10-31T14:31:45.043Z","etag":null,"topics":["avro","hacktoberfest","schema-registry"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salsify.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-03-22T12:23:02.000Z","updated_at":"2024-10-31T13:32:44.000Z","dependencies_parsed_at":"2024-01-02T16:27:50.493Z","dependency_job_id":"c38df40b-b912-40d0-8c41-af9c2868b93a","html_url":"https://github.com/salsify/avro-schema-registry","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favro-schema-registry","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favro-schema-registry/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favro-schema-registry/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favro-schema-registry/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salsify","download_url":"https://codeload.github.com/salsify/avro-schema-registry/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226143895,"owners_count":17580245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","hacktoberfest","schema-registry"],"created_at":"2024-08-06T08:02:19.423Z","updated_at":"2025-06-10T15:39:14.160Z","avatar_url":"https://github.com/salsify.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# avro-schema-registry\n\n[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy)\n\n[![Build Status](https://circleci.com/gh/salsify/avro-schema-registry.svg?style=svg)][circleci]\n\n[circleci]: https://circleci.com/gh/salsify/avro-schema-registry\n\nA service for storing and retrieving versions of Avro schemas.\n\nSchema versions stored by the service are assigned an id. These ids can be\nembedded in messages published to Kafka avoiding the need to send the full\nschema with each message.\n\n## Upgrading to v0.11.0\n\nv0.11.0 changes the default fingerprint version to 2. Set `FINGERPRINT_VERSION=1`\nbefore upgrading if you have not migrated to fingerprint version 2.\n\n## Upgrading to v0.6.0\n\nThere is a compatibility break when upgrading to v0.6.0 due to the way that\nfingerprints are generated. Prior to v0.6.0 fingerprints were generated based\non the Parsing Canonical Form for Avro schemas. This does not take into account\nattributes such as `default` that are used during schema resolution and for\ncompatibility checking. The new fingerprint is based on [avro-resolution_canonical_form](https://github.com/salsify/avro-resolution_canonical_form).\n\nTo upgrade:\n\n1. Set `FINGERPRINT_VERSION=1` and `DISABLE_SCHEMA_REGISTRATION=true` in the\n  environment for the application, and restart the application.\n2. Deploy v0.6.0 and run migrations to create and populate the new `fingerprint2`\n  column.\n3. If NOT using the fingerprint endpoint, move to the final step.\n4. Set `FINGERPRINT_VERSION=all`, unset `DISABLE_SCHEMA_REGISTRATION`, and restart the application.\n5. Update all clients to use the v2 fingerprint.\n6. Set `FINGERPRINT_VERSION=2` and unset `DISABLE_SCHEMA_REGISTRATION` (if still set) and\n  restart the application.\n\nAt some point in the future the original `fingerprint` column will be removed.\n\n## Overview\n\nThis application provides the same API as the Confluent\n[Schema Registry](http://docs.confluent.io/3.1.0/schema-registry/docs/api.html).\n\nThe service is implemented as a Rails 7.1 application and stores Avro schemas in\nPostgres. The API is implemented using [Grape](https://github.com/ruby-grape/grape).\n\n### Why?\n\nThe Confluent Schema Registry has been reimplemented because the original\nimplementation uses Kafka to store schemas. We view the messages that pass\nthrough Kafka as more ephemeral and want the flexibility to change how we host Kafka.\nIn the future we may also apply per-subject permissions to the Avro schemas that\nare stored by the registry.\n\n## Extensions\n\nIn addition to the Confluent Schema Registry API, this application provides some\nextensions.\n\n### Schema ID by Fingerprint\n\nThere is an endpoint that can be used to determine by\nfingerprint if a schema is already registered for a subject.\n\nThis endpoint provides a success response that can be cached indefinitely since\nthe id for a schema will not change once it is registered for a subject.\n\n`GET /subjects/(string: subject)/fingerprints/(:fingerprint)`\n\nGet the id of the schema registered for the subject by fingerprint. The\nfingerprint may either be the hex string or the integer value produced by the\n[SHA256 fingerprint](http://avro.apache.org/docs/1.8.1/spec.html#Schema+Fingerprints).\n\n**Parameters:**\n- **subject** (_string_) - Name of the subject that the schema is registered under\n- **fingerprint** (_string_ or _integer_) - SHA256 fingerprint for the schema\n\n**Response JSON Object:**\n- **id** (_int_) - Globally unique identifier of the schema\n\n**Status Codes:**\n- 404 Not Found - Error Code 40403 - Schema not found\n- 500 Internal Server Error - Error code 50001 - Error in the backend datastore\n\n**Example Request:**\n```\nGET /subjects/test/fingerprints/90479eea876f5d6c8482b5b9e3e865ff1c0931c1bfe0adb44c41d628fd20989c HTTP/1.1\nHost: schemaregistry.example.com\nAccept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json\n```\n\n**Example response:**\n```\nHTTP/1.1 200 OK\nContent-Type: application/vnd.schemaregistry.v1+json\n\n{\"id\":1}\n```\n\n### Test Compatibility with a Specified Level\n\nThe Compatibility API is extended to support a `with_compatibility` parameter\nthat controls the level used for the compatibility check against the specified\nschema version.\n\nWhen `with_compatibility` is specified, it overrides any configuration for the\nsubject and the global configuration.\n\n**Example Request:**\n```\nPOST /subjects/test/versions/latest HTTP/1.1\nHost: schemaregistry.example.com\nAccept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json\n\n{\n  \"schema\": \"{ ... }\",\n  \"with_compatibility\": \"BACKWARD\"\n}\n```\n\n### Register A New Schema With Specified Compatibility Levels\n\nThe Subject API is extended to support a `with_compatibility` parameter that\ncontrols the level used for the compatibility check when registering a new\nschema for a subject.\n\nAn `after_compatibility` parameter is also supported to set a new compatibility\nlevel for the subject after a new schema version is registered. This option is\nignored if no new version is created.\n\n**Example Request:**\n```\nPOST /subjects/test/versions HTTP/1.1\nHost: schemaregistry.example.com\nAccept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json\n\n{\n  \"schema\": \"{ ... }\",\n  \"with_compatibility\": \"NONE\",\n  \"after_compatibility\": \"BACKwARD\"\n}\n```\n\n## Setup\n\nThe application is written using Ruby 3.4.3. Start the service using the following\nsteps:\n\n```bash\ngit clone git@github.com:salsify/avro-schema-registry.git\ncd avro-schema-registry\nbundle install\nbin/rake db:setup\nbin/rails s\n```\n\nBy default the service runs on port 21000.\n\n## Deployment\n\nSalsify hosts a public instance of this application at\n[avro-schema-registry.salsify.com](https://avro-schema-registry.salsify.com) that\nanyone can experiment with, just please don't rely on it for production!\n\nThere is also a button above to easily deploy your own copy of the application to Heroku.\n\n### Docker\n\nA Dockerfile is provided to run the application within a container. To\nbuild the image, navigate to the root of the repo and run `docker build`:\n\n```bash\ndocker build . -t avro-schema-registry\n```\n\nThe container is built for the `production` environment, so you'll\nneed to pass in a few environment flags to run it:\n\n```bash\ndocker run -p 5000:5000 -d \\\n  -e DATABASE_URL=postgresql://user:pass@host/dbname \\\n  -e FORCE_SSL=false \\\n  -e SECRET_KEY_BASE=supersecret \\\n  -e SCHEMA_REGISTRY_PASSWORD=avro \\\n  avro-schema-registry\n```\n\nIf you also want to run PostgreSQL in a container, you can link the two containers:\n\n```bash\ndocker run --name avro-postgres -d \\\n  -e POSTGRES_PASSWORD=avro \\\n  -e POSTGRES_USER=avro \\\n  postgres:9.6\n\ndocker run --name avro-schema-registry --link avro-postgres:postgres -p 5000:5000 -d \\\n  -e DATABASE_URL=postgresql://avro:avro@postgres/avro \\\n  -e FORCE_SSL=false \\\n  -e SECRET_KEY_BASE=supersecret \\\n  -e SCHEMA_REGISTRY_PASSWORD=avro \\\n  avro-schema-registry\n```\n\nTo setup the database the first time you run the app, you can call\n`rails db:setup` from within the container:\n\n```bash\ndocker exec avro-schema-registry bundle exec rails db:setup\n```\n\nAlternatively, your can pass `-e AUTO_MIGRATE=1` to your `docker run` command to have the \ncontainer automatically create and migrate the database schema.\n\n## Security\n\nThe service is secured using HTTP Basic authentication and should be used with\nSSL. The default password for the service is 'avro' but it can be set via\nthe environment as `SCHEMA_REGISTRY_PASSWORD`.\n\nAuthentication can be disabled by setting `DISABLE_PASSWORD` to 'true' in the\nenvironment.\n\n## Caching\n\nWhen the environment variable `ALLOW_RESPONSE_CACHING` is set to `true` then the\nservice sets headers to allow responses from the following endpoints to be cached:\n\n- `GET /schemas/(int: id)`\n- `GET /subjects/(string: subject)/fingerprints/(:fingerprint)`\n\nBy default, responses for these endpoints are allowed to be cached for 30 days.\nThis max age can be configured by setting `CACHE_MAX_AGE` to a number of seconds.\n\nTo populate a cache of the responses from these endpoints, the application\ncontains a rake task that can be run in a development environment to iterate\nthrough all registered schemas and issue the cacheable requests:\n\n```bash\nrake cache_all_requests registry_url=https://anything:avro@registry.example.com\n```\n\n## Usage\n\nFor more details on the REST API see the Confluent\n[documentation](http://docs.confluent.io/3.1.0/schema-registry/docs/api.html).\n\nA [client](https://github.com/dasch/avro_turf/blob/master/lib/avro_turf/confluent_schema_registry.rb)\n(see [AvroTurf](https://github.com/dasch/avro_turf)) can be used to\ncommunicate with the service:\n\n```ruby\nrequire 'avro_turf'\nrequire 'avro_turf/confluent_schema_registry'\n\nurl = 'https://anything:avro@registry.example.com'\nclient = AvroTurf::ConfluentSchemaRegistry.new(url)\n\n# registering a new schema returns an id\nid = client.register('test_subject', avro_json_schema)\n# =\u003e 99\n\n# attempting to register the same schema for a subject returns the existing id\nid = client.register('test_subject', avro_json_schema)\n# =\u003e 99\n\n# the JSON for an Avro schema can be fetched by id\nclient.fetch(id)\n# =\u003e avro_json_schema\n```\n\n## Tests\n\nTests for the application can be run using:\n\n```\nbundle exec rspec\n```\n\n## License\n\nThis code is available as open source under the terms of the\n[MIT License](http://opensource.org/licenses/MIT).\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at\nhttps://github.com/salsify/avro-schema-registry.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalsify%2Favro-schema-registry","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalsify%2Favro-schema-registry","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalsify%2Favro-schema-registry/lists"}