{"id":15159373,"url":"https://github.com/nryanov/schemakeeper","last_synced_at":"2025-10-24T16:30:48.025Z","repository":{"id":36228566,"uuid":"203465496","full_name":"nryanov/schemakeeper","owner":"nryanov","description":"Schemakeeper - yet another schema registry for Avro, Thrift and Protobuf schemas","archived":false,"fork":false,"pushed_at":"2022-06-30T12:33:24.000Z","size":545,"stargazers_count":6,"open_issues_count":24,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-31T03:01:33.226Z","etag":null,"topics":["avro","avro-schema","java","kafka","metadata","protobuf","scala","schema-registry","schemakeeper","schemas","thrift"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nryanov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-20T22:44:48.000Z","updated_at":"2024-01-28T10:24:10.000Z","dependencies_parsed_at":"2022-09-12T09:12:11.718Z","dependency_job_id":null,"html_url":"https://github.com/nryanov/schemakeeper","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nryanov%2Fschemakeeper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nryanov%2Fschemakeeper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nryanov%2Fschemakeeper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nryanov%2Fschemakeeper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nryanov","download_url":"https://codeload.github.com/nryanov/schemakeeper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237999527,"owners_count":19399900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","avro-schema","java","kafka","metadata","protobuf","scala","schema-registry","schemakeeper","schemas","thrift"],"created_at":"2024-09-26T21:05:18.918Z","updated_at":"2025-10-24T16:30:42.086Z","avatar_url":"https://github.com/nryanov.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Schemakeeper\n[![GitHub license](https://img.shields.io/github/license/nryanov/Schemakeeper)](https://github.com/nryanov/Schemakeeper/blob/master/LICENSE.txt)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.nryanov.schemakeeper/schemakeeper-common/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.nryanov.schemakeeper/schemakeeper-common)\n[![Codecov](https://img.shields.io/codecov/c/github/nryanov/Schemakeeper)](https://img.shields.io/codecov/c/github/nryanov/Schemakeeper)\n[![Schemakeeper CI](https://github.com/nryanov/Schemakeeper/actions/workflows/scala.yml/badge.svg?branch=master)](https://github.com/nryanov/Schemakeeper/actions/workflows/scala.yml)\n\nSchemakeeper - yet another schema registry for Avro, Thrift and Protobuf schemas. \nIt provides a RESTful interface for storing and retrieving Subjects and Schemas metadata.\n\n## Features\n- Allows to store metadata for Avro, Thrift and Protobuf schemas\n- Allows to use PostgreSQL, MySQL, H2, MariaDB or Oracle as backend for server\n- Can easily be used not only with Kafka \n- Rich REST api\n- Swagger doc \u0026 swagger ui\n- More control on existing subjects - you can lock it to prevent adding new schemas to this subject\n\n### Oracle notes\nTo be able to use Oracle as your backend for schemas it is required to install ojdbc jar manually. You can do it using maven (or any other tool):\n```text\nmvn install:install-file -Dfile=path/to/your/ojdbc8.jar -DgroupId=com.oracle -DartifactId=ojdbc8 -Dversion=19.3.0.0 -Dpackaging=jar\n```\n\n## Requirements\n- Java 8+\n\n## Modules\n- [`schemakeeper-common`](schemakeeper-common) - common classes for SerDe's and API\n- [`schemakeeper-client`](schemakeeper-client) - http client implementation\n- [`schemakeeper-avro`](schemakeeper-avro) - Avro SerDe\n- [`schemakeeper-thrift`](schemakeeper-thrift) - Thrift SerDe\n- [`schemakeeper-protobuf`](schemakeeper-protobuf) - Protobuf SerDe\n- [`schemakeeper-kafka-common`](schemakeeper-kafka-common) - common classes for Kafka SerDe\n- [`schemakeeper-kafka-avro`](schemakeeper-kafka-avro) - Avro SerDe for Kafka\n- [`schemakeeper-kafka-thrift`](schemakeeper-kafka-thrift) - Thrift SerDe for Kafka\n- [`schemakeeper-kafka-protobuf`](schemakeeper-kafka-protobuf) - Protobuf SerDe for Kafka\n\n## Installation\nEvery Schemakeeper module is published at Maven Central\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.nryanov.schemakeeper\u003c/groupId\u003e\n  \u003cartifactId\u003eschemakeeper-${module.name}\u003c/artifactId\u003e\n  \u003cversion\u003e${module.version}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n```groovy\ncompile 'com.nryanov.schemakeeper:${module.name}:${module.version}'\n```\n\n```sbt\n\"com.nryanov.schemakeeper\" %% \"\u003cschemakeeper-module\u003e\" % \"[version]\"\n```\n\n## Build\nServer build:\n```bash\nsbt server/stage\n// executable should be in server/target/universal/stage/bin/*\n```\n### Docker\n```bash\nsbt server/docker:publishLocal\n```\n\n## Test\n```bash\nsbt server/docker:publishLocal\nsbt test\n```\n\n## Run\n```bash\ndocker pull nryanov/schemakeeper:{version}\ndocker run --name={container_name} -p 9081:9081 -p 9990:9990 -d nryanov/schemakeeper:{version}\n```\n\n## Settings\nServer uses [Lightbend HOCON config](https://github.com/lightbend/config) for configuration.\nBy default server is listening **9081** port for rest. Also, H2 is used as default server backend with next settings:\n\n```hocon\nschemakeeper {\n  server {\n    port = 9081\n    host = 0.0.0.0\n  }\n\n  storage {\n    username = \"\"\n    password = \"\"\n    driver = \"org.h2.Driver\"\n    schema = \"public\"\n    url = \"jdbc:h2:mem:schemakeeper;DB_CLOSE_DELAY=-1\"\n  }\n}\n```\n\n### CORS\n```hocon\nschemakeeper {\n  server {\n    cors {\n      allowedCredentials = \"\u003ctrue/false\u003e\"\n      anyOrigin = \"\u003ctrue/false\u003e\"\n      anyMethod = \"\u003ctrue/false\u003e\"\n      maxAge = \"\u003cnumber\u003e\"\n      allowsOrigins = \"\u003cORIGIN\u003e\"\n      allowsMethods = \"\u003cComma-separated methods\u003e\"\n      allowsHeaders = \"\u003cComma-separated headers\u003e\"\n      exposedHeaders = \"\u003cComma-separated headers\u003e\"\n    }\n  }\n}\n```\n\n### Jar\nIf you using jar for starting server, you can configure your app using **-Dconfig.file** java option: \n```bash\njava -jar -Dconfig.file=\u003cPATH TO application.conf\u003e schemakeeper-server.jar\n```\n\n### Docker\nTo configure docker image you can use environment variables:\n- SCHEMAKEEPER_LISTENING_PORT - listening port for rest api\n- SCHEMAKEEPER_LISTENING_HOST - host\n- SCHEMAKEEPER_STORAGE_USERNAME - db username\n- SCHEMAKEEPER_STORAGE_PASSWORD - db password\n- SCHEMAKEEPER_STORAGE_DRIVER - driver (org.h2.Driver, com.mysql.jdbc.Driver, org.postgresql.Driver, org.mariadb.jdbc.Driver, oracle.jdbc.driver.OracleDriver)\n- SCHEMAKEEPER_STORAGE_SCHEMA - db schema\n- SCHEMAKEEPER_STORAGE_URL - jdbc connection url\n\n**Cors settings:**\n- SCHEMAKEEPER_ALLOWS_ORIGINS - Allowed origins\n- SCHEMAKEEPER_ALLOWS_METHODS - Comma-separated methods\n- SCHEMAKEEPER_ALLOWS_HEADERS - Comma-separated headers\n\n## Usage\n\n### Client settings\nUnder the hood the [Unirest](https://github.com/Kong/unirest-java) is used as http client. You can configure it:\n```java\nMap\u003cString, Object\u003e properties = new HashMap();\nproperties.put(ClientConfig.CLIENT_MAX_CONNECTIONS, 10);\nproperties.put(ClientConfig.CLIENT_CONNECTIONS_PER_ROUTE, 5);\nproperties.put(ClientConfig.CLIENT_SOCKET_TIMEOUT, 5000);\nproperties.put(ClientConfig.CLIENT_CONNECT_TIMEOUT, 5000);\n\n// Optionally, you can confgure your http client to use proxy\nproperties.put(ClientConfig.CLIENT_PROXY_HOST, \"proxyHost\");\nproperties.put(ClientConfig.CLIENT_PROXY_PORT, port);\nproperties.put(ClientConfig.CLIENT_PROXY_USERNAME, \"username\");\nproperties.put(ClientConfig.CLIENT_PROXY_PASSWORD, \"password\");\n```  \n\n### Avro\n```java\nMap\u003cString, Object\u003e properties = new HashMap();\nproperties.put(SerDeConfig.SCHEMAKEEPER_URL_CONFIG, \"url\");\nAvroSerDeConfig config = new AvroSerDeConfig(properties);\nAvroSerializer serializer = new AvroSerilizer(config);\nAvroDeserializer deserializer = new AvroDeserializer(config);\n\nbyte[] b = serializer.serialize(\"subject\", data);\nObject r = deserializer.deserialize(b);\n```\n\nAvro SerDe also compatible with Thrift and Protobuf serialized data. It is possible to serialize Thrift/Protobuf using Thrift/Protobuf serializer and then deserialize it using AvroDeserializer.\n\n**With Kafka**\n\nUse `KafkaAvroSerializer.class` and `KafkaAvroDeserializer.class`.\nAlso, the required property is `SerDeConfig.SCHEMAKEEPER_URL_CONFIG`. Other settings are optional for SerDe.\n\n### Thrift\nBefore using Thrift SerDe, you should generate Thrift classes.\n```java\nMap\u003cString, Object\u003e properties = new HashMap();\nproperties.put(SerDeConfig.SCHEMAKEEPER_URL_CONFIG, \"url\");\nThriftSerDeConfig config = new ThriftSerDeConfig(properties);\nThriftSerializer serializer = new ThriftSerializer(config);\nThriftDeserializer deserializer = new ThriftDeserializer(config);\n\nbyte[] b = serializer.serialize(\"subject\", data);\nObject r = deserializer.deserialize(b);\n```\n\n**With Kafka**\n\nUse `KafkaThriftSerializer.class` and `KafkaThriftDeserializer.class`.\nAlso, the required property is `SerDeConfig.SCHEMAKEEPER_URL_CONFIG`. Other settings are optional for SerDe.\n### Protobuf\nBefore using Protobuf SerDe, you should generate Protobuf classes.\n```java\nMap\u003cString, Object\u003e properties = new HashMap();\nproperties.put(SerDeConfig.SCHEMAKEEPER_URL_CONFIG, \"url\");\nProtobufSerDeConfig config = new ProtobufSerDeConfig(properties);\nProtobufSerializer serializer = new ProtobufSerializer(config);\nProtobufDeserializer deserializer = new ProtobufDeserializer(config);\n\nbyte[] b = serializer.serialize(\"subject\", data);\nObject r = deserializer.deserialize(b);\n```\n\n**With Kafka**\n\nUse `KafkaProtobufSerializer.class` and `KafkaProtobufDeserializer.class`.\nAlso, the required property is `SerDeConfig.SCHEMAKEEPER_URL_CONFIG`. Other settings are optional for SerDe.\n\n## Swagger\nSwagger DOC: **/docs**\n\n## API\n### subjects\n**GET /v2/subjects**\n\nGet list of registered subjects\n\n**Response:**\n- Json array of strings\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n\n### subjectMetadata\n**GET /v2/subjects/{subject_name}**\n\nGet subject metadata by name\n\n**Response:**\n- Json object:\n    - subject (string)\n    - compatibilityType (string)\n    - isLocked (boolean)\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Not found 404\n    - Code 1001 Subject does not exist\n    \n### updateSubjectSettings\n**PUT /v2/subjects/{subject_name}**\n\nUpdate subject settings\n\n\n**Body:**\n```json\n{\n \"compatibilityType\": \"Compatibility type\",\n \"isLocked\": \"true/false\"\n}\n```\n\n**Response:**\n- Json object:\n    - subject (string)\n    - compatibilityType (string)\n    - isLocked (boolean)\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Not found 404\n    - Code 1001 Subject does not exist\n### subjectVersions\n**GET /v2/subjects/{subject_name}/versions**\n\nGet list of subject's schema versions\n\n**Response:**\n- Json array of ints\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Not found 404\n    - Code 1001 Subject does not exist\n### subjectSchemasMetadata\n**GET /v2/subjects/{subject_name}/schemas**\n\nGet list of subject's schemas metadata\n\n**Response:**\n- Json array of objects:\n    - schemaId (int)\n    - version (int)\n    - schemaText (string)\n    - schemaHash (string)\n    - schemaType (string)\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Not found 404\n    - Code 1001 Subject does not exist\n### subjectSchemaByVersion\n**GET /v2/subjects/{subject_name}/versions/{version}**\n\nGet subject's schema metadata by version\n\n**Response:**\n- Json object:\n    - schemaId (int)\n    - version (int)\n    - schemaText (string)\n    - schemaHash (string)\n    - schemaType (string)\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Not found 404\n    - Code 1001 Subject does not exist\n    - Code 1004 Subject schema with such version does not exist\n    \n### schemaById\n**GET /v2/schemas/{id}**\n\nGet schema by id\n\n**Response:**\n- Json object:\n    - schemaId (int)\n    - schemaText (string)\n    - schemaHash (string)\n    - schemaType (string)\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Not found 404\n    - Code 1005 Schema does not exist\n### schemaIdBySubjectAndSchema\n**POST /v2/subjects/{subject_name}/schemas/id**\n\n**Body:**\n```json\n{\n \"schemaText\": \"AVRO SCHEMA STRING\",\n \"schemaType\": \"IDENTIFIER Of SCHEMA TYPE [avro, thrift or protobuf]\"\n}\n```\n\nCheck if schema is registered and connected with current subject and return it's id\n\n**Response:**\n- Json object:\n    - schemaId (int)\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Not found 404\n    - Code 1011 Schema is not registered\n    - Code 1012 Schema is not connected to subject\n- Bad request 400\n    - Code 1007 Schema is not valid \n### deleteSubject\n**DELETE /v2/subjects/{subject_name}**\n\nDelete subject metadata\n\n**Response:**\n- Json boolean\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n\n### deleteSubjectSchemaByVersion\n**DELETE /v2/subjects/{subject_name}/versions/{version_number}**\n\nDelete subject schema by version\n\n**Response:**\n- Json boolean\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Not found 404\n    - Code 1001 Subject does not exist\n- Bad request 400\n    - Code 1004 Subject schema with such version does not exist\n### checkSubjectSchemaCompatibility\n**POST /v2/subjects/{subject_name}/compatibility/schemas**\n\n**Body:**\n```json\n{\n \"schemaText\": \"AVRO SCHEMA STRING\",\n \"schemaType\": \"IDENTIFIER Of SCHEMA TYPE [avro, thrift or protobuf]\"\n}\n```\n\nCheck schema compatibility \n\n**Response:**\n- Json boolean\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Not found 404\n    - Code 1001 Subject does not exist\n- Bad request 400\n    - Code 1007 Schema is not valid\n    \n### registerSchema\n**POST /v2/schemas**\n\n**Body:**\n```json\n{\n \"schemaText\": \"AVRO SCHEMA STRING\",\n \"schemaType\": \"IDENTIFIER Of SCHEMA TYPE [avro, thrift or protobuf]\"\n}\n```\n\nRegister new schema\n\n**Response:**\n- Json object:\n    - schemaId (int)\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Bad request 400\n    - Code 1007 Schema is not valid\n    - Code 1008 Schema is already exist\n\n### registerSchemaAndSubject\n**POST /v2/subjects/{subject_name}/schemas**\n\n**Body:**\n```json\n{\n \"schemaText\": \"AVRO SCHEMA STRING\",\n \"schemaType\": \"IDENTIFIER Of SCHEMA TYPE [avro, thrift or protobuf]\",\n \"compatibilityType\": \"SUBJECT COMPATIBILITY TYPE\"\n}\n```\n\nRegister new subject (if not exists), schema (if not exists) and connect it to each other\n\n**Response:**\n- Json object:\n    - schemaId (int)\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Bad request 400\n    - Code 1007 Schema is not valid\n    - Code 1008 Schema is already exist\n    - Code 1010 Schema is not compatible\n    - Code 1013 Subject is locked\n### registerSubject\n**POST /v2/subjects**\n\n**Body:**\n```json\n{\n \"subject\": \"subject name\",\n \"compatibilityType\": \"compatibility type name\",\n \"isLocked\": \"false or true\"\n}\n```\n\nRegister new subject\n\n**Response:**\n- Json object:\n    - subject (string)\n    - compatibilityType (string)\n    - isLocked (boolean)\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Bad request 400\n    - Code 1002 Subject is already exist\n### addSchemaToSubject\n**POST /v2/subjects/{subject_name}/schemas/{schema_id}**\n\nConnect schema to subject as next version\n\n**Response:**\n- Json int - version number\n\n**Status codes:**\n- Internal server error 500 \n    - Code 1000 Backend error\n- Bad request 400\n    - Code 1009 Schema is already connected to subject\n    - Code 1013 Subject is locked\n- Not found 404\n    - Code 1001 Subject does not exist\n    - Code 1005 Schema does not exist\n    ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnryanov%2Fschemakeeper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnryanov%2Fschemakeeper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnryanov%2Fschemakeeper/lists"}