{"id":15287738,"url":"https://github.com/agile-lab-dev/darwin","last_synced_at":"2025-08-09T03:20:40.788Z","repository":{"id":45233977,"uuid":"154390013","full_name":"agile-lab-dev/darwin","owner":"agile-lab-dev","description":"Avro Schema Evolution made easy","archived":false,"fork":false,"pushed_at":"2024-02-08T12:09:09.000Z","size":2801,"stargazers_count":34,"open_issues_count":16,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-13T06:04:11.342Z","etag":null,"topics":["avro","avro-schema","hadoop","hbase","scala","schema-evolution","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/agile-lab-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-10-23T20:08:34.000Z","updated_at":"2024-06-03T18:28:08.000Z","dependencies_parsed_at":"2024-02-07T12:43:21.914Z","dependency_job_id":"83594586-e921-4ddb-af20-925b897e9a84","html_url":"https://github.com/agile-lab-dev/darwin","commit_stats":null,"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"purl":"pkg:github/agile-lab-dev/darwin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agile-lab-dev%2Fdarwin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agile-lab-dev%2Fdarwin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agile-lab-dev%2Fdarwin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agile-lab-dev%2Fdarwin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/agile-lab-dev","download_url":"https://codeload.github.com/agile-lab-dev/darwin/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agile-lab-dev%2Fdarwin/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260109459,"owners_count":22960025,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","avro-schema","hadoop","hbase","scala","schema-evolution","spark"],"created_at":"2024-09-30T15:36:11.221Z","updated_at":"2025-06-16T06:04:23.773Z","avatar_url":"https://github.com/agile-lab-dev.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Darwin\n\n[![CI](https://github.com/agile-lab-dev/darwin/actions/workflows/ci.yml/badge.svg)](https://github.com/agile-lab-dev/darwin/actions/workflows/ci.yml)\n[![Gitter](https://badges.gitter.im/agile-lab-darwin/community.svg)](https://gitter.im/agile-lab-darwin/community?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge)\n\n\u003cimg align=\"right\" height=\"260\" src=\"docs/img/logo/darwin-icon.svg\"\u003e\n\nTable of contents\n-------------\n\n- General\n  - [Overview](#overview)\n  - [Artifacts](#artifacts)\n  - [Background](#background)\n  - [Architecture](#architecture)\n  - [JVM compatibility](#jvm-compatibility)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Configuration](#configuration)\n  - [General](#general)\n  - [HBase](#hbase)\n  - [PostgreSql](#postgresql)\n  - [REST](#rest)\n  - [Confluent](#confluent)\n---\n\nOverview\n-------------\nDarwin is a repository of Avro schemas that maintains all the schema versions used during your application lifetime.\nIts main goal is to provide an easy and transparent access to the Avro data in your storage independently from \nschemas evolutions.\nDarwin is portable and it doesn't require any application server.\nTo store its data, you can choose from multiple storage managers (HBase, Postgres) easily pluggable importing the \ndesired connector.\n\nArtifacts\n--------------\nDarwin artifacts are published for scala 2.10, 2.11, 2.12 and 2.13 (from version 1.0.12). \nFrom version 1.0.2 Darwin is available from maven central so there is no need to configure additional repositories \nin your project.\n\nIn order to access to Darwin core functionalities add the core dependency to you project:\n\n### core\n\n#### sbt\n```scala\nlibraryDependencies += \"it.agilelab\" %% \"darwin-core\" % \"1.2.1-SNAPSHOT\"\n```\n#### maven\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eit.agilelab\u003c/groupId\u003e\n  \u003cartifactId\u003edarwin-core_2.11\u003c/artifactId\u003e\n  \u003cversion\u003e1.2.1-SNAPSHOT\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n### HBase connector\n\nThen add the connector of your choice, either HBase:\n\n#### sbt\n```scala\nlibraryDependencies += \"it.agilelab\" %% \"darwin-hbase-connector\" % \"1.2.1-SNAPSHOT\"\n```\n#### maven\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eit.agilelab\u003c/groupId\u003e\n  \u003cartifactId\u003edarwin-hbase-connector_2.11\u003c/artifactId\u003e\n  \u003cversion\u003e1.2.1-SNAPSHOT\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n### Postgresql connector\n\nOr PostgreSql:\n\n### sbt\n\n```scala\nlibraryDependencies += \"it.agilelab\" %% \"darwin-postgres-connector\" % \"1.2.1-SNAPSHOT\"\n```\n#### maven\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eit.agilelab\u003c/groupId\u003e\n  \u003cartifactId\u003edarwin-postgres-connector_2.11\u003c/artifactId\u003e\n  \u003cversion\u003e1.2.1-SNAPSHOT\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n\n### Rest Connector\n\nOr Rest\n\n### sbt\n\n```scala\nlibraryDependencies += \"it.agilelab\" %% \"darwin-rest-connector\" % \"1.2.1-SNAPSHOT\"\n```\n#### maven\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eit.agilelab\u003c/groupId\u003e\n  \u003cartifactId\u003edarwin-rest-connector_2.11\u003c/artifactId\u003e\n  \u003cversion\u003e1.2.1-SNAPSHOT\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n#### Rest server\n\nTo use the rest connector implement the required endpoints or use the reference implementation provided by rest-server module\n\n### Mock connector\n\nOr Mock (only for test scenarios):\n\n### sbt\n\n```scala\nlibraryDependencies += \"it.agilelab\" %% \"darwin-mock-connector\" % \"1.2.1-SNAPSHOT\"\n```\n#### maven\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eit.agilelab\u003c/groupId\u003e\n  \u003cartifactId\u003edarwin-mock-connector_2.11\u003c/artifactId\u003e\n  \u003cversion\u003e1.2.1-SNAPSHOT\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n\n### Confluent schema registry Connector\n\nDarwin can be used as a *facade* over confluent schema registry.\n\n### sbt\n\n```scala\nlibraryDependencies += \"it.agilelab\" %% \"darwin-confluent-connector\" % \"1.2.1-SNAPSHOT\"\n```\n#### maven\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eit.agilelab\u003c/groupId\u003e\n  \u003cartifactId\u003edarwin-confluent-connector_2.11\u003c/artifactId\u003e\n  \u003cversion\u003e1.2.1-SNAPSHOT\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n\nBackground\n-------------\nIn systems where objects encoded using Avro are stored, a problem arises when there is an evolution of the structure \nof those objects. In these cases, Avro is not capable of reading the old data using the schema extracted from the \nactual version of the object: in this scenario each avro-encoded object must be stored along with its schema. \nTo address this problem Avro defined the [Single-Object Encoding specification](https://avro.apache.org/docs/1.8.2/spec.html#single_object_encoding_spec):\n\u003e### Single-object encoding\n\u003eIn some situations a single Avro serialized object is to be stored for a longer period of time.\n\u003eIn the period after a schema change this persistance system will contain records that have been written with \ndifferent schemas. So the need arises to know which schema was used to write a record to support schema evolution correctly.\nIn most cases the schema itself is too large to include in the message, so this binary wrapper format supports the use case more effectively.\n\nDarwin is compliant to this specification and provides utility methods that can generate a Single-Object encoded from\n an Avro byte array and extract an Avro byte array (along with its schema) from a Single-Object encoded one.\n\nArchitecture\n-------------\n### Darwin architecture schema\nDarwin maintains a repository of all the known schemas in the configured storage, and can access these data in three \nconfigurable ways:\n1. ##### Eager Cached\n    Darwin loads all schemas once from the selected storage and fills with them an internal cache that is used for \n    all the subsequent queries. The only other access to the storage is due to the invocation of the `registerAll` \n    method which updates both the cache and the storage with the new schemas. Once the cache is loaded, all the \n    `getId` and `getSchema` method invocations will perform lookups only in the cache.\n\n    ![Darwin schema](docs/img/darwin_eager_cached_schema.jpg)\n\n2. ##### Lazy Cached\n    Darwin behaves like the Eager Cached scenario, but each cache miss is then attempted also into the storage. If \n    the data is found on the storage, the cache is then updated with the fetched data.\n    \n    ![Darwin schema](docs/img/darwin_lazy_cached_schema.jpg)\n    \n3. ##### Lazy\n    Darwin performs all lookups directly on the storage: there is no applicative cache.\n    \n    ![Darwin schema](docs/img/darwin_lazy_schema.jpg)\n\n### Darwin interaction\nDarwin can be used to easily read and write data encoded in Avro Single-Object using the \n`generateAvroSingleObjectEncoded` and `retrieveSchemaAndAvroPayload` methods of a `AvroSchemaManager` instance (they \nrely on the `getId` and `getSchema` methods discussed before). These methods allow your application to convert and \nencoded avro byte array into a single-object encoded one, and to extract the schema and payload from a single-object \nencoded record that was written.\nIf there is the need to use single-object encoding utilities without creating an `AvroSchemaManager` instance, the \nutilities object `AvroSingleObjectEncodingUtils` exposes some generic purpose functionality, such as:\n- check if a byte array is single-object encoded\n- create a single-object encoded byte array from payload and schema ID\n- extract the schema ID from a single-object encoded byte array\n- remove the header (schema ID included) of a single-object encoded byte array\n\n![Darwin interaction](docs/img/darwin_interaction.jpg)\n\nJVM compatibility\n-------------\nDarwin is cross-published among different scala versions (2.10, 2.11, 2.12, 2.13).\nDepending on the Scala version, it targets different JVM versions.\n\nPlease refer to the following compatibility matrix:\n\n| Scala version | JVM version |\n|---------------|-------------|\n| 2.10 | 1.7 |\n| 2.11 | 1.7 |\n| 2.12 | 1.8 |\n| 2.13 | 1.8 |\n\nInstallation\n-------------\nTo use Darwin in your application, simply add it as dependency along with one of the available connectors.\nDarwin can automatically load the defined connector, and it can be used directly to register and to retrieve \nAvro schemas.\n\nUsage\n-------------\nDarwin main functionality are exposed by the `AvroSchemaManager`, which can be used to store and retrieve the known \navro schemas.\nTo get an instance of `AvroSchemaManager` there are two main ways:\n1. You can create an instance of `AvroSchemaManager` directly, passing a `Connector` as constructor argument; the \navailable implementations of `AvroSchemaManager` are the ones introduced in te chapter [Architecture](#architecture):\n `CachedEagerAvroSchemaManager`, `CachedLazyAvroSchemaManager` and `LazyAvroSchemaManager`.\n2. You can obtain an instance of `AvroSchemaManager` using the `AvroSchemaManagerFactory`: for each configuration \npassed as input of the `initialize` method, a new instance is created. The instance can be retrieved later using the \n`getInstance` method.\n\nTo get more insight on how the Typesafe configuration must be defined to create an `AvroSchemaManager` instance (or \ndirectly a `Connector` instance), please check how the configuration file should be created in the Configuration \nsection of the storage you chose.\n\nOnce you created an instance of `AvroSchemaManager`, first of all an application should register all its known Avro\n schemas invoking the `registerAll` method:\n```\n  val manager: AvroSchemaManager = AvroSchemaManagerFactory.initialize(config)\n  val schemas: Seq[Schema] = //obtain all the schemas\n  val registered: Seq[(Long, Schema)] = manager.registerAll(schemas)\n```\nTo generate the Avro schema for your classes there are various ways, if you are using standard Java pojos:\n```\n  val schema: Schema = ReflectData.get().getSchema(classOf[MyClass])\n```\nIf your application uses the _avro4s library_ you can instead obtain the schemas through the `AvroSchema` typeclass \nimplicitly generated by _avro4s_, e.g.:\n```\n  val schema: Schema = new AvroSchema[MyClass]\n```\nOnce you have registered all the schemas used by your application, you can use them directly invoking the \n`AvroSchemaManager` object: it exposes functionality to retrieve the schema from an ID and vice-versa.\n```\n  val id: Long = manager.getId(schema)\n  val schema: Schema = manager.getSchema(id)\n```\n\nAs said previously, in addition to the basic methods, the `AvroSchemaManager` object exposes also some utility methods \nthat can be used to encode/decode a byte array in single-object encoding:\n```\n  def generateAvroSingleObjectEncoded(avroPayload: Array[Byte], schema: Schema): Array[Byte]\n\n  def retrieveSchemaAndAvroPayload(avroSingleObjectEncoded: Array[Byte]): (Schema, Array[Byte])\n```\n\nIf new schemas are added to the storage and the application must reload all the data from it (in order to manage also\n objects encoded with the new schemas), the `reload` method can be used:\n ```\n manager.reload()\n ```\n Please note that this method can be used to reload all the schemas in cached scenarios (this method does nothing if \n you are using a `LazyAvroSchemaManager` instance, because all the find are performed directly on the storage).\n\nConfiguration\n-------------\n\n## General\n\nThe general configuration keys are:\n- **endianness**: tells the factory the endianness which will be used to store and parse schema fingerprints.\nAllowed values are: \"LITTLE_ENDIAN\" and \"BIG_ENDIAN\".\n- **type**: tells the factory which instance of `AvroSchemaManager` must be created. Allowed values are: \n\"cached_eager\", \"cached_lazy\" and \"lazy\".\n- **connector** (optional): used to choose the connector if there are multiple instances of connectors found at \nruntime. If multiple instances are found and this key is not configured, the first connector is taken. All available \nconnectors names are suitable for this value (e.g. \"hbase\", \"postgresql\", etc)\n- **createTable** (optional): if true, tells the chosen Connector to create the repository table if not already \npresent in the storage.\n\n## HBase\n\nThe configuration keys managed by the `HBaseConnector` are:\n- **namespace** (optional): namespace of the table used by Darwin to store the schema repository (if it isn't set, \nthe default value \"AVRO\" is used)\n- **table** (optional): name of the table used by Darwin to store the schema repository (if it isn't set, the default\n value \"SCHEMA_REPOSITORY\" is used)\n- **coreSite** (optional): path of the core-site.xml file (not mandatory if the file is already included in the \nclasspath)\n- **hbaseSite** (optional): path of the hbase-site.xml file (not mandatory if the file is already included in the \nclasspath)\n- **isSecure**: true if the HBase database is kerberos-secured\n- **keytabPath** (optional): path to the keytab containing the key for the principal\n- **principal** (optional): name of the principal, usually in the form of `primary/node@REALM`\n\nExample of configuration for the `HBaseConnector`:\n```\n\"isSecure\": false,\n\"namespace\": \"DARWIN\",\n\"table\": \"REPOSITORY\",\n\"coreSite\": \"/etc/hadoop/conf/core-site.xml\",\n\"hbaseSite\": \"/etc/hadoop/conf/hbase-site.xml\",\n```\n### HBase Connector dependencies\nDarwin HBase Connector does not provide HBase dependencies in a transitive manner since that would lead to hard to \nmanage classpath and class versions conflicts (see Maven hell). Therefore it is mandatory to include also HBase \ndependencies into your project. \n\n## Postgresql\n\nThe configuration keys managed by the `PostgresConnector` are:\n- **table** (optional): name of the table used by Darwin to store the schema repository (if it isn't set, the default\n value \"SCHEMA_REPOSITORY\" is used)\n- **host**: the host of the PostgreSql database\n- **db**: the name of the database where the table will be looked for\n- **username**: the user to connect to PostgreSql\n- **password**: the password of the user to connect to PostgreSql\n- **mode**: controls the way upserts are implemented, either with a single transaction (transaction), or trying to \ninsert and then falling back to update when key violation is raised (exception). default: `transaction` \n\nExample of configuration for the `PostgresConnector`:\n```\n\"host\": \"localhost:5432\"\n\"db\": \"srdb\"\n\"username\": \"postgres\"\n\"password\": \"srpsql\"\n\"table\": \"schema_registry\"\n\"mode\": \"transaction\"\n```\n\n## MongoDB\n\nMongoDB Connector works with scala 2.11, 2.12 and 2.13.\n\nMongoDB dependencies added for creating this connector should be provided by the user.\n\nThere are two possibilities to create a `MongoConnector`:\n- Use the MongoConnectorCreator: A connection to MongoDB will be created by reading the information from a configuration file\n- Use the MongoConnector constructor: Create a MongoConnector. This constructor allows you to build a connector and pass a user-customized connection to it.\n\nThe configuration keys managed by the `MongoConnectorCreator` are:\n- **collection**: name of the collection used by Darwin to store the schema repository\n- **host**: list of the hosts where you want to connect\n- **database**: the name of the database where the table will be looked for\n- **username**: the user to connect to MongoDB\n- **password**: the password of the user to connect to MongoDB\n- **timeout**: maximum waiting time in milliseconds to obtain the results\n\nThe configuration keys managed by the `MongoConnector` are:\n- **collection**: name of the collection used by Darwin to store the schema repository\n- **database**: the name of the database where the table will be looked for\n- **timeout**: maximum waiting time in milliseconds to obtain the results.\n\nEntering other configuration keys will not lead to errors, they will not simply be considered.\n\n```\nusername = \"mongo\"\npassword = \"mongo\"\nhost = [\"localhost:12345\"]\ndatabase = \"test\"\ncollection = \"collection_test\"\ntimeout = 5000\n```\n\n## REST\n\nThe configuration keys managed by the `RestConnector` are:\n- **protocol**: http or https\n- **host**: the hostname where rest-server (or an http proxy) is deployed\n- **port**: the port where rest-server (or an http proxy) is listening\n- **basePath**: the path that should be prefixed to all requests (useful if rest-server is running behind a reverse proxy)\n\nExample of configuration for the `RestConnector`:\n```\n\"protocol\": \"http\"\n\"host\": \"localhost\"\n\"port\": 8080\n\"basePath\": \"/\"\n```\n\n### REST Server\n\nA rest server is provided by module rest-server (only for scala 2.11, 2.12 and 2.13), just run main class\n\n`it.agilelab.darwin.server.rest.Main`\n\n### REST Server configuration\n\nThe same configuration options of darwin as a library should be configured under the darwin key.\n\nThe rest server also accepts a rest specific configuration under darwin-rest key.\n\nExample configuration for the `RestServer`:\n\n```\ndarwin {\n  type = \"lazy\"\n  connector = \"mock\"\n}\n\ndarwin-rest {\n  interface = \"localhost\"\n  port = 8080\n}\n```\n\n## Confluent\n\nDarwin can be used as a `facade` over the `Confluent schema registry`.\n\nConnecting to the confluent schema registry will help all applications currently using darwin to function correctly\nwhen running over confluent platform.\n\nThe connector can be used even if the only confluent component used is the schema registry.\n\nWhen using the confluent connector a the avro single object encoding will be performed using the *Confluent* flavour.\n\n### Confluent Single object encoding\n\nThe schema registry will assign globally unique ids to schemas, each avro message is encoded as following\n\n```\n0x00                       |   1 byte magic number representing confluent encoded avro\n0xXX 0xXX 0xXX 0xXX        |   4 byte schema identifier interpreted as an integer\n...                        |   avro encoded payload without schema (raw avro bytes not prepended with the json schema)\n```\n\n### Subject\n\nConfluent schema registry supports attaching schemas to a `subject`, the subject is the granularity at which schema\ncompatibility is enforced, schemas can be registered with 3 subject strategies\n\n* topic: The subject is the name of the topic (topic contains a single avro data type)\n* record: The subject is the fully qualified name of the topic (multiple topics can contain the same avro data type)\n* topic-record: The subject is derived from topic and record fqdn (a topic can have multiple data types, compatibility on \nsame avro data type will be enforced for each topic instead of globally)\n\nIn order to support this scheme avro schemas registered via darwin should have a custom extension (`x-darwin-subject`)\nlike in this example\n\n```json\n{\n  \"type\" : \"record\",\n  \"name\" : \"record\",\n  \"fields\" : [ {\n    \"name\" : \"stringField\",\n    \"type\" : \"string\"\n  }, {\n    \"name\" : \"stringField2\",\n    \"type\" : [ \"string\", \"null\" ],\n    \"default\" : \"default-for-nullable\"\n  } ],\n  \"x-darwin-subject\" : \"subject-string\"\n}\n```\n\n## Configuration\n\n```hocon\ndarwin {\n  type = \"lazy\"\n  connector = \"confluent\"\n\n  endpoints: [\"http://schema-registry-00:7777\", \"http://schema-registry-01:7777\"]\n  max-cached-schemas: 1000\n  kafka.schemaregistry.standard-property-1: 1\n  kafka.schemaregistry.standard-property-2: \"default\" \n}\n\n```\n\nThe confluent connector can be used by declaring `confluent` as connector.\n\nThe `endpoints` configuration is a list of url to the confluent schema registry\n\nthe `max-cached-schemas` configures how many schemas are internally cached by the confluent schema registry connector\n\nall other properties will be injected in the confluent schema registry client configuration.\n\nFor example if confluent schema registry declares a property `kafka.schemaregistry.auth` this property can simply be\nadded to the darwin configuration like this\n\n```hocon\ndarwin {\n  type = \"lazy\"\n  connector = \"confluent\"\n\n  endpoints: [\"http://schema-registry-00:7777\", \"http://schema-registry-01:7777\"]\n  max-cached-schemas: 1000\n  kafka.schemaregistry.auth: \"true\"\n}\n```\n\n## Mock\n\nMockConnector can be conveniently used during tests or if all the schemas (past and current) are known when launching \nthe application.\nThe MockConnector can load schemas from local files and classpath resources, it can run in two modes: strict and permissive.\nStrict mode fails if any schema cannot be read, while Permissive one will just warn on non-readable schemas. \nDefault mode is \"strict\".\n\nHere is an example of configuration:\n\n```json\n\"files\": [\"/home/user/schema1.avsc\", \"/home/user/schema2.avsc\"]\n\"resources\": [\"schemas/Apple.avsc\", \"schemas/Orange.avsc\"]\n\"mode\": \"permissive\"\n```\n\n----\n\n## Multi-Connector\n\nMulti-connector can connect to multiple connectors in a hierarchical order. It is useful when schemas are registered on different datastore (i.e. confluent + hbase).\n\nYou configure it in the following way:\n\n```\ndarwin {\n  type = \"lazy\"\n  connector = \"multi\"\n  registrar = \"hbase\"\n  confluent-single-object-encoding: \"confluent\"\n  standard-single-object-encoding: [\"hbase\", \"mongo\"]\n  confluent {\n    endpoints: [\"http://schema-registry-00:7777\", \"http://schema-registry-01:7777\"]\n    max-cached-schemas: 1000\n  }\n  hbase {\n    isSecure: false\n    namespace: \"DARWIN\"\n    table: \"REPOSITORY\"\n    coreSite: \"/etc/hadoop/conf/core-site.xml\"\n    hbaseSite: \"/etc/hadoop/conf/hbase-site.xml\"\n  }\n  mongo {\n    username = \"mongo\"\n    password = \"mongo\"\n    host = [\"localhost:12345\"]\n    database = \"test\"\n    collection = \"collection_test\"\n    timeout = 5000\n  }\n}\n```\n\nWhen extracting the schemaId, it will check if the single object encoding is \"confluent\" or \"standard\" way and extract the id.\nGiven the id, it will go through the chain of connectors to find the schema: first confluent-single-object-encoding then\nstandard-single-object-encoding **in order**.\nThe first that matches, is the one that will be used.\n\nIn order to initialize the single connectors, a configuration will be created merging the specific part\n(i.e. hbase/mongo/confluent) with the outer layer: in case of duplicated entries the more specific one will be used.\n\nRegistration of the schema, will work with the connector set as registrar.\n\n### Notes on Avro Single-Object Encoding Specification\n\n#### Canonical form ignores default fields\n\nAs specified in the background section, Darwin leverages the Avro Single-Object Encoding specification to allow the schema fingerprint to be stored along the avro data.\nIn order to create the fingerprint, the schema is converted into its [parsing-canonical form](https://avro.apache.org/docs/1.8.1/spec.html#Transforming+into+Parsing+Canonical+Form), which strips away all the fields that are not needed for reading/writing, such as doc, alias, comment. However, it will also remove the default field, allowing two schemas that are semantically different due to a default field to have the identical fingerprint.\nThe default field is important for compatibility; it is useful to know this tip for debugging purposes in case of a broken compatibility on a subject.\n\nSources:\n\n- [AVRO-2002](https://issues.apache.org/jira/browse/AVRO-2002)\n- [Adding or removing a default does not register a new schema version](https://github.com/salsify/avro-schema-registry/issues/38)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagile-lab-dev%2Fdarwin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagile-lab-dev%2Fdarwin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagile-lab-dev%2Fdarwin/lists"}