{"id":19028678,"url":"https://github.com/researchgate/mongoose-avro-schema-generator","last_synced_at":"2025-04-23T15:44:21.333Z","repository":{"id":141869994,"uuid":"111943704","full_name":"researchgate/mongoose-avro-schema-generator","owner":"researchgate","description":"Generates Apache avro schemas from mongoose schemas.","archived":false,"fork":false,"pushed_at":"2023-06-16T13:58:49.000Z","size":305,"stargazers_count":5,"open_issues_count":5,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-13T22:23:39.329Z","etag":null,"topics":["avro","avro-schema","mongoose","mongoose-schema","nodejs"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/researchgate.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-24T17:51:01.000Z","updated_at":"2022-02-16T16:32:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"f958b4dc-69ce-459d-85db-e7d816c01660","html_url":"https://github.com/researchgate/mongoose-avro-schema-generator","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchgate%2Fmongoose-avro-schema-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchgate%2Fmongoose-avro-schema-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchgate%2Fmongoose-avro-schema-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchgate%2Fmongoose-avro-schema-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/researchgate","download_url":"https://codeload.github.com/researchgate/mongoose-avro-schema-generator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250463114,"owners_count":21434725,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","avro-schema","mongoose","mongoose-schema","nodejs"],"created_at":"2024-11-08T21:12:01.011Z","updated_at":"2025-04-23T15:44:21.312Z","avatar_url":"https://github.com/researchgate.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Mongoose Avro Schema Creator\" src=\"./.github/logo.svg\" width=\"888\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://travis-ci.org/researchgate/mongoose-avro-schema-generator\"\u003e\u003cimg alt=\"Build Status\" src=\"https://travis-ci.org/researchgate/mongoose-avro-schema-generator.svg?branch=master\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://codecov.io/gh/researchgate/mongoose-avro-schema-generator\"\u003e\u003cimg src=\"https://codecov.io/gh/researchgate/mongoose-avro-schema-generator/branch/master/graph/badge.svg\" /\u003e\u003c/a\u003e  \n  \u003ca href=\"https://dependencyci.com/github/researchgate/mongoose-avro-schema-generator\"\u003e\u003cimg alt=\"Dependency Status\" src=\"https://dependencyci.com/github/researchgate/mongoose-avro-schema-generator/badge\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://greenkeeper.io/\"\u003e\u003cimg alt=\"Greenkeeper\" src=\"https://badges.greenkeeper.io/researchgate/mongoose-avro-schema-generator.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@researchgate/mongoose-avro-schema-generator\"\u003e\u003cimg alt=\"NPM version\" src=\"https://img.shields.io/npm/v/@researchgate/mongoose-avro-schema-generator.svg\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nA node module that generates Apache avro schemas from mongoose schemas.\n\n## Getting Started\n### Prerequesites\nThe Mongoose Avro Schema Generator requires `node \u003e= 9.2.0`. The only dependency is `mongoose \u003e= 5.0.9`, a connection to MongoDB is not necessary.\n\n### Installation\nUsing yarn:\n```shell\nyarn add @researchgate/mongoose-avro-schema-generator\n```\n\nUsing npm:\n```shell\nnpm install @researchgate/mongoose-avro-schema-generator\n```\n\n### Quick Start\nIn order to generate schemas for all registered mongoose models we import the module, create a new `MongooseAvroSchemaGenerator` and run the `generate()` method.\n\nLet's first register a simple mongoose model.\n```js\nlet schema = new Schema({\n    something: String,\n});\nmongoose.model('mySchema', schema);\n```\nNow we instantiate the Mongoose Avro Schema Generator with a the mongoose instance.\n```js\nconst Generator = require('mongoose-avro-schema-generator');\nconst mongooseAvroSchemaGenerator = new Generator(mongoose);\n```\nThen `mongooseAvroSchemaGenerator.generate()` will output an array of all generated schemas.\n```json\n[\n    {\n        \"dbcollection\": \"myschemas\",\n        \"dbtype\": \"mongodb\",\n        \"fields\": [\n            {\n                \"name\": \"something\",\n                \"type\": [\"null\", \"string\"],\n                \"default\": null\n            },\n            {\n                \"name\": \"_id\",\n                \"type\": [\n                    \"null\",\n                    {\n                        \"subtype\": \"objectid\",\n                        \"type\": \"string\"\n                    }\n                ],\n                \"default\": null\n            },\n            {\n                \"name\": \"__v\",\n                \"type\": [\"null\", \"double\"],\n                \"default\": null\n            }\n        ],\n        \"name\": \"mySchema\",\n        \"namespace\": \"some.namespace\"\n    }\n]\n```\nPlease note that the schema also contains the auto-generated fields `_id` and `_v`. In the following chapters we will have a look at further details.\n\n## Usage\nGenerate avro schemas for all registered mongoose models:\n```js\nmongooseAvroSchemaGenerator.generate();\n```\nRestrict the schema generation to a set of models:\n```js\nmongooseAvroSchemaGenerator.generate(['User', 'Transaction']);\n```\nOverride the default \"mongoose\" namespace by setting the second parameter in the constructor: \n```js\nnew Generator(mongoose, 'some.custom.namespace');\n```\n\n## Mapping\nThis section will explain the mapping from mongoose types to avro schemas.\n\n### Primitive Types\nThe following table lists all the primitive types and their mapped equivalent in the avro schema.\n\n| Native/Mongoose Type | Avro Type |\n| -------------------- | --------- |\n| `String` | `\"string\"` |\n| `Schema.Types.String` | `\"string\"` |\n| `Boolean` | `\"boolean\"` |\n| `Number` | `\"double\"` |\n| `Buffer` | `\"bytes\"` |\n| `Date` | `{ \"type\": \"long\", \"subtype\": \"date\" }` |\n| `Schema.Types.ObjectId` | `{ \"type\": \"string\", \"subtype\": \"objectid\" }` |\n\nPlease note that `Date` and `Schema.Types.ObjectId` are mapped to objects with a more specific subtype.\n\nWarning: The `Schema.Types.Mixed` type is not supported. The same applies to the equivalent empty object literal `{}` or the empty array `[]`. Trying to generate a schema from a model with such a type will result in an error.\n\n### Arrays\nArrays are mapped to the avro type array. For example the mongoose field\n```some: [Number]```\nis getting mapped to the following field in the avro schema:\n```json\n{\n    \"name\": \"some\",\n    \"type\": [\n        \"null\", \n        {\n            \"type\": \"array\",\n            \"items\": [\"null\", \"double\"]\n        }\n    ],\n    \"default\": null\n}\n```\nWe will later see how the default and null values are generated.\n\n### Embedded Documents\nEmbedded documents are mapped to avro records. For example\n```some: { thing: String }```\nis represented by following avro definition:\n```json\n{\n\t\"name\": \"some\",\n\t\"type\": [\n\t\t\"null\",\n\t\t{\n\t\t\t\"name\": \"someEmbedded\",\n\t\t\t\"type\": \"record\",\n\t\t\t\"fields\": [{\n\t\t\t\t\"name\": \"thing\",\n\t\t\t\t\"type\": [\"null\", \"string\"],\n\t\t\t\t\"default\": null\n\t\t\t}]\n\t\t},\n\t\t\"record\"\n\t],\n\n\t\"default\": null\n}\n```\nPlease note the name \"someEmbedded\" of the embedded record which is autogenerated from the parent object with name \"some\". This is necessary because in an avro schema embedded documents always need a name.\n\nRecords can also appear as array items. Like with embedded documents the name will be inferred from the parent object. For example\n```some: [{ thing: String }]```\nwill be mapped to an avro schema containing the following field:\n```json\n{\n\t\"name\": \"some\",\n\t\"type\": [\n\t\t\"null\",\n\t\t{\n\t\t\t\"type\": \"array\",\n\t\t\t\"items\": {\n\t\t\t\t\"type\": [\n\t\t\t\t\t\"null\",\n\t\t\t\t\t{\n\t\t\t\t\t\t\"name\": \"someItemEmbedded\",\n\t\t\t\t\t\t\"type\": \"record\",\n\t\t\t\t\t\t\"fields\": [{\n\t\t\t\t\t\t\t\"name\": \"thing\",\n\t\t\t\t\t\t\t\"type\": [\"null\", \"string\"],\n\t\t\t\t\t\t\t\"default\": null\n\t\t\t\t\t\t}]\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t],\n\t\"default\": null\n}\n```\nPlease note the name \"someItemEmbedded\" of the embedded record which is autogenerated from the parent object with name \"some\". The same applies for records embedded in arrays of arrays which will have the suffix \"ItemItem\". In general every array layer appends another \"Item\" suffix.\n\n### Attributes\nThe Mongoose Avro Schema Generator makes use of the mongoose attributes `required` and `default`.\n\n#### Nullable\nThe avro type `null` will be included automatically if no `required : true` attribute is set for a field in mongoose.\nHence the avro schema for the following mongoose schema\n```js\nlet schema = new Schema({\n    something: { type: String },\n});\nmongoose.model('mySchema', schema);\n```\nwill have a union type of `null` and `string`\n```json\n{\n    \"name\": \"something\",\n    \"type\": [\"null\", \"string\"],\n    \"default\": null\n}\n```\nHowever if we add the required attribute\n```js\nlet schema = new Schema({\n    something: { type: String, required: true },\n});\nmongoose.model('mySchema', schema);\n```\nthe type will be restricted to string.\n```json\n{\n  \"name\": \"something\",\n  \"type\": \"string\"\n}\n```\nWe note that also the default value of `null` has been removed.\n\n#### Default Values\nDefault values are either defined explicitly or are implicitly inferred.\n\n##### Explicit Defaults\nThe mongoose schema\n```js\nnew Schema({\n    something: { type: String, default: 'foo' },\n});\n```\nwill be transformed into an avro schema containing the following field\n```json\n{\n    \"name\": \"something\",\n    \"type\": [\"null\", \"string\"],\n    \"default\": \"foo\"\n}\n```\nArrays will always have a default of `null`.\n\nImportant: Functional default values are not supported and will be ignored, i.e. treated as if the field doesn't have any default.\n##### Implicit Defaults\nIf a field can be null and no explicit default value is defined, the default will be set to `null`.\n\n## Complex Example\nLet's register a schema with mongoose.\n```js\nconst mongoose = require('mongoose');\nconst mongooseAvroSchemaGenerator = require('mongoose-avro-schema-generator');\nmongooseAvroSchemaGenerator.init(mongoose);\n\nlet schema = new Schema({\n    something: { type: [[Number]], default: ['foo'] },\n    else: [String]\n});\nmongoose.model('mySchema', schema);\n```\nThen `mongoosevroSchemaGenerator.generate(['mySchema'], { namespace: 'some.namespace' })` will return the following avro schema:\n```json\n[\n    {\n        \"dbcollection\": \"myschemas\",\n        \"dbtype\": \"mongodb\",\n        \"type\": \"record\",\n        \"fields\": [\n            {\n                \"name\": \"something\",\n                \"type\": [\n                    \"null\",\n                    {\n                        \"type\": \"array\",\n                        \"items\": [\n                            \"null\",\n                            {\n                                \"type\": \"array\",\n                                \"items\": [\"null\", \"string\"]\n                            }\n                        ]\n                    }\n                ],\n                \"default\": null\n            },\n            {\n                \"name\": \"else\",\n                \"type\": [\n                    \"null\",\n                    {\n                        \"type\": \"array\",\n                        \"items\": [\"null\", \"string\"]\n                    }  \n                ],\n                \"default\": null\n            },\n            {\n                \"default\": null,\n                \"name\": \"_id\",\n                \"type\": [\n                    \"null\",\n                    {\n                        \"subtype\": \"objectid\",\n                        \"type\": \"string\"\n                    }\n                ]\n            },\n            {\n                \"default\": null,\n                \"name\": \"__v\",\n                \"type\": [\"null\", \"double\"]\n                \n            }\n        ],\n        \"name\": \"mySchema\",\n        \"namespace\": \"some.namespace\"\n    }\n]\n```\n\n## Running tests\nMocha tests for Mongoose Avro Schema Generator can be found in `/test`. A yarn job is configured to run those tests using `yarn test`.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresearchgate%2Fmongoose-avro-schema-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fresearchgate%2Fmongoose-avro-schema-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresearchgate%2Fmongoose-avro-schema-generator/lists"}