{"id":13626515,"url":"https://github.com/mongodb-js/mongodb-schema","last_synced_at":"2025-08-03T02:13:11.661Z","repository":{"id":18301866,"uuid":"21477868","full_name":"mongodb-js/mongodb-schema","owner":"mongodb-js","description":"Infer a probabilistic schema for a MongoDB collection.","archived":false,"fork":false,"pushed_at":"2025-04-04T12:50:22.000Z","size":2282,"stargazers_count":144,"open_issues_count":14,"forks_count":15,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-06-19T08:45:46.855Z","etag":null,"topics":["compass-tools","mongodb","mongodb-cli","schema"],"latest_commit_sha":null,"homepage":"https://github.com/mongodb-js/mongodb-schema","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mongodb-js.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-07-03T21:43:41.000Z","updated_at":"2025-04-23T10:38:14.000Z","dependencies_parsed_at":"2023-01-13T19:45:52.848Z","dependency_job_id":"face8a03-ef0f-440e-b325-847402929abd","html_url":"https://github.com/mongodb-js/mongodb-schema","commit_stats":{"total_commits":356,"total_committers":18,"mean_commits":19.77777777777778,"dds":0.4578651685393258,"last_synced_commit":"afac3d95b04ab34cc162a3d9dcfba92b35af7c79"},"previous_names":[],"tags_count":53,"template":false,"template_full_name":null,"purl":"pkg:github/mongodb-js/mongodb-schema","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-js%2Fmongodb-schema","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-js%2Fmongodb-schema/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-js%2Fmongodb-schema/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-js%2Fmongodb-schema/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mongodb-js","download_url":"https://codeload.github.com/mongodb-js/mongodb-schema/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-js%2Fmongodb-schema/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266207973,"owners_count":23892904,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compass-tools","mongodb","mongodb-cli","schema"],"created_at":"2024-08-01T21:02:21.117Z","updated_at":"2025-08-03T02:13:11.597Z","avatar_url":"https://github.com/mongodb-js.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# mongodb-schema [![][npm_img]][npm_url] [![][coverage_img]][coverage_url]\n\nInfer a probabilistic schema for a MongoDB collection.\n\n## Usage\n\n`mongodb-schema` can be used as a command line tool or programmatically in your application as a node module.\n\n### Command line\n\nTo install mongodb-schema for command line use, run `npm install -g mongodb-schema`. This will add a new\nshell script which you can run directly from the command line.\n\nThe command line tool expects a MongoDB connection URI and a namespace in the form `\u003cdatabase\u003e.\u003ccollection\u003e`.\nWithout further arguments, it will sample 100 random documents from the collection and print a schema of\nthe collection in JSON format to stdout.\n\n```\nmongodb-schema mongodb://localhost:27017 mongodb.fanclub\n```\n\nAdditional arguments change the number of samples (`--number`), print additional statistics about the\nschema analysis (`--stats`), switch to a different output format (`--format`), or let you suppress the\nschema output altogether (`--no-output`) if you are only interested in the schema statistics, semantic\ntype discovery (`--semantic-types`), and the ability to disable value collection (`--no-values`).\n\nFor more information, run\n\n```\nmongodb-schema --help\n```\n\n### API\n\nThe following example demonstrates how `mongodb-schema` can be used programmatically from\nyour node application. You need to additionally install the MongoDB node driver to follow\nalong with this example.\n\nMake sure you have a `mongod` running on localhost on port 27017 (or change the example\nbelow accordingly).\n\n1. From your application folder, install the driver and `mongodb-schema` locally:\n\n   ```\n   npm install --save mongodb mongodb-schema\n   ```\n\n2. (optional) If you don't have any data in your MongoDB instance yet, you can create a\n`test.data` collection with this command:\n\n    ```\n    mongosh --eval \"db.data.insertMany([{_id: 1, a: true}, {_id: 2, a: 'true'}, {_id: 3, a: 1}, {_id: 4}])\" localhost:27017/test\n    ```\n\n3. Create a new file `parse-schema.js` and paste in the following code:\n\n    ```javascript\n    const { parseSchema } = require('mongodb-schema');\n    const { MongoClient } = require('mongodb');\n\n    const dbName = 'test';\n    const uri = `mongodb://localhost:27017/${dbName}`;\n    const client = new MongoClient(uri);\n\n    async function run() {\n      try {\n        const database = client.db(dbName);\n        const documentStream = database.collection('data').find();\n\n        // Here we are passing in a cursor as the first argument. You can\n        // also pass in a stream or an array of documents directly.\n        const schema = await parseSchema(documentStream);\n\n        console.log(JSON.stringify(schema, null, 2));\n      } finally {\n        await client.close();\n      }\n    }\n\n    run().catch(console.dir);\n    ```\n\n4. When we run the above with `node ./parse-schema.js`, we'll see output\n  similar to this (some fields not present here for clarity):\n\n  ```javascript\n  {\n    \"count\": 4,                   // parsed 4 documents\n    \"fields\": [                   // an array of Field objects, @see `./lib/field.js`\n      {\n        \"name\": \"_id\",\n        \"count\": 4,               // 4 documents counted with _id\n        \"type\": \"Number\",         // the type of _id is `Number`\n        \"probability\": 1,         // all documents had an _id field\n        \"hasDuplicates\": false,  // therefore no duplicates\n        \"types\": [                // an array of Type objects, @see `./lib/types/`\n          {\n            \"name\": \"Number\",     // name of the type\n            \"count\": 4,           // 4 numbers counted\n            \"probability\": 1,\n            \"unique\": 4,\n            \"values\": [           // array of encountered values\n              1,\n              2,\n              3,\n              4\n            ]\n          }\n        ]\n      },\n      {\n        \"name\": \"a\",\n        \"count\": 3,               // only 3 documents with field `a` counted\n        \"probability\": 0.75,      // hence probability 0.75\n        \"type\": [                 // found these types\n          \"Boolean\",\n          \"String\",\n          \"Number\",\n          \"Undefined\"             // for convenience, we treat Undefined as its own type\n        ],\n        \"hasDuplicates\": false,   // there were no duplicate values\n        \"types\": [\n          {\n            \"name\": \"Boolean\",\n            \"count\": 1,\n            \"probability\": 0.25,  // probabilities for types are calculated factoring in Undefined\n            \"unique\": 1,\n            \"values\": [\n              true\n            ]\n          },\n          {\n            \"name\": \"String\",\n            \"count\": 1,\n            \"probability\": 0.25,\n            \"unique\": 1,\n            \"values\": [\n              \"true\"\n            ]\n          },\n          {\n            \"name\": \"Number\",\n            \"count\": 1,\n            \"probability\": 0.25,\n            \"unique\": 1,\n            \"values\": [\n              1\n            ]\n          },\n          {\n            \"name\": \"Undefined\",\n            \"count\": 1,\n            \"probability\": 0.25,\n            \"unique\": 0\n          }\n        ]\n      }\n    ]\n  }\n```\n\nA high-level view of the schema tree structure is as follows:\n\n![](./docs/mongodb-schema_diagram.png)\n\n\n## BSON Types\n\n`mongodb-schema` supports all [BSON types][bson-types].\nCheckout [the tests][tests] for more usage examples.\n\n## Semantic Types\n\nAs of version 6.1.0, mongodb-schema has a new feature called \"Semantic Type Detection\". It allows to override the type identification of a value. This enables users to provide specific domain knowledge of their data, while still using\nthe underlying flexible BSON representation and nested documents and arrays.\n\nOne of the built-in semantic types is GeoJSON, which traditionally would just be detected as \"Document\" type. With the new option `semanticTypes` enabled, these sub-documents are now considered atomic values with a type \"GeoJSON\". The original BSON type name is still available under the `bsonType` field.\n\nTo enable this mode, use the `-t` or `--semantic-types` flag at the command line. When using the API, pass an option object as the second parameter with the `semanticTypes` flag set to `true`:\n\n```javascript\nparseSchema(db.collection('data').find(), {semanticTypes: true}, function(err, schema) {\n  ...\n});\n```\nThis mode is disabled by default.\n\n#### Custom Semantic Types\n\nIt is also possible to provide custom semantic type detector functions. This is useful to provide domain\nknowledge, for example to detect trees or graphs, special string encodings of data, etc.\n\nThe detector function is called with `value` and `path` (the full field path in dot notation)\nas arguments, and must return a truthy value if the data type applies to this field or value.\n\nHere is an example to detect email addresses:\n\n```javascript\n\nvar emailRegex = /[a-z0-9!#$%\u0026'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%\u0026'*+/=?^_`{|}~-]+)*@(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?/;\n\nfunction emailDetector(value, path) {\n  return emailRegex.test(value);\n};\n\nparseSchema(db.collection('data').find(), { semanticTypes: { EmailAddress: emailDetector } }, function(err, schema) {\n  ...\n});\n```\n\nThis returns a schema with the following content (only partially shown):\n\n```\n{\n  \"name\": \"email\",\n  \"path\": \"email\",\n  \"count\": 100,\n  \"types\": [\n    {\n      \"name\": \"EmailAddress\",     // custom type \"EmailAddress\" was recognized\n      \"bsonType\": \"String\",       // original BSON type available as well\n      \"path\": \"email\",\n      \"count\": 100,\n      \"values\": [\n        \"twinbutterfly28@aim.com\",\n        \"funkymoney45@comcast.net\",\n        \"beauty68@msn.com\",\n        \"veryberry8@hotmail.com\",\n\n```\n\nAs can be seen, the field name \"email\" was correctly identified as a custom type \"EmailAddress\".\n\n## Value Sampling\n\nAs of version 6.1.0, mongodb-schema supports analysing only the structure of the documents, without collection data samples. To enable this mode, use the `--no-values` flag at the command line. When using the API, pass an option object as the second parameter with the `storeValues` flag set to `false`.\n\nThis mode is enabled by default.\n\n## Schema Statistics\n\nTo compare schemas quantitatively we introduce the following measurable metrics on a schema:\n\n#### Schema Depth\nThe schema depth is defined as the maximum number of nested levels of keys in the schema. It does not matter if the subdocuments are nested directly or as elements of an array. An empty document has a depth of 0, whereas a document with some top-level keys but no nested subdocuments has a depth of 1.\n\n#### Schema Width\nThe schema width is defined as the number of individual keys, added up over all nesting levels of the schema. Array values do not count towards the schema width.\n\n#### Examples\n\n```js\n{}\n```\n\nStatistic    | Value\n:----------- | :---:\nSchema Depth | 0\nSchema Width | 0\n\n\n```js\n{\n  one: 1\n}\n```\n\nStatistic    | Value\n:----------- | :---:\nSchema Depth | 1\nSchema Width | 1\n\n\n```js\n{\n  one: [\n    \"foo\",\n    \"bar\",\n    {\n      two: {\n        three: 3\n      }\n    },\n    \"baz\"\n  ],\n  foo: \"bar\"\n}\n```\n\nStatistic    | Value\n:----------- | :---:\nSchema Depth | 3\nSchema Width | 4\n\n```js\n{\n  a: 1,\n  b: false,\n  one: {\n    c: null,\n    two: {\n      three: {\n        four: 4,\n        e: \"deepest nesting level\"\n      }\n    }\n  },\n  f: {\n    g: \"not the deepest level\"\n  }\n}\n```\n\nStatistic    | Value\n:----------- | :---:\nSchema Depth | 4\nSchema Width | 10\n\n\n```js\n// first document\n{\n  foo: [\n    {\n      bar: [1, 2, 3]\n    }\n  ]\n},\n// second document\n{\n  foo: 0\n}\n```\n\nStatistic    | Value\n:----------- | :---:\nSchema Depth | 2\nSchema Width | 2\n\n\n## Testing\n\n```\nnpm test\n```\n\n## License\n\nApache 2.0\n\n\n\n[bson-types]: http://docs.mongodb.org/manual/reference/bson-types/\n[tests]: https://github.com/mongodb-js/mongodb-schema/tree/main/test\n\n[npm_img]: https://img.shields.io/npm/v/mongodb-schema.svg\n[npm_url]: https://www.npmjs.org/package/mongodb-schema\n[coverage_img]: https://coveralls.io/repos/mongodb-js/mongodb-schema/badge.svg\n[coverage_url]: https://coveralls.io/r/mongodb-js/mongodb-schema\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb-js%2Fmongodb-schema","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmongodb-js%2Fmongodb-schema","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb-js%2Fmongodb-schema/lists"}