{"id":20528740,"url":"https://github.com/zmap/zschema","last_synced_at":"2025-04-14T04:52:56.804Z","repository":{"id":34836578,"uuid":"38825534","full_name":"zmap/zschema","owner":"zmap","description":"A schema language for JSON documents that allows validation and compilation into various database engines","archived":false,"fork":false,"pushed_at":"2025-01-23T21:43:35.000Z","size":257,"stargazers_count":40,"open_issues_count":9,"forks_count":15,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-03-27T18:52:38.550Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zmap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-09T14:33:08.000Z","updated_at":"2025-02-01T18:54:19.000Z","dependencies_parsed_at":"2022-07-08T13:18:18.167Z","dependency_job_id":null,"html_url":"https://github.com/zmap/zschema","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmap%2Fzschema","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmap%2Fzschema/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmap%2Fzschema/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmap%2Fzschema/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zmap","download_url":"https://codeload.github.com/zmap/zschema/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248824693,"owners_count":21167343,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T23:26:57.939Z","updated_at":"2025-04-14T04:52:56.793Z","avatar_url":"https://github.com/zmap.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"ZSchema\n=======\n\nZSchema is a generic (meta-)schema language for defining database schemas. It\nfacilitates (1) validating JSON documents against a schema definition and (2)\ncompilating a schema to multiple database engines. For example, if you wanted\nto maintain a single database schema for both MongoDB and ElasticSearch.\nProperties can also be documented inline and documentation compiled to HTML or\na console-friendly text document.\n\nSchemas are defined in native Python code. Example:\n\n```python\nRecord({\n    \"name\":String(required=True),\n    \"addresses\":ListOf(SubRecord({\n        \"street\":String(),\n        \"zipcode\":String()\n    }),\n    \"area_code\":Integer()\n})\n```\n\nWhile this might initially seem strange, Python provides a lot flexibility that\nyou don't have in JSON when you're defining a schema. For example, you can\nreuse components without redefining them or define metaclasses for slighty\ndifferent parts of the schema. Overall, ZSchema has a higher learning curve\nthan the languages that ZSchema can compile. However, it makes defining complex\nschemas much easier.\n\nRunning ZSchema\n===============\n\nCommand Line Interface\n----------------------\n\n`zschema [command] [schema] [file (optional)]`\n\nCommands:\n\n * elasticsearch (compile to Elastic Search)\n\n * bigquery (compile to Google BigQuery)\n\n * json (compile documentation to JSON)\n\n * proto (compile documentation to proto3)\n\n * text (compile documentation to plain text)\n\n * html (compile documentation to HTML)\n\n * validate (validate JSON file (one document per line) against schema)\n\nThe schema file can be defined on the command line as module:var. File is only\nneeded when validating whether a data file matches a schema (i.e., using\n`validate` command).\n\n\nCompiling a Schema\n------------------\n\nZSchema allows compiling a registered `Record` to a schema file that can be\nread by another service. For example, if you have the following schema in a\nmodule named `myschema`:\n\n```python\np = Record({\n    \"name\":String(required=True),\n    \"addresses\":ListOf(SubRecord({\n        \"street\":String(),\n        \"zipcode\":String()\n    }),\n    \"area_code\":Integer()\n})\nzschema.registry.register_schema(\"person\", p)\n```\n\nThen you can compile this to Elasticsearch by running the following:\n```\nzschema elasticsearch myschema:person\n```\n\nYou can also register a record by simply calling `.register()` on it:\n\n```python\nRecord({\n    \"name\":String(required=True),\n    \"addresses\":ListOf(SubRecord({\n        \"street\":String(),\n        \"zipcode\":String()\n    }),\n}).register(\"person\")\n```\n\n\nValidating a Schema\n-------------------\n\nIf you wanted to validate a JSON file containing data, you can pass this in along with the schema:\n\n```\nzschema validate myschema:person people.json\n```\n\n\nDeveloping a Schema\n===================\n\nSchemas are created by defining a `Record` object. Records are a set of named\nfields (and their associated types). They can also contain lists of fields and\nsubrecords. Below is a very simple record:\n\n```python\nRecord({\n    \"name\":String(required=True),\n    \"address\":SubRecord({\n        \"street\":String(),\n        \"zipcode\":ZipCode()\n    }),\n    \"area_code\":Integer(),\n\t\"emails\":ListOf(EmailAddress()),\n\t\"enabled\":Boolean(),\n})\n```\n\nYou will immediately notice a few things:\n\n * Fields are instantiated classes and can take initialization options\n\n * You can have customized fields (e.g., `EmailAddress`). These are useful for both\n   maintaining your sanity as well as adding additional validation logic.\n\n\nThese types are known as _leaves_ and you can find the full list here:\nhttps://github.com/zmap/zschema/blob/master/zschema/leaves.py. You'll likely\nnotice that many are Elasticsearch themed (e.g., `EnglishString`,\n`AnalyzedString`), but these will compile down to normal types in other systems\ntoo.\n\nOne of the benefits of ZSchema is that you can define and embed subrecords\nother places:\n\n```python\naddress = SubRecord({\n    \"street\":String(),\n    \"zipcode\":ZipCode()\n    \"country\":String()\n})\n\nRecord({\n    \"name\":String(required=True),\n    \"business_address\":ListOf(address),\n    \"home_address\":ListOf(address),\n    \"area_code\":Integer(),\n\t\"emails\":ListOf(EmailAddress()),\n\t\"enabled\":Boolean(),\n})\n```\n\nOne thing that needs to be careful of here is that all `address` entries here\npoint to the exact same Python object, so you cannot customize one without\nchanging all. To support this use case (which comes up frequently because\ndifferent fields will have different documentation), you can create a new `SubRecordType`:\n\n```python\n\nAddress = SubRecordType({\n    \"street\":String(),\n    \"zipcode\":ZipCode()\n    \"country\":String()\n})\n\nRecord({\n\thome:Address(doc=\"Home Address\"),\n\twork:Address(doc=\"Work Address\"),\n})\n\n```\n\nSimilar to `doc`, fields can have a description, examples, units, min/max\nvalues, etc. A full list of attributes can be found here:\nhttps://github.com/zmap/zschema/blob/master/zschema/leaves.py#L25.\n\n\nRunning Tests\n=============\n\nTests are run with [pytest](https://docs.pytest.org/en/stable/). Run them via:\n```zsh\npip3 install \".[tests]\"\npytest\n```\n\n\nLicense and Copyright\n=====================\n\nZSchema Copyright 2020 ZMap Team\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use\nthis file except in compliance with the License. You may obtain a copy of the\nLicense at http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software distributed\nunder the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR\nCONDITIONS OF ANY KIND, either express or implied. See LICENSE for the specific\nlanguage governing permissions and limitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzmap%2Fzschema","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzmap%2Fzschema","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzmap%2Fzschema/lists"}