{"id":14531180,"url":"https://github.com/ananthdurai/schemata","last_synced_at":"2026-01-16T14:06:22.654Z","repository":{"id":40524173,"uuid":"440730070","full_name":"ananthdurai/schemata","owner":"ananthdurai","description":"Schema modelling framework for decentralised domain-driven ownership of data.","archived":false,"fork":false,"pushed_at":"2023-12-05T22:11:41.000Z","size":619,"stargazers_count":234,"open_issues_count":10,"forks_count":15,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-04-20T12:54:08.647Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ananthdurai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null}},"created_at":"2021-12-22T04:17:32.000Z","updated_at":"2024-04-11T12:11:08.000Z","dependencies_parsed_at":"2023-01-19T20:15:43.596Z","dependency_job_id":null,"html_url":"https://github.com/ananthdurai/schemata","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ananthdurai%2Fschemata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ananthdurai%2Fschemata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ananthdurai%2Fschemata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ananthdurai%2Fschemata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ananthdurai","download_url":"https://codeload.github.com/ananthdurai/schemata/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":217598107,"owners_count":16201827,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-05T00:01:12.614Z","updated_at":"2026-01-16T14:06:17.618Z","avatar_url":"https://github.com/ananthdurai.png","language":"Java","funding_links":[],"categories":["Tools","大数据"],"sub_categories":["Spring Cloud框架"],"readme":"\u003cimg align=\"center\" src=\"./asset/logo.png\" alt=\"Schemata logo\"/\u003e\n\n# Schemata\n\nSchemata is a schema modeling framework for decentralized domain-driven ownership of data. Schemata combines a set of\nstandard metadata definitions for each schema \u0026 data field and a scoring algorithm to provide a feedback loop on how\nefficient the data modeling of your data warehouse is. Schemata support ProtoBuf \u0026 Avro formats.\n\n\u003cp\u003e\n    \u003ca href=\"https://github.com/ananthdurai/schemata/releases/\"\u003e\u003cimg src=\"https://img.shields.io/github/v/release/ananthdurai/schemata?color=orange\" alt=\"Latest Release\"\u003e\u003c/a\u003e   \n    \u003ca href=\"https://github.com/pdevito3/craftsman/blob/master/LICENSE.txt\"\u003e\u003cimg src =\"https://img.shields.io/github/license/ananthdurai/schemata\" alt=\"License\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://twitter.com/ananthdurai\" target=\"\\_parent\"\u003e\n    \u003cimg alt=\"\" src=\"https://img.shields.io/twitter/url?style=social\u0026url=https%3A%2F%2Ftwitter.com%2Fananthdurai\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n------\n\nBefore we jump on let's walk through Garbage-In Garbage-Out (GIGO) problem in the data lake\n\n# The Garbage-In Garbage-Out (GIGO) Problem\n\nA data Warehouse is a centralized repository that allows you to store all your structured and unstructured data at any scale.\nAs the definition suggests, Data Warehouse focuses on centralized data storage to break the organization’s data silo. The\ncentral repository removes entry barriers to integrating and analyzing various data sources in an organization.\nHowever, as the data warehouse grows, the complexity of the data management grows.\n\n1. The data producer generates data and sends it to the data lake.\n2. The consumers down the line have no domain understanding of the producer and struggle to understand the data lake\n   data.\n3. The consumers then connect with the data producer to understand the data. The producer side’s domain\n   expertise depends on human knowledge that may or may not be available.\n\nData Warehouse becomes a technical debt rather than a strategic advantage as it grows.\n\n# How do Schemata solve the Garbage-In Garbage-Out (GIGO) problem?\n\n## Schemata enables domain-oriented data ownership\nSchemata focuses on treating data as a product. The feature team that works on the product feature has the domain understanding of the data, not the data's consumer. \nSchemata enables a feature team to create, attach metadata, catalog the data, and store it for easier consumption.\n\nThe data curation and the cataloging of the data at the data creation phase bring more visibility and make it easier for consumption. \nThe process also eliminates the human knowledge silo and truly democratizes the data. \nIt helps the data consumers not worry about the data discovery and focuses on producing value from the data.\n\n## Schemata facilitates decentralized data modeling\n\nTraditionally upfront data modeling came with a cost. Often a centralized data architecture/ modeling team coordinates with multiple teams to design an enterprise data modeling. \nIt is hard for one individual human to hold the entire company's data architecture in their head. \nThe data modeling tools don't reflect the current state of the data modeling. \nDecentralized data modeling is the only scalable approach, and Schemata enables the bottom-up crowdsourcing data modeling approach to democratize data access in an organization.\n\n## Schemata brings DevOps principles to data modeling \n\nThe decentralized data modeling principle brings a unique collaborative approach to managing the data asset's lifecycle. \nIt brings all the proven devops principles like ownership, accountability, collaboration, automation, continuous improvement, and customer-centric action to the data management.\n\n## Schemata ensures the connectivity \u0026 integrity of the data model\nData is inherently social in nature. The significant challenge of decentralized data management is that the lack of connectivity among the data will degrade its usability of the data. \nSchemata is an opinionated data modeling framework that programmatically measures the connectivity of the data model and assigns a score to it. We call this Schemata Score.\n\nObservability metrics like SLO \u0026 Apdex Score inspired the formation of Schemata Score. \nA lower the Schemata Score means lesser the data connectivity of a data model. \nIt allows the teams collaboratively fix the data model and bring uniformity to the data.\n\n## WIP: Schemata Ruby on Rails Experience for Data Engineering\n\n🚧 This is still under development. Watch this space for more details soon.\n\n# Design\n\nSchemata frameworks contain two parts.\n\n📘 **Schema metadata annotations:**\n\nThe metadata annotations enrich the context of the schema definitions. It enforces a few mandatory metadata fields such\nas the owner of the schema, the domain it represents, and further classification of the Schema into Entity stream \u0026\nevent stream.\n\n🎼 **Schemata Score:**\n\nA ranking function parses all the metadata and assigns a score for each Schema definition to define how integrated the\nSchema design is and validate if all the Schema definition adheres to the Schemata metadata annotations.\n\n## Schema Metadata\n\n### Core Metadata (shared across Schema and Fields)\n\n```protobuf\n// CoreMetadata is the set of attribute apply to both the Message \u0026 Field\nmessage CoreMetadata {\n  // Mandatory Metadata: description of the entity\n  optional string description = 50001;\n  // Optional Metadata: additional comments about the entity\n  optional string comment = 50002;\n  // Optional Metadata: Any related entity that has \"hierarchy\" or \"has a\"  relationships.\n  optional string see_also = 50003;\n  // Optional Metadata: Additional link reference for further reading.\n  // It could be a confluent page, An ADR or RFC or a Slack message link.\n  optional string reference = 50004;\n}\n```\n\n### Schema Metadata\n\n```protobuf\nextend google.protobuf.MessageOptions {\n\n  // message.description is a Mandatory Metadata\n  CoreMetadata message_core = 60001;\n  // Mandatory Metadata: owner of the entity. Usually it is the team name.\n  string owner = 60002;\n  // Mandatory Metadata: domain = 'core' indicates the entity is common across all the domains.\n  // Other possible domains are `sales`, `marketing`, `product` etc\n  string domain = 60003;\n  // Mandatory Metadata: define the type of the message.\n  Type type = 60004;\n  // Status of the entity. You can have `testing`, `production` or `staging` depends on the lifecycle of schema definition.\n  string status = 60005;\n  // Slack or Teams channel name to communicate with the team which owns ths entity\n  string team_channel = 60006;\n  // Slack or Teams channel name to alert for any validation errors.\n  string alert_channel = 60007;\n  // Type of the event. Set if the Type = 'EVENT'\n  EventType event_type = 60008;\n}\n```\n\n### Field Metadata\n\n```protobuf\nextend google.protobuf.FieldOptions {\n  // message.description is a Mandatory Metadata\n  CoreMetadata field_core = 70001;\n  // Set true if the field contains classified data (Optional).\n  bool is_classified = 70002;\n  // Set the classification level if is_classified is true (This is Mandatory if is_classified set to true)\n  string classification_level = 7003;\n  // Specify the product type. product_type is an useful annotation to represent a field in a business perspective.\n  // (e.g) user_id can be an INT field, but in the system design it could represent External Users rather than internal users.\n  string product_type = 70004;\n  // Set true if the field is a primary key.\n  bool is_primary_key = 70005;\n}\n```\n\n## Schema Classification\n\n\u003cimg src=\"./asset/ds_classification.png\" alt=\"Schema Classification\"/\u003e\n\nAt any point in time, the data producer should provide two types of data products.\n\n### Entity\n\nEntity streams represent the current state of the Entity. In the classical Data Warehouse concepts, Entities typically\nrepresent the dimensions. \n\n**Sample Entity Definition**\n\n```protobuf\nmessage User {\n\n  option(message_core).description = \"This is the description of the users table\";\n  option(message_core).comment = \"The comment added after thought\";\n  option(message_core).see_also = \"db.user MySQL table\";\n  option(owner) = \"Platform\";\n  option(domain) = \"Core\";\n  option(type) = ENTITY;\n  option(team_channel) = \"#team-platform\";\n  option(alert_channel) = \"#alerts-platform\";\n\n  int32 id = 1\n  [(field_core).description = \"Unique identifier for User\", (is_primary_key) = true];\n\n  string name = 2\n  [(field_core).description = \"Name of the user\"] ;\n\n  string email = 3\n  [(field_core).description = \"email id for the user\", (product_type) = \"username\", (is_classified) = true, (classification_level) = \"LEVEL1\"] ;\n\n  bool is_active = 4\n  [(field_core).description = \"define the active status of the user. `true` == active; `false` = inactive`\", (field_core).comment = \"should refactor to non-binary status\"];\n\n  string timezone = 5\n  [(field_core).description = \"preferred time zone for the user\"] ;\n}\n```\n\n### Event\n\nEvent streams are typically immutable. Event streams represent the state change of an Entity. In the classical data\nwarehouse concepts, Event streams represent the facts. Event streams will not have a primary key field.\n\nEvents classified further into three types.\n\n#### Type 1: Lifecycle\n\nLifecycle event captures the state changes of an Entity. (e.g) User created, User deleted et al.\n\n**Sample Lifecycle Event**\n\n```protobuf\n\nenum ActivityType {\n  CREATED = 0;\n  DELETED = 1;\n  UPDATED = 2;\n}\nmessage UserEvent {\n  option(message_core).description = \"This is the description of the users table\";\n  option(owner) = \"Platform\";\n  option(domain) = \"Core\";\n  option(type) = EVENT;\n  option(event_type) = LIFECYCLE;\n  option(team_channel) = \"#team-platform\";\n  option(alert_channel) = \"#alerts-platform\";\n\n  User previous_user_state = 1\n  [(field_core).description = \"Previous version of the user entity before the mutation\"];\n\n  User current_user_state = 2\n  [(field_core).description = \"Current version of the user entity before the mutation\"];\n\n  ActivityType activity_type = 3\n  [(field_core).description = \"Lifecycle event type for the Users table\"];\n\n  google.protobuf.Timestamp timestamp = 4 [(field_core).description = \"Timestamp of the activity\"];\n}\n```\n\n#### Type 2: Activity\n\nACTIVITY event captures the events that resulted from one Entity changing the state of another Entity.\n(e.g.) User A purchases Product B. The ACTIVITY event is often the result of a business transaction.\n\n**Sample ACTIVITY Event**\n\n```protobuf\nenum UserActivityType {\n  VIEW = 0;\n  READ_REVIEW = 1;\n  VIEW_DESCRIPTION = 2;\n}\nmessage UserActivityEvent {\n  option(message_core).description = \"This is the description of the users table\";\n  option(owner) = \"Product\";\n  option(domain) = \"Growth\";\n  option(type) = EVENT;\n  option(event_type) = ACTIVITY;\n  option(team_channel) = \"#team-growth\";\n  option(alert_channel) = \"#alerts-growth\";\n  User user = 1 [(field_core).description = \"User entity reference\"];\n  Product product = 2 [(field_core).description = \"Product entity reference\"];\n  UserActivityType activity_type = 3 [(field_core).description = \"Type of the user activity\"];\n  google.protobuf.Timestamp timestamp = 4 [(field_core).description = \"Timestamp of the activity\"];\n}\n```\n\n#### Type 3: Aggregated\n\nAggregated event captures the computed metrics over a specified window of time. (e.g) Number of views by a User for a\nProduct.\n\n**Sample Aggregated Event**\n\n```protobuf\nenum TimeUnit {\n  SECONDS = 0;\n  MINUTES = 1;\n  HOURS = 2;\n}\nmessage UserActivityAggregate {\n\n  option(message_core).description = \"This is the aggregated user activity view count. The event aggregated by user \u0026 product\";\n  option(owner) = \"Product\";\n  option(domain) = \"Growth\";\n  option(type) = EVENT;\n  option(event_type) = AGGREGATED;\n  option(team_channel) = \"#team-growth\";\n  option(alert_channel) = \"#alerts-growth\";\n\n  User user = 1[(field_core).description = \"User entity reference\"];\n  Product product = 2 [(field_core).description = \"Product entity reference\"];\n  int64  count = 3 [(field_core).description = \"Aggregated count of the user activity per product\", (product_type) = \"activity_count\"];\n  int32 windowTime = 4 [(field_core).description = \"Max window time for the aggregation\"];\n  TimeUnit window_time_unit = 5 [(field_core).description = \"TimeUnit of window for the aggregation\"];\n  google.protobuf.Timestamp timestamp = 6 [(field_core).description = \"Timestamp of the activity\"];\n\n}\n```\n\n# Schema Score:\n\n## The goal of Schemata Score:\n\nSchemata Score is the core part of establishing a feedback loop to maintain the integrity of the decentralized domain\nownership to build data products. In a decentralized data management world, The feature teams (domain owners) define the\nSchema to track the Events \u0026 Entities. Often it goes to a central schema group to validate the Schema since the feature\nteam visibility is limited to its domain. It brings the human into the loop and kills the purpose of distributed data\nownership. The workflow is also harder for a centralized team since it is hard for one human to keep the entire\norganization's Schema in their head.\n\nThe intuition behind the Score is to see if we can programmatically find out which event or entity is less connected in\nthe Schema to improve the connectivity of Schema. Schemata Score provides schema abstraction and the feedback loop to\nhelp model the Schema with less communication overhead.\n\nSchemata construct a Directed Weighted MultiGraph to represent the Schema definitions (Entity \u0026 Events). The Graph walk\nalgorithm derives the Schemata Score indicating how connected each entity is.\n\n## How it works?\n\nLet's take a sample schema and walk through how Schemata score is computed.\n\n\u003cimg src=\"./asset/sample_schema.png\" alt=\"Schema Modeling\"/\u003e\n\n## How the Schemata score computed?\n\nEvery type of schema have its own unique properties. So we can't apply the same scoring technique to each type if\nschema.\n\n### Entity Score:\n\n```math\nScore = 1 - ((Total Incoming Edges + Total Outgoing Edges)) / Total Edges in the Graph \n```\n\nIf you run the Schemata Score for User you'll get 0.222\n\n```shell\n./score.sh org.schemata.schema.User\nSchemata score for org.schemata.schema.User : 0.222\n```\n\nIf you run the Schemata Score for Product you'll get 0.389\n\n```shell\n./score.sh org.schemata.schema.Product\nSchemata score for org.schemata.schema.Product : 0.389\n```\n\nThe Schemata score indicates that Product Entity much more connected than User Entity\n\n#### Lifecycle Events Score\n\n```math\nScore = Total Outgoing Edges \u003e 1 ? 1 : 0\n```\n\nUser event captures the lifecycle of User Entity. So if you run Schemata Score for UserEvent, it will give you 1.0\n\n```shell\n./score.sh org.schemata.schema.UserEvent\nSchemata score for org.schemata.schema.UserEvent : 1.0\n```\n\n### Activity \u0026 Aggregated Events Score\n\n```math\nScore = 1 - ((Total Outgoing Entity Vertex + Total Outgoing Vertex of all Entity Vertex Connected by ACTIVITY or Aggregated Event) / Total Entity Vertex in the Graph) \n```\n\nIf you run schemata score for CampaignCategoryTrackerEvent you will get **0.4**\n\n```shell\n./score.sh org.schemata.schema.CampaignCategoryTrackerEvent\nSchemata score for org.schemata.schema.CampaignCategoryTrackerEvent : 0.4\n```\n\nif you run schema score for CampaignProductTrackerEvent you'll get **0.8**\n\n```shell\n./score.sh org.schemata.schema.CampaignProductTrackerEvent\nSchemata score for org.schemata.schema.CampaignProductTrackerEvent : 0.8\n```\n\nThe CampaignProductTrackerEvent connect to Product, which has high connectivity with other dimensions such as Brand \u0026\nCategory where CampaignCategoryTrackerEvent is the leaf dimension. The score indicates a clear schema modeling issue.\n\n## Schema Score Classification\n\n***Excellent:*** 0.75 to 1.00 is excellent.\n\n***Good:*** 0.50 to 0.75 is good.\n\n***Requires Attention:*** 0.25 to 0.50 require attention\n\n***Blocker:*** less than 0.25 is a code blocker\n\n# Curious to Try?\n\n## Download and install Protobuf Open Contract definitions\n\nGo to the home page of your project and run the following command\n\n```shell\n/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/ananthdurai/schemata/main/install.sh v1)\"\n```\nIt will add the opencontract/v1/org/schemata/protobuf/schemata.proto file in your project home.\nMake sure you include the schemata.proto path while compiling the proto files.\n(e.g)\n```shell\nprotoc  --proto_path=opencontract/v1/org --proto_path=\u003cyour proto schema directory\u003e --include_imports\n```\n\nThe code ships with an example ProtoBuf schema definition for easier understanding.\n\n## Prerequisites\n\nThe project requires the following dependencies\n\n1. JDK 17\n2. ProtoBuf\n3. Makefile\n4. Maven\n\n## How to execute\n\n🏃 compile the project\n\n```shell\nmake build-all\n```\n\n### Directly via the packaged jar\n\n```\nalias schemata=\"java -jar target/schemata-1.0.jar\"\nschemata --help\nschemata score --source=src/test/resources/descriptors/entities.desc --provider=PROTOBUF org.schemata.schema.CampaignCategoryTrackerEvent\nschemata validate --source=src/test/resources/descriptors/entities.desc --provider=PROTOBUF # this has some validation errors you can inspect\nschemata document --source=src/test/resources/descriptors/entities.desc --provider=PROTOBUF # see JSON representation of schema\n```\n\n### Via convenience scripts\n\n🏃 To validate the schema definition\n\n```shell\n./validate.sh\n```\n\n🏃 To see of Schemata Score\n\n```shell\n./score.sh org.schemata.schema.CampaignProductTrackerEvent\n```\n\n🏃 To see the JSON documentation\n\n```shell\n./document.sh\n```\n\n## Using protobuf descriptors for your own data model\n\nCompile the protobuf descriptors using `protoc` to output\nbinary [google.protobuf.FileDescriptorSet](https://github.com/protocolbuffers/protobuf/blob/b48ba578dd01adfebeb4fac0887db1eeb163e00f/src/google/protobuf/descriptor.proto#L57-L59)\nfiles.\n\n```shell\nprotoc --include-imports --descriptor_set_out=mymodel.desc -I path/to/schema -I path/to/protocol/schemas path/to/schema/**/*.proto\n\n./score.sh validate -s=mymodel.desc -p=PROTOBUF\n```\n\n## Supported Providers:\n\n✅ Avro.\n\n✅ ProtoBuf.\n\n✅ dbt\n\n## InProgress:\n\n🚧 Support for JSON schema.\n\n## TODO:\n\n🚧 Support for Thrift.\n\n🚧 Add visualization layer to show the graph representation of the Schema. (Looking for contributors)\n\n# Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fananthdurai%2Fschemata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fananthdurai%2Fschemata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fananthdurai%2Fschemata/lists"}