{"id":51611635,"url":"https://github.com/cloud3000/opensource-data-dict","last_synced_at":"2026-07-12T08:31:10.150Z","repository":{"id":364874611,"uuid":"1269542438","full_name":"cloud3000/opensource-data-dict","owner":"cloud3000","description":"Open-source Business Data Dictionary in SQLite — 3,051 standardized data items across 12 business categories, aggregated and cross-referenced from 9   public/open standards ","archived":false,"fork":false,"pushed_at":"2026-06-27T09:36:19.000Z","size":2797,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-27T10:06:23.473Z","etag":null,"topics":["business-data","data-catalog","data-dictionary","data-modeling","erp","fhir","gs1","healthcare","isa-95","master-data","metadata","open-data","openapi","python","schema","schema-org","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloud3000.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-14T20:57:32.000Z","updated_at":"2026-06-27T09:36:21.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cloud3000/opensource-data-dict","commit_stats":null,"previous_names":["cloud3000/opensource-data-dict"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cloud3000/opensource-data-dict","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloud3000%2Fopensource-data-dict","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloud3000%2Fopensource-data-dict/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloud3000%2Fopensource-data-dict/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloud3000%2Fopensource-data-dict/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloud3000","download_url":"https://codeload.github.com/cloud3000/opensource-data-dict/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloud3000%2Fopensource-data-dict/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35387180,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-12T02:00:06.386Z","response_time":87,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["business-data","data-catalog","data-dictionary","data-modeling","erp","fhir","gs1","healthcare","isa-95","master-data","metadata","open-data","openapi","python","schema","schema-org","sqlite"],"created_at":"2026-07-12T08:31:09.369Z","updated_at":"2026-07-12T08:31:10.140Z","avatar_url":"https://github.com/cloud3000.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Business Application Data Dictionary\n\n[![Build](https://github.com/cloud3000/opensource-data-dict/actions/workflows/build.yml/badge.svg)](https://github.com/cloud3000/opensource-data-dict/actions/workflows/build.yml)\n[![Descriptions](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fcloud3000%2Fopensource-data-dict%2Fmain%2Fdiagrams%2Fcoverage-badge.json)](DATA_MODEL.md#3-description-coverage--provenance)\n[![Code license: MIT](https://img.shields.io/badge/code-MIT-blue)](LICENSE)\n[![Data license: CC BY-SA 4.0](https://img.shields.io/badge/data-CC_BY--SA_4.0-blue)](DATA_LICENSE)\n\nA comprehensive, **open-source** business data dictionary stored in SQLite\n(`datadict.db`, with a full schema + data dump in `datadict.sql`). It collects\nstandardized business data items across major industries and functional areas,\nextracted **only from public / open-source resources**.\n\n\u003e **3,688 data items · 12 categories · 9 open-source standards** \u0026nbsp;|\u0026nbsp;\n\u003e 31 items corroborated across multiple sources.\n\n---\n\n## Contents by category\n\n| Category | Items | Category | Items |\n|---|---:|---|---:|\n| Manufacturing | 648 | Supply Chain / Logistics | 226 |\n| Finance / Accounting | 612 | Human Resources | 195 |\n| Sales / Order Management | 538 | Inventory / Warehouse | 131 |\n| Customer Relationship Management (CRM) | 484 | Procurement / Purchasing | 111 |\n| Healthcare | 331 | Quality Management | 81 |\n| Product Master Data | 299 | Maintenance / Asset Management | 32 |\n\n## Sources (all public / open-source)\n\n| Standard | License | Items |\n|---|---|---:|\n| Microsoft Common Data Model (CDM) | CDLA-Permissive-2.0 | 712 |\n| Tryton | GPL-3.0 | 634 |\n| ERPNext / Frappe Health | GPL-3.0 | 610 |\n| Odoo | LGPL-3.0 | 546 |\n| Stripe API (OpenAPI) | MIT | 393 |\n| Schema.org | CC BY-SA 3.0 | 373 |\n| GS1 (Barcode Syntax Dictionary) | Apache-2.0 | 220 |\n| ISA-95 / B2MML | Royalty-free (MESA) | 122 |\n| HL7 FHIR (R4) | CC0 (public domain) | 116 |\n\nFull provenance and extraction notes: **[`sources.md`](sources.md)**.\n\n## Diagrams\n\nSchema and source/category maps live in **[`DATA_MODEL.md`](DATA_MODEL.md)**\n(Mermaid, renders on GitHub) with static **SVG/PNG** exports in\n[`diagrams/`](diagrams/):\n\n| | |\n|---|---|\n| [ER diagram](diagrams/er-diagram.svg) | two-table star: `Categories` → `DataItems` |\n| [Categories](diagrams/categories.svg) | items per category (pie) |\n| [Description coverage](diagrams/description-coverage.svg) | description provenance: from-source vs curated (pie) |\n| [Source→Category map](diagrams/source-category-map.svg) | which standards feed which categories |\n\n![Categories](diagrams/categories.png)\n\nEvery item is described (**100% coverage**) — 3,234 descriptions come straight\nfrom the upstream source and 454 are curated editorial text added where the\nsource provided none:\n\n![Description coverage](diagrams/description-coverage.png)\n\n---\n\n## Quick start\n\n```bash\n# Build / refresh datadict.db and datadict.sql from the seed modules\npython3 build_dict.py\n\n# Print summary statistics only (no rebuild)\npython3 build_dict.py --stats\n\n# Query the database\nsqlite3 datadict.db \"SELECT * FROM DataItems WHERE CategoryID = 1 LIMIT 10;\"\n```\n\nThe build is **idempotent** — re-running never creates duplicates. To rebuild\nfrom scratch, delete `datadict.db` first (it is fully regenerable from the\nseeds). See the **[Query Cookbook](QUERY_COOKBOOK.md)** for ready-to-run SQL.\n\n---\n\n## Find fields by business term\n\n`tools/find.py` looks up data items by a plain business word and tells you how\nit matched. A term resolves through four layers, first hit wins:\n\n1. **category** — `inventory`, `accounting`, `quality` → every item in that category\n2. **entity** — `patient`, `work_order`, `invoice` → that object's field family\n3. **alias** — business vocabulary the source systems don't name directly\n   (`billing`, `payee`, `shipping`, `appointment`, …) is mapped to real\n   entities via a reviewable `SEARCH_ALIASES` map (~55 terms)\n4. **keyword** — substring fallback across name/title/description (flagged as scattered)\n\n```bash\npython3 tools/find.py billing                 # alias -\u003e invoice/charge/...\npython3 tools/find.py patient invoice          # multiple terms (union)\npython3 tools/find.py appointment --ddl        # emit a CREATE TABLE\npython3 tools/find.py inventory --limit 0      # list every match\n```\n\n```text\n$ python3 tools/find.py billing --limit 3\n=== 'billing'  [~ alias -\u003e [invoice, invoiceitem, charge, payment_intent]]  382 items, 5 entities, 1 categories\n    invoice.account              INTEGER   Account\n    invoice.account_country      VARCHAR   account_country\n    invoice.account_id           VARCHAR   Account\n    ... +379 more\n```\n\nThe **`--ddl`** flag turns a match into a ready-to-run `CREATE TABLE` per\nentity — types, lengths, `NOT NULL`, and enum `CHECK`s all come from the\ndictionary metadata — handy for scaffolding an app schema:\n\n```sql\n$ python3 tools/find.py appointment --ddl\n-- patient_appointment (48 fields)\nCREATE TABLE patient_appointment (\n    patient_appointment_id INTEGER PRIMARY KEY,\n    appointment_date  DATE NOT NULL,\n    appointment_for   VARCHAR(255) NOT NULL\n                        CHECK (appointment_for IN ('Practitioner', 'Department', 'Service Unit')),\n    ...\n);\n```\n\nBusiness terms that don't yet resolve well are a one-line addition to\n`SEARCH_ALIASES` in `tools/find.py`.\n\nFor a **full multi-table schema with foreign keys**, use `tools/export_ddl.py`.\nIt resolves the same terms, then emits one `CREATE TABLE` per entity with real\n`FOREIGN KEY` constraints derived from the dictionary's relationship hints,\nordered so the DDL loads, in your choice of dialect:\n\n```bash\npython3 tools/export_ddl.py manufacturing                 # SQLite (self-validated)\npython3 tools/export_ddl.py billing patient --dialect postgres\npython3 tools/export_ddl.py inventory --dialect mysql --out schema.sql\n```\n\n---\n\n## Schema\n\n```sql\nCREATE TABLE Categories (\n    CategoryID  INTEGER PRIMARY KEY,\n    Name        TEXT NOT NULL UNIQUE,   -- \"Manufacturing\", \"Finance\", ...\n    Description TEXT,\n    Source      TEXT\n);\n\nCREATE TABLE DataItems (\n    DataItemID   INTEGER PRIMARY KEY,\n    CategoryID   INTEGER NOT NULL REFERENCES Categories(CategoryID),\n    Name         TEXT NOT NULL,         -- normalized \"entity.field\" (snake_case)\n    Title        TEXT,\n    Description  TEXT,\n    DataType     TEXT,                  -- VARCHAR, INTEGER, DECIMAL, DATE, ...\n    ByteLength   INTEGER,\n    DecimalScale INTEGER,\n    IsRequired   BOOLEAN DEFAULT FALSE,\n    IsNullable   BOOLEAN DEFAULT TRUE,\n    DefaultValue TEXT,\n    AllowedValues TEXT,                 -- JSON array, or per-source JSON object\n    FormatMask    TEXT,                 -- e.g. \"YYYY-MM-DD\", GS1 \"N14,csum\"\n    SourceStandard TEXT,                -- one or more \"; \"-joined standards\n    SourceURL      TEXT,                -- one or more \" | \"-joined URLs\n    Version        TEXT,\n    CreatedAt DATETIME DEFAULT CURRENT_TIMESTAMP,\n    UpdatedAt DATETIME DEFAULT CURRENT_TIMESTAMP\n);\n\nCREATE INDEX idx_dataitems_category ON DataItems(CategoryID);\nCREATE INDEX idx_dataitems_name     ON DataItems(Name);\n-- Idempotency key for upserts:\nCREATE UNIQUE INDEX ux_dataitems_natural\n    ON DataItems(CategoryID, Name, SourceStandard);\n```\n\n### Two field conventions worth knowing\n\n- **`Name`** is normalized to `entity.field` in **snake_case** (e.g.\n  `account.account_id`, `product.gtin`). The entity is everything before the\n  last dot.\n- **`SourceStandard` / `SourceURL`** hold a **`; `-** / **` | `-joined list**\n  when an item was corroborated by more than one source (a cross-source merge).\n- **`AllowedValues`** is usually a **JSON array** (`[\"male\",\"female\",\"other\"]`).\n  When merged sources have *divergent* enums, it becomes a **JSON object keyed\n  by source** (e.g. `purchase_order.state`). Consumers should handle both.\n\n---\n\n## UI projection (`ui_datadict.db`)\n\nA second, **derived** database aimed at UI / form generation and resource\ngovernance. `tools/build_ui_dict.py` reads `datadict.db` (read-only) and rebuilds\n`ui_datadict.db` + `ui_datadict.sql` from scratch:\n\n```bash\npython3 tools/build_ui_dict.py\nsqlite3 ui_datadict.db \"SELECT * FROM UI_DataItems LIMIT 5;\"\n```\n\nIt reshapes the dictionary into a 3-level hierarchy — **`Categories → Groups →\nUI_DataItems`** (12 categories, **120 groups**, 3,688 items):\n\n- **`Groups`** — one per entity (the part before the last dot of `Name`). The\n  43 transient wizard / relation (junction) entities collapse into a single\n  **`Wizard`** group; `gs1` is kept. Each group carries its `CategoryID`.\n- **`UI_DataItems`** — `Name` is reduced to just the field; every item is treated\n  as **UTF-8** with an *implied* `DataType`, plus governance columns:\n  - **`CharLength`** — declared VARCHAR length, else a per-type default\n  - **`ByteLength`** = `CharLength × 4` (UTF-8 worst case)\n  - **`ValidationSpecs`** — the source mask (e.g. GS1) when present, else\n    generated from type / allowed values / scale\n\n| Groupname | Name | DataType | CharLength | ByteLength | ValidationSpecs |\n|---|---|---|--:|--:|---|\n| patient | dob | DATE | 10 | 40 | `^\\d{4}-\\d{2}-\\d{2}$` |\n| patient | first_name | VARCHAR | 255 | 1020 | `maxlength: 255` |\n| patient | blood_group | VARCHAR | 255 | 1020 | `one of: A Positive\\|A Negative\\|…` |\n\n`datadict.db` is never modified; CI rebuilds and validates `ui_datadict.db`\n(completeness, FK integrity, `ByteLength = CharLength*4`, unique `(GroupID, Name)`).\n\n---\n\n## How it's built\n\n```\nbuild_dict.py          # orchestrator: schema, load seeds, normalize, export, stats\nnormalize.py           # Phase 3: snake_case naming, entity/field aliases,\n                        #          cross-source merge, per-source enum union\nseeds/                 # one module per source (CATEGORIES + ITEMS); some\n                        #   are auto-generated by the tools below\ntools/fetch_*.py       # generators that (re)build seed modules from upstream\ndatadict.db            # the SQLite database (build output)\ndatadict.sql           # full schema + INSERTs (build output)\nsources.md             # every source, license, and extraction method\nNORMALIZATION_REPORT.md # auto-generated: aliases, merges, related concepts\nDATA_MODEL.md          # auto-generated Mermaid ER + category/source diagrams\nQUERY_COOKBOOK.md      # ready-to-run SQL recipes\nPROGRESS.md            # running build log\n```\n\n### Pipeline (4 phases)\n\n1. **Discovery** — explore each open repo/spec for business entities.\n2. **Extraction** — `tools/fetch_*.py` parse upstream schemas (XSD, JSON-LD,\n   Python `ast`, DocType JSON, OpenAPI, FHIR StructureDefinitions) into seed\n   modules. Each is re-runnable to refresh from upstream.\n3. **Normalization** (`normalize.py`) — consistent snake_case names; deliberate,\n   reviewable **entity aliases** (e.g. `account.invoice` → `invoice`) and\n   **field aliases** (e.g. `gs1.gtin` → `product.gtin`); conservative\n   cross-source **merge** (same `entity.field` in the same category) that keeps\n   the richest metadata and records all sources; per-source **enum union**.\n4. **SQLite generation** — `datadict.db` + `datadict.sql`, plus stats and the\n   normalization report.\n\n### Regenerating a single source\n\n```bash\npython3 tools/fetch_cdm.py        # Microsoft CDM   -\u003e seeds/cdm_microsoft.py\npython3 tools/fetch_odoo.py       # Odoo            -\u003e seeds/odoo.py\npython3 tools/fetch_frappe.py     # ERPNext+Health  -\u003e seeds/frappe.py\npython3 tools/fetch_schemaorg.py  # Schema.org      -\u003e seeds/schemaorg.py\npython3 tools/fetch_tryton.py     # Tryton          -\u003e seeds/tryton.py\npython3 tools/fetch_gs1.py        # GS1             -\u003e seeds/gs1.py\npython3 tools/fetch_fhir.py       # HL7 FHIR        -\u003e seeds/fhir.py\npython3 tools/fetch_openapi.py    # OpenAPI (Stripe)-\u003e seeds/openapi.py\npython3 build_dict.py             # then rebuild the DB\npython3 tools/gen_diagram.py      # refresh DATA_MODEL.md from the DB\npython3 tools/render_diagrams.py  # render diagrams/*.svg + *.png (Chromium or mmdc)\n```\n\n(ISA-95 / B2MML lives in a hand-curated `seeds/isa95_b2mml.py`.)\n\n### Adding a new source\n\nCreate a `seeds/\u003cname\u003e.py` exposing two lists — `CATEGORIES` and `ITEMS`\n(each item a dict with at least `category` and `name`) — then run\n`python3 build_dict.py`. For OpenAPI/Swagger specs, just add an entry to\n`SPECS` in `tools/fetch_openapi.py`.\n\n---\n\n## Requirements\n\nPython 3 standard library only (`sqlite3`, `json`, `ast`, `urllib`,\n`re`). No third-party packages. The `tools/fetch_*.py` generators need network\naccess to reach the upstream repos; `build_dict.py` works fully offline from\nthe committed seed modules.\n\n## Contributing\n\nContributions — new open-source sources, corrections, better descriptions,\ntooling — are welcome. See **[CONTRIBUTING.md](CONTRIBUTING.md)** for the\nseed-module contract, how to add a source, the aliasing/merge rules, and the\npre-PR checklist. Core rule: **public/open-source sources only, accuracy over\nquantity, always record provenance.**\n\n## Changelog\n\nRelease-facing changes are in **[CHANGELOG.md](CHANGELOG.md)** (latest release:\n**[v1.3.0](https://github.com/cloud3000/opensource-data-dict/releases/tag/v1.3.0)**).\nThe granular development log lives in [`PROGRESS.md`](PROGRESS.md).\n\n## Contributors\n\n- **Michael Anderson** ([@cloud3000](https://github.com/cloud3000)) — creator \u0026 maintainer\n- **Claude** (Anthropic) — AI pair-programmer; contributions are co-authored via\n  `Co-Authored-By` trailers in the commit history\n\nThe full, always-current list is on the\n[contributors graph](https://github.com/cloud3000/opensource-data-dict/graphs/contributors).\nNew contributors welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).\n\n## License \u0026 attribution\n\nThis repository is **dual-licensed** to reflect its two layers:\n\n- **Code** — the build/generator scripts (`build_dict.py`, `normalize.py`,\n  `tools/*.py`) and the project's documentation are licensed under the\n  **[MIT License](LICENSE)**.\n- **Data** — `datadict.db` / `datadict.sql` is a compilation derived from\n  multiple open sources. The original compilation, arrangement, and this\n  project's own descriptions are licensed under **[Creative Commons\n  Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)](DATA_LICENSE)**.\n\n**Upstream terms still apply per item.** Individual data items remain subject\nto the license of the source they came from, recorded in each row's\n`SourceStandard` / `SourceURL` and summarized in [`sources.md`](sources.md):\nHL7 FHIR (CC0), Stripe (MIT), GS1 (Apache-2.0), Microsoft CDM\n(CDLA-Permissive-2.0), Schema.org (CC BY-SA 3.0), Odoo (LGPL-3.0),\nERPNext/Frappe Health \u0026 Tryton (GPL-3.0), ISA-95/B2MML (MESA royalty-free).\nIf you redistribute the data, **provide attribution** (the per-item source\nfields + `sources.md` satisfy this) and honor each upstream license —\nparticularly Schema.org's ShareAlike and the copyleft (GPL) sources.\n\nNo paywalled, proprietary, or scraped commercial content is included.\n\n\u003e Not legal advice. This dual-license setup is a good-faith, conservative\n\u003e reflection of the sources; consult a professional for your specific use.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloud3000%2Fopensource-data-dict","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloud3000%2Fopensource-data-dict","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloud3000%2Fopensource-data-dict/lists"}