{"id":49387529,"url":"https://github.com/metabase/database-metadata","last_synced_at":"2026-04-28T10:08:26.644Z","repository":{"id":352312362,"uuid":"1212055315","full_name":"metabase/database-metadata","owner":"metabase","description":"The Metabase Database Metadata Format: a human-readable, version-controllable YAML tree for Metabase's synced databases, tables, and fields. Spec, CLI, and examples.","archived":false,"fork":false,"pushed_at":"2026-04-18T23:32:24.000Z","size":204,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-19T00:38:13.646Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metabase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-16T02:42:01.000Z","updated_at":"2026-04-18T23:32:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/metabase/database-metadata","commit_stats":null,"previous_names":["metabase/database-metadata"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/metabase/database-metadata","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metabase%2Fdatabase-metadata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metabase%2Fdatabase-metadata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metabase%2Fdatabase-metadata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metabase%2Fdatabase-metadata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metabase","download_url":"https://codeload.github.com/metabase/database-metadata/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metabase%2Fdatabase-metadata/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32375841,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T09:24:15.638Z","status":"ssl_error","status_checked_at":"2026-04-28T09:24:15.071Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-28T10:08:25.969Z","updated_at":"2026-04-28T10:08:26.638Z","avatar_url":"https://github.com/metabase.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Metabase Database Metadata Format\n\nMetabase represents database metadata — synced databases, their tables, and their fields — as a tree of YAML files. Files are diff-friendly: numeric IDs are omitted entirely, and foreign keys use natural-key tuples like `[\"Sample Database\", \"PUBLIC\", \"ORDERS\"]` instead of database identifiers.\n\nThis repository contains the specification, examples, and a CLI that converts the JSON returned by Metabase's `GET /api/database/metadata` endpoint into YAML.\n\n## Specification\n\nThe format is defined in **[core-spec/v1/spec.md](core-spec/v1/spec.md)** (v1.0.3). It covers entity keys, field types, folder structure, sampled field values, and the shape of each entity.\n\nReference output for the Sample Database lives in **[examples/v1/](examples/v1/)** — both the raw `metadata.json` returned by the endpoint and the extracted YAML tree.\n\n### Entities\n\n| Entity | Description |\n|--------|-------------|\n| Database | A connected data source (Postgres, MySQL, BigQuery, etc.) |\n| Table | A physical table (or view) inside a database |\n| Field | A column on a table, including JSON-unfolded nested fields |\n\n## Obtaining metadata\n\nMetadata is fetched on demand from a running Metabase instance via `GET /api/database/metadata`. The response is a flat JSON document with three arrays — `databases`, `tables`, and `fields` — streamed so that even warehouses with very large schemas can be exported without exhausting server memory.\n\nAuthenticate with an API key (`X-API-Key`) or session token (`X-Metabase-Session`).\n\n### Downloading metadata\n\nThe CLI can fetch `metadata.json`, `field-values.json`, and extract the YAML tree in one streaming pass:\n\n```sh\nexport METABASE_API_KEY=...\nbunx @metabase/database-metadata download-metadata \"$METABASE_URL\"\n```\n\nWith no flags, the command writes:\n\n- `.metabase/metadata.json`\n- `.metabase/field-values.json`\n- `.metabase/databases/` — extracted YAML tree\n\nFlags override any default or opt out of individual steps:\n\n| Flag | Default | Purpose |\n|------|---------|---------|\n| `--metadata \u003cpath\u003e` | `.metabase/metadata.json` | Where to write the raw metadata JSON |\n| `--field-values \u003cpath\u003e` | `.metabase/field-values.json` | Where to write the raw field-values JSON |\n| `--extract \u003cfolder\u003e` | `.metabase/databases` | Where to extract the YAML tree |\n| `--no-field-values` | — | Skip downloading field values |\n| `--no-extract` | — | Skip YAML extraction |\n| `--api-key \u003ckey\u003e` | `METABASE_API_KEY` env var | API key |\n\nFiles are streamed to disk directly — responses are never fully buffered in memory, so multi-GB exports stay lean.\n\n### Extracting metadata to YAML\n\nIf you already have a `metadata.json` on disk (e.g. downloaded via `curl`), you can skip the download and extract directly:\n\n```sh\nbunx @metabase/database-metadata extract-metadata \u003cinput-file\u003e \u003coutput-folder\u003e\n```\n\n- `\u003cinput-file\u003e` — path to the `metadata.json` produced by the API.\n- `\u003coutput-folder\u003e` — destination directory. Database folders are created directly under it.\n\n### Extracting field values\n\nMetabase keeps a sampled list of distinct values for each field that's low-cardinality enough to enumerate (the same list that powers filter dropdowns in the UI).\n\n```sh\nbunx @metabase/database-metadata extract-field-values \u003cmetadata-file\u003e \u003cfield-values-file\u003e \u003coutput-folder\u003e\n```\n\n- `\u003cmetadata-file\u003e` — the same `metadata.json` used by `extract-metadata`. Field values reference fields by numeric ID, which the CLI resolves to natural keys using the metadata.\n- `\u003cfield-values-file\u003e` — path to the `field-values.json` returned by the endpoint.\n- `\u003coutput-folder\u003e` — destination directory; typically the same one used for `extract-metadata`, so values files land next to the table YAMLs they belong to.\n\nOne YAML file is written per field that has values. Fields with empty samples are skipped; field IDs not present in the metadata are reported as orphans and skipped. See the spec's [Field Values](core-spec/v1/spec.md#field-values) section for the on-disk shape and when agents should consult these files.\n\n### Uploading metadata to a target instance\n\n`upload-metadata` streams the JSON files previously written by `download-metadata` into a target Metabase instance, remapping numeric IDs across multiple NDJSON passes (see [metabase-api-contract.md](metabase-api-contract.md)):\n\n```sh\nexport METABASE_API_KEY=...\nbunx @metabase/database-metadata upload-metadata \"$TARGET_METABASE_URL\"\n```\n\nWith no flags, it reads `.metabase/metadata.json` and `.metabase/field-values.json` — the same layout `download-metadata` writes by default.\n\n| Flag | Default | Purpose |\n|------|---------|---------|\n| `--metadata \u003cpath\u003e` | `.metabase/metadata.json` | Path to the metadata JSON to upload |\n| `--field-values \u003cpath\u003e` | `.metabase/field-values.json` | Path to the field-values JSON |\n| `--no-field-values` | — | Skip uploading field values |\n| `--api-key \u003ckey\u003e` | `METABASE_API_KEY` env var | API key |\n\nThe source JSON files are streamed through `@streamparser/json-node` — they are never fully loaded into memory, so 100 GB+ exports upload fine. Rows are sent in batches of 2000 per HTTP POST (matching the server's per-transaction batch size) with HTTP keep-alive, so each request is one clean server-side transaction.\n\nExits non-zero if any step reports row-level errors, or if the server acknowledges fewer rows than were sent in a batch (so CI can catch partial imports).\n\n### Extracting the spec\n\nThe bundled spec can be extracted to any file — convenient for agents that need to read it locally:\n\n```sh\nbunx @metabase/database-metadata extract-spec --file ./spec.md\n```\n\nOmit `--file` to write `spec.md` into the current directory.\n\n## Recommended workflow\n\nThe following is the **default** workflow for a project that wants to use Metabase metadata. It is a convention, not a requirement — teams are free to organize things differently.\n\n### 1. A `.metabase/` directory at the repo root\n\nCreate a top-level `.metabase/` directory and **add it to `.gitignore`**. This is where the raw `metadata.json` and the extracted `databases/` YAML tree live:\n\n```\n.metabase/\n├── metadata.json\n└── databases/\n    └── …\n```\n\n### 2. Why `.metabase/` should not be committed\n\nOn a large data warehouse the metadata export can easily reach **hundreds of megabytes or several gigabytes**. Committing it:\n\n- bloats the repository and slows every clone and fetch,\n- produces noisy diffs on unrelated PRs whenever someone resyncs,\n- can make the repo effectively unusable for CI and for new contributors.\n\nEach developer (or a CI job) fetches metadata on demand from their own Metabase instance instead.\n\n### 3. Credentials via a gitignored `.env` file\n\nCheck in an **`.env.template`** at the repo root with placeholders:\n\n```env\nMETABASE_URL=https://metabase.example.com\nMETABASE_API_KEY=\n```\n\nEach developer copies it to `.env` (also gitignored) and fills in the real values:\n\n```sh\ncp .env.template .env\n# edit .env to set METABASE_URL and METABASE_API_KEY\n```\n\n### 4. Fetch and extract on demand\n\nWith `.env` populated, the end-to-end flow is a single command:\n\n```sh\nset -a; source .env; set +a\n\nrm -rf .metabase/databases\nbunx @metabase/database-metadata download-metadata \"$METABASE_URL\"\n```\n\nThat downloads `.metabase/metadata.json`, `.metabase/field-values.json`, and extracts the YAML tree into `.metabase/databases/`. Use `--no-field-values` or `--no-extract` to skip parts of the pipeline.\n\nAfter this, tools and agents should read the YAML tree under `.metabase/databases/` — not `metadata.json` or `field-values.json`, which exist only as input to the extractors.\n\n## Publishing to NPM\n\nReleases are published automatically by the **Release to NPM** GitHub Actions workflow on every push to `main`. The workflow compares the `version` in `package.json` against the version published on npm and publishes (with the `latest` dist-tag) if they differ.\n\nTo cut a release, bump `version` in `package.json` and merge to `main`.\n\nThe workflow requires an `NPM_RELEASE_TOKEN` secret with publish access to the `@metabase` npm org.\n\n## Development\n\n```sh\nbun install\nbun bin/cli.ts extract-metadata examples/v1/metadata.json /tmp/.metabase/databases\n```\n\n### Scripts\n\n- `bun run build` — compile TypeScript to `dist/` and bundle the spec.\n- `bun run type-check` — `tsc --noEmit`.\n- `bun run lint-eslint` — ESLint with no warnings allowed.\n- `bun run lint-format` — oxfmt format check.\n- `bun run test` — bun test suite.\n\nThe **Lint**, **Test**, and **Validate** GitHub workflows run on every push and pull request. **Validate** regenerates the bundled examples and fails if they drift from what's checked in.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetabase%2Fdatabase-metadata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetabase%2Fdatabase-metadata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetabase%2Fdatabase-metadata/lists"}