{"id":49998649,"url":"https://github.com/jtwebman/bigquery-local","last_synced_at":"2026-05-19T09:01:30.224Z","repository":{"id":358308608,"uuid":"1240878494","full_name":"jtwebman/bigquery-local","owner":"jtwebman","description":"Node.js + DuckDB local emulator for the Google BigQuery REST API. Drop-in for testing, CI, and local dev — with working PATCH.","archived":false,"fork":false,"pushed_at":"2026-05-16T19:34:22.000Z","size":120,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-16T19:37:00.118Z","etag":null,"topics":["bigquery","duckdb","emulator","local-development","nodejs","sql","testing","typescript"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jtwebman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-16T17:23:47.000Z","updated_at":"2026-05-16T19:28:58.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jtwebman/bigquery-local","commit_stats":null,"previous_names":["jtwebman/bigquery-local"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jtwebman/bigquery-local","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtwebman%2Fbigquery-local","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtwebman%2Fbigquery-local/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtwebman%2Fbigquery-local/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtwebman%2Fbigquery-local/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jtwebman","download_url":"https://codeload.github.com/jtwebman/bigquery-local/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtwebman%2Fbigquery-local/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33153665,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T09:28:26.183Z","status":"ssl_error","status_checked_at":"2026-05-17T09:27:52.702Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","duckdb","emulator","local-development","nodejs","sql","testing","typescript"],"created_at":"2026-05-19T09:01:12.132Z","updated_at":"2026-05-19T09:01:30.218Z","avatar_url":"https://github.com/jtwebman.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bigquery-local\n\n[![npm](https://img.shields.io/npm/v/bigquery-local?label=npm)](https://www.npmjs.com/package/bigquery-local)\n[![Docker Hub](https://img.shields.io/docker/v/jtwebman/bigquery-local?label=Docker%20Hub\u0026sort=semver)](https://hub.docker.com/r/jtwebman/bigquery-local)\n[![Image size](https://img.shields.io/docker/image-size/jtwebman/bigquery-local/latest?label=image%20size)](https://hub.docker.com/r/jtwebman/bigquery-local)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n\nA Node.js, Docker-friendly local emulator for the Google BigQuery REST\nAPI, backed by [DuckDB](https://duckdb.org/). Aims to be a **full local\nstand-in for BigQuery** for testing, CI, and local development — any\nBigQuery client (`@google-cloud/bigquery`, the Python client, `bq` CLI,\nJDBC/ODBC drivers) can point at it without code changes. Native arm64\nimage, and `PATCH` on datasets and tables actually mutates state (which\nsome existing emulators don't).\n\nNot production-ready, but the architecture stays close to real BigQuery\non purpose — so this can also be a **migration on-ramp** for projects\nthat want to move off BigQuery onto DuckDB.\n\n\u003e **Status:** v0.6.0 — published to both\n\u003e [Docker Hub](https://hub.docker.com/r/jtwebman/bigquery-local) and\n\u003e [npm](https://www.npmjs.com/package/bigquery-local). See `plan.md`\n\u003e for the v0 plan + full-BigQuery scope appendix, and `BACKLOG.md` for\n\u003e the work items.\n\n---\n\n## Feature status\n\nLegend: ✅ shipped · 🚧 in progress · ⏳ planned for v0 · 🔭 later · ❌ not planned\n\n### REST API\n\n| Resource / endpoint | Status |\n|---|---|\n| `GET /discovery/v1/apis/bigquery/v2/rest` | ✅ |\n| Datasets — `GET`, `POST`, **`PATCH`**, `DELETE` | ✅ |\n| Tables — `GET`, `POST`, **`PATCH`**, `DELETE` | ✅ |\n| `POST .../tables/{t}/insertAll` (streaming inserts) | ✅ |\n| `POST /projects/{p}/queries` (sync query) | ✅ |\n| `POST /projects/{p}/jobs` (jobs.insert) | ✅ |\n| `GET /projects/{p}/jobs/{j}` | ✅ |\n| `GET /projects/{p}/queries/{j}` (getQueryResults) | ✅ |\n| `GET /projects/{p}/datasets` (list, paginated) | ✅ |\n| `GET /projects/{p}/datasets/{d}/tables` (list, paginated) | ✅ |\n| `GET /projects/{p}/jobs` (list w/ stateFilter, time bounds, projection) | ✅ |\n| `POST .../jobs/{j}/cancel`, `DELETE .../jobs/{j}/delete` | ✅ |\n| `GET .../tables/{t}/data` (tabledata.list, paginated, selectedFields) | ✅ |\n| `insertAll` insertId dedup (60s window, per-table) | ✅ |\n| `dryRun: true` on queries + jobs (DuckDB `DESCRIBE`-backed) | ✅ |\n| `insertAll` `templateSuffix` (auto-create target on first hit) | ✅ |\n| Multi-project isolation (same dataset+table id in two projects) | ✅ |\n| `--data-from-yaml` initial seed file | ✅ |\n| `jobs.cancel`, `jobs.delete` | 🔭 |\n| `tabledata.list` | 🔭 |\n| `--data-from-yaml` initial seed | 🔭 |\n| Routines, Models, IAM, Reservations, RowAccessPolicies | 🔭 |\n| Storage Read API (gRPC, Avro/Arrow) | 🔭 |\n| Storage Write API (gRPC) | 🔭 |\n| BigQuery ML, SEARCH, VECTOR_SEARCH | 🔭 |\n| Federated external queries (Bigtable / Spanner / Cloud SQL) | 🔭 |\n\n### SQL features\n\n| Feature | Status |\n|---|---|\n| `SELECT` / `JOIN` (INNER/LEFT/RIGHT/FULL/CROSS) / `WHERE` / `ORDER BY` / `GROUP BY` / `HAVING` / `LIMIT` / `OFFSET` | ✅ |\n| Named parameters (`@name`) with `parameterMode=NAMED` | ✅ |\n| Backtick-quoted refs: `` `dataset.table` ``, `` `project.dataset.table` `` | ✅ |\n| `UNNEST(@arr)` (DuckDB-native) | ✅ |\n| `JSON_VALUE` (with quoted JSON path segments) | ✅ |\n| `TIMESTAMP_ADD`, `TIMESTAMP_SUB`, `CURRENT_TIMESTAMP`, `INTERVAL n {DAY,HOUR,...}` | ✅ |\n| `STARTS_WITH`, `ENDS_WITH` | ✅ |\n| `IS NOT NULL`, `COALESCE`, `IFNULL`, `NULLIF`, `LEAST`, `GREATEST` | ✅ |\n| Subqueries (correlated, scalar, `EXISTS`, `IN`, `ANY`/`SOME`/`ALL`) | ✅ |\n| `WITH` / `WITH RECURSIVE` (CTE) | ✅ |\n| Set ops: `UNION`, `INTERSECT`, `EXCEPT` (ALL / DISTINCT) | ✅ |\n| `SAFE_CAST` → `try_cast` | ✅ |\n| `INSERT INTO … SELECT …` | 🚧 |\n| `INSERT` / `UPDATE` / `DELETE` (single-table) | 🚧 |\n| `JSON_EXTRACT_*` family | 🚧 |\n| Window / analytic functions (`OVER`) | 🔭 |\n| `QUALIFY`, `PIVOT` / `UNPIVOT`, `TABLESAMPLE` | 🔭 |\n| `MERGE` | 🔭 |\n| Wildcard tables (`events_*`, `_TABLE_SUFFIX`) | 🔭 |\n| Scripting (`BEGIN`/`END`, `DECLARE`, `SET`, `IF`, `WHILE`, `CALL`, …) | 🔭 |\n| SQL \u0026 JS UDFs, table-valued functions, stored procedures | 🔭 |\n| Materialized views, snapshots, clones, time travel (`FOR SYSTEM_TIME AS OF`) | 🔭 |\n| Geography (`ST_*`) | 🔭 |\n| BigQuery ML (`CREATE MODEL`, `ML.PREDICT`, …) | 🔭 |\n| `SEARCH()`, `VECTOR_SEARCH` | 🔭 |\n\n### Types\n\n| BQ type | Status | Stored as |\n|---|---|---|\n| `STRING`, `BYTES`, `INT64`, `FLOAT64`, `BOOL` | ✅ | `VARCHAR`, `BLOB`, `BIGINT`, `DOUBLE`, `BOOLEAN` |\n| `TIMESTAMP`, `DATETIME`, `DATE`, `TIME` | ✅ | DuckDB native temporal types |\n| `NUMERIC` | ✅ | `DECIMAL(38,9)` |\n| `BIGNUMERIC` | ✅ | `VARCHAR` (decimal string; DuckDB max precision is 38) |\n| `JSON` | ✅ | DuckDB `JSON` |\n| `ARRAY\u003cT\u003e` / `REPEATED` mode | ✅ | DuckDB `T[]` (LIST) |\n| `STRUCT\u003c…\u003e` / `RECORD` | ✅ | DuckDB `STRUCT(…)` |\n| `GEOGRAPHY` | ✅ | `VARCHAR` (WKT round-trip; no `ST_*` functions) |\n| `INTERVAL` | 🔭 | |\n| `RANGE\u003cT\u003e` | 🔭 | |\n\n### Modes / nullability\n\n| Mode | Status |\n|---|---|\n| `NULLABLE` | ✅ |\n| `REQUIRED` | ✅ |\n| `REPEATED` | ✅ (via DuckDB LIST) |\n\n### Operational\n\n| Capability | Status |\n|---|---|\n| REST on port 9050 | ✅ |\n| gRPC port 9060 (bound, returns UNIMPLEMENTED) | ✅ |\n| `--project`, `--port`, `--grpc-port`, `--database`, `--log-level`, `--log-format` | ✅ |\n| Multi-arch Docker image (`linux/amd64` + `linux/arm64`) | ✅ |\n| Persistent file store (`--database=path.duckdb`) and `:memory:` mode | ✅ |\n| No auth required; accepts any/no credentials. For the official client, use `emulatorGoogleAuth()` (see [Quick start](#pointing-the-bigquery-node-client-at-it)) | ✅ |\n| Accepts both raw (`/projects/...`) and prefixed (`/bigquery/v2/projects/...`) URL shapes | ✅ |\n| Multi-tenant: one server serves any project id (URL-scoped) | ✅ |\n\n---\n\n## Quick start\n\n### Docker\n\n```bash\ndocker run --rm -p 9050:9050 -p 9060:9060 \\\n  jtwebman/bigquery-local:latest\n```\n\nThe default port is `9050`. The container also exposes `9060` for gRPC\n(every RPC returns `UNIMPLEMENTED` — see the [gRPC](#grpc) section).\n\n### npx (no install)\n\n```bash\nnpx bigquery-local --port=9050 --database=./bq.duckdb\n```\n\n### Pointing the BigQuery Node client at it\n\n```ts\nimport { BigQuery } from '@google-cloud/bigquery';\nimport { emulatorGoogleAuth } from 'bigquery-local/auth';\n\nconst bigQuery = new BigQuery({\n  projectId: 'local',\n  apiEndpoint: 'http://localhost:9050',\n  authClient: emulatorGoogleAuth(),\n});\n```\n\n`emulatorGoogleAuth()` lives at the `bigquery-local/auth` subpath so\nthe core entry has zero auth dependencies. The helper itself imports\n`google-auth-library`, which is declared as an **optional peer\ndependency** — if you're using `@google-cloud/bigquery` you already\nhave it transitively; otherwise:\n\n```bash\nnpm install --save-dev google-auth-library\n```\n\nThe helper returns an `OAuth2Client` that attaches a placeholder\n`Authorization: Bearer emulator` header without ever calling Google.\nThe emulator ignores the token; the header is only there so the\nofficial client doesn't error before sending the request. The\nemulator itself accepts any (or no) auth header.\n\nOne server serves any project id — projects are isolated by URL path,\nthe same way real BigQuery does it. The official client sends URLs\nprefixed with `/bigquery/v2/...`; the emulator strips that prefix\ninternally, so a single route table serves both raw HTTP callers and\nthe client library.\n\nIf you don't want a fake auth client at all, the BigQuery client also\nhas a built-in fallthrough: when ADC finds **no** credentials, the\nclient sends unauthenticated requests, which the emulator accepts.\nThat works on clean CI runners but is flaky on dev machines that have\nstale `gcloud auth login` state — the helper above makes it\ndeterministic.\n\nIf you can't use the `authClient` option (different client library,\nconstructed deep inside framework code, etc.), the\n`BIGQUERY_EMULATOR_HOST` env var also works:\n\n```bash\nBIGQUERY_EMULATOR_HOST=http://localhost:9050 \\\nGOOGLE_APPLICATION_CREDENTIALS=$(pwd)/fake-creds.json \\\nnode my-app.js\n```\n\n`fake-creds.json` can be any valid-shaped service-account JSON — the\nclient lib uses it to skip the ADC lookup, then the request still\nlands on the emulator.\n\n### Embedding it in your tests\n\n`bigquery-local` is also a Node library. Spin one up in-process —\nno Docker, no global port — and tear it down in `afterAll`:\n\n```bash\nnpm install --save-dev bigquery-local\n```\n\n```ts\nimport { createServer } from 'bigquery-local';\nimport { emulatorGoogleAuth } from 'bigquery-local/auth';\nimport { BigQuery } from '@google-cloud/bigquery';\n\nconst server = await createServer({ database: ':memory:' });\nawait server.listen(0); // 0 = pick a random free port\n\nconst bigQuery = new BigQuery({\n  projectId: 'test',\n  apiEndpoint: server.url,\n  authClient: emulatorGoogleAuth(),\n});\n\n// ...run your tests against `bigQuery`...\n\nawait server.close(); // closes the HTTP listener and the DB\n```\n\nIf you want to inspect or assert on the raw HTTP wire format directly,\n`server.url` is just a normal `http://127.0.0.1:\u003cport\u003e` URL — `fetch()`\nhits the same routes the client library uses.\n\n---\n\n## CLI\n\n```\nUsage: bigquery-local [options]\n\nOptions:\n  --project=\u003cid\u003e         Default project id (informational; routes accept any).\n  --port=\u003cn\u003e             REST API port (default: 9050; 0 = pick a free port).\n  --grpc-port=\u003cn\u003e        gRPC port (default: 9060). Returns UNIMPLEMENTED to all RPCs.\n  --database=\u003cpath\u003e      DuckDB file path (default: \":memory:\").\n  --log-level=\u003clevel\u003e    debug | info | warn | error (default: info).\n  --log-format=\u003cfmt\u003e     json | text (default: text).\n  --data-from-yaml=\u003cf\u003e   Seed data file (reserved; not yet implemented).\n  -v, --version          Print version and exit.\n  -h, --help             Print this help text and exit.\n```\n\n`--project` is informational — the server is multi-tenant by URL path,\nso any project id a client uses just works. There's no need to declare\nprojects up front.\n\n### gRPC\n\nThe container also binds the gRPC port (default `9060`). Every RPC\nreturns `UNIMPLEMENTED` (gRPC status 12), which is the canonical wire\nshape every conforming gRPC client expects for synchronous errors.\nThis means a client like `@google-cloud/bigquery-storage` pointed at\nthis port gets a clean error instead of a hung connection — useful\nwhen you have shared client code that constructs both REST and\nStorage Read API handles.\n\n---\n\n## Storage\n\nBacked by **DuckDB** via `@duckdb/node-api`. Datasets map to DuckDB\nschemas, tables map to DuckDB tables, and BQ types map directly onto\nDuckDB types (`ARRAY\u003cT\u003e` → `T[]`, `STRUCT\u003c…\u003e` → `STRUCT(…)`, `NUMERIC` →\n`DECIMAL(38,9)`, `JSON` → `JSON`, etc.). Metadata (datasets, tables,\njobs) lives in a dedicated `_bq` schema.\n\nEither point `--database` at a file path for persistence, or omit it to\nrun fully in-memory.\n\n---\n\n## Compatibility\n\nThe target is Google's published BigQuery REST API — anywhere your\nclient successfully hits real BigQuery, it should also work against\nthis emulator (within the features listed above).\n\nCommon CLI flags and the default port `9050` match the conventions used\nby other BigQuery emulators, so swapping an existing emulator container\nis typically a one-line image change plus dropping any `platform:\nlinux/amd64` pin (this image is multi-arch).\n\n---\n\n## Development\n\nSource is TypeScript end-to-end, run directly under Node 24's native\ntype stripping — no transpile step.\n\n```bash\nnpm install\nnpm run typecheck       # tsc --noEmit\nnpm run lint            # biome lint\nnpm run format:check    # biome format\nnpm test                # node --conditions=src --test\nnpm run test:coverage   # ≥ 90% lines / branches / functions\n\nnode bin/bigquery-local.ts --port=0\n```\n\nCI runs the full toolchain on **Ubuntu, macOS, and Windows × Node 24\nand Node 26** — six jobs per PR. `noExplicitAny` is enforced;\n`tsconfig.json` sets `erasableSyntaxOnly` so no syntax that would\nrequire runtime transformation can slip in.\n\nThe library entrypoint resolves from `src/index.ts` in dev (via the\n`src` export condition + `node --conditions=src`) and from\n`dist/index.js` after publish — same import specifier in both worlds,\nno rebuild step needed during local iteration.\n\n---\n\n## Releasing\n\nReleases are cut as **GitHub Releases** — publishing a release creates the\ngit tag and triggers the publish workflow.\n\n1. Land a PR that bumps `package.json` `version` to `X.Y.Z`.\n2. From `main`, create the release:\n\n   ```bash\n   gh release create vX.Y.Z --generate-notes --title \"vX.Y.Z\"\n   ```\n\n   Or use the UI at https://github.com/jtwebman/bigquery-local/releases/new\n   and tick \"Generate release notes\" — GitHub assembles the changelog from\n   the PRs merged since the previous release.\n\nPublishing the release triggers `.github/workflows/publish.yml`, which:\n\n- verifies the tag matches `package.json` (fails fast if not),\n- builds `linux/amd64` + `linux/arm64` and pushes\n  `jtwebman/bigquery-local:X.Y.Z` and `:latest` to Docker Hub,\n- runs `tsc -p tsconfig.build.json` and publishes the npm package as\n  `bigquery-local@X.Y.Z` with `--provenance` (signed attestation tied\n  to the GitHub release).\n\n**Setup (one-time):**\n\n- GitHub repository secrets: `DOCKERHUB_USERNAME`, `DOCKERHUB_TOKEN`.\n- npm Trusted Publisher: on npmjs.com → package settings →\n  **Trusted Publishers** → add a GitHub Actions publisher for repo\n  `jtwebman/bigquery-local` and workflow `publish.yml`. No `NPM_TOKEN`\n  secret needed; the workflow authenticates via GitHub's OIDC token,\n  which also enables `npm publish --provenance`.\n\n---\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjtwebman%2Fbigquery-local","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjtwebman%2Fbigquery-local","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjtwebman%2Fbigquery-local/lists"}