{"id":13480762,"url":"https://github.com/goccy/bigquery-emulator","last_synced_at":"2025-05-14T11:10:55.654Z","repository":{"id":39166109,"uuid":"505411372","full_name":"goccy/bigquery-emulator","owner":"goccy","description":"BigQuery emulator server implemented in Go","archived":false,"fork":false,"pushed_at":"2024-12-10T15:04:26.000Z","size":423,"stargazers_count":934,"open_issues_count":167,"forks_count":133,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-05-14T04:43:28.300Z","etag":null,"topics":["bigquery","emulator","gcp","go","golang","google-cloud","google-cloud-platform"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/goccy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["goccy"]}},"created_at":"2022-06-20T11:24:23.000Z","updated_at":"2025-05-08T09:42:37.000Z","dependencies_parsed_at":"2024-04-09T02:38:43.651Z","dependency_job_id":"b1e53d0c-1243-45ce-8e6c-4d17fd5e6e66","html_url":"https://github.com/goccy/bigquery-emulator","commit_stats":{"total_commits":157,"total_committers":14,"mean_commits":"11.214285714285714","dds":"0.15923566878980888","last_synced_commit":"316038b8dd3f6534a87948f28ac54c41dc6afc21"},"previous_names":[],"tags_count":61,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goccy%2Fbigquery-emulator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goccy%2Fbigquery-emulator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goccy%2Fbigquery-emulator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goccy%2Fbigquery-emulator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/goccy","download_url":"https://codeload.github.com/goccy/bigquery-emulator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254129491,"owners_count":22019628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","emulator","gcp","go","golang","google-cloud","google-cloud-platform"],"created_at":"2024-07-31T17:00:44.714Z","updated_at":"2025-05-14T11:10:55.564Z","avatar_url":"https://github.com/goccy.png","language":"Go","readme":"# BigQuery Emulator\n\n[![build and test](https://github.com/goccy/bigquery-emulator/actions/workflows/test.yml/badge.svg)](https://github.com/goccy/bigquery-emulator/actions/workflows/test.yml)\n[![GoDoc](https://godoc.org/github.com/goccy/bigquery-emulator?status.svg)](https://pkg.go.dev/github.com/goccy/bigquery-emulator?tab=doc)\n\n\nBigQuery emulator server implemented in Go.  \nBigQuery emulator provides a way to launch a BigQuery server on your local machine for testing and development.\n\n# Features\n\n- If you can choose the Go language as BigQuery client, you can launch a BigQuery emulator on the same process as the testing process by [httptest](https://pkg.go.dev/net/http/httptest) .\n- BigQuery emulator can be built as a static single binary and can be launched as a standalone process. So, you can use the BigQuery emulator from programs written in non-Go languages or such as the [bq](https://cloud.google.com/bigquery/docs/bq-command-line-tool) command, by specifying the address of the launched BigQuery emulator.\n- BigQuery emulator utilizes SQLite for storage. You can select either memory or file as the data storage destination at startup, and if you set it to file, data can be persisted.\n- You can load seeds from a YAML file on startup\n\n# Status\n\nAlthough this project is still in **beta** version, many features are already available.\n\n## BigQuery API\n\nWe've been implemented all the [BigQuery APIs](https://cloud.google.com/bigquery/docs/reference/rest) except the API to manipulate IAM resources. It is possible that some options are not supported, in which case please report them in an Issue.\n\n## Google Cloud Storage linkage\n\nBigQuery emulator supports loading data from Google Cloud Storage and extracting table data. Currently, only CSV and JSON data types can be used for extracting. If you use Google Cloud Storage emulator, please set `STORAGE_EMULATOR_HOST` environment variable.\n\n## BigQuery Storage API\n\nSupports gRPC-based read/write using [BigQuery Storage API](https://cloud.google.com/bigquery/docs/reference/storage).\nSupports both Apache `Avro` and `Arrow` formats.\n\n## Google Standard SQL\n\nBigQuery emulator supports many of the specifications present in [Google Standard SQL](https://cloud.google.com/bigquery/docs/reference/standard-sql/introduction).\nFor example, it has the following features.\n\n- 200+ standard functions\n- Wildcard table\n- Templated Argument Function\n- JavaScript UDF\n\nIf you want to know the specific features supported, please see [here](https://github.com/goccy/go-zetasqlite#status)\n\n# Goals and Sponsors\n\nThe goal of this project is to build a server that behaves exactly like BigQuery from the BigQuery client's perspective. To do so, we need to support all features present in BigQuery ( Model API / Connection API / INFORMATION SCHEMA etc.. ) in addition to evaluating Google Standard SQL.\n\nHowever, this project is a personal project and I develop it on my days off and after work. I work full time and maintain a lot of OSS. Therefore, the time available for this project is also limited. Of course, I will be adding features and fixing bugs on a regular basis to get us closer to our goals, but if you want me to implement the features you want, please consider sponsoring me. Of course, you can use this project for free, but if you sponsor me, that will be my motivation. Especially if you are part of a commercial company and could use this project, I'd be glad if you could consider sponsoring me at the same time.\n\n# Install\n\nIf Go is installed, you can install the latest version with the following command\n\n```console\n$ go install github.com/goccy/bigquery-emulator/cmd/bigquery-emulator@latest\n```\n\nThe BigQuery emulator depends on [go-zetasql](https://github.com/goccy/go-zetasql).\nThis library takes a very long time to install because it automatically builds the ZetaSQL library during install.\nIt may look like it hangs because it does not log anything during the build process, but if the `clang` process is running in the background, it is working fine, so just wait it out.\nAlso, for this reason, the following environment variables must be enabled for installation.\n\n```console\nCGO_ENABLED=1\nCXX=clang++\n```\n\nYou can also download the docker image with the following command\n\n```console\n$ docker pull ghcr.io/goccy/bigquery-emulator:latest\n```\n\nYou can also download the darwin(amd64) and linux(amd64) binaries directly from [releases](https://github.com/goccy/bigquery-emulator/releases)\n\n# How to start the standalone server\n\nIf you can install the `bigquery-emulator` CLI, you can start the server using the following options.\n\n```console\n$ ./bigquery-emulator -h\nUsage:\n  bigquery-emulator [OPTIONS]\n\nApplication Options:\n      --project=        specify the project name\n      --dataset=        specify the dataset name\n      --port=           specify the http port number. this port used by bigquery api (default: 9050)\n      --grpc-port=      specify the grpc port number. this port used by bigquery storage api (default: 9060)\n      --log-level=      specify the log level (debug/info/warn/error) (default: error)\n      --log-format=     specify the log format (console/json) (default: console)\n      --database=       specify the database file if required. if not specified, it will be on memory\n      --data-from-yaml= specify the path to the YAML file that contains the initial data\n  -v, --version         print version\n\nHelp Options:\n  -h, --help            Show this help message\n```\n\nStart the server by specifying the project name\n\n```console\n$ ./bigquery-emulator --project=test\n[bigquery-emulator] REST server listening at 0.0.0.0:9050\n[bigquery-emulator] gRPC server listening at 0.0.0.0:9060\n```\n\nIf you want to use docker image to start emulator, specify like the following.\n\n```console\n$ docker run -it ghcr.io/goccy/bigquery-emulator:latest --project=test\n```\n\n* If you are using an M1 Mac ( and Docker Desktop ) you may get a warning. In that case please use `--platform linux/x86_64` option.\n\n## How to use from bq client\n\n### 1. Start the standalone server\n\n```console\n$ ./bigquery-emulator --project=test --data-from-yaml=./server/testdata/data.yaml\n[bigquery-emulator] REST server listening at 0.0.0.0:9050\n[bigquery-emulator] gRPC server listening at 0.0.0.0:9060\n```\n\n* `server/testdata/data.yaml` is [here](https://github.com/goccy/bigquery-emulator/blob/main/server/testdata/data.yaml)\n\n### 2. Call endpoint from bq client\n\n```console\n$ bq --api http://0.0.0.0:9050 query --project_id=test \"SELECT * FROM dataset1.table_a WHERE id = 1\"\n\n+----+-------+---------------------------------------------+------------+----------+---------------------+\n| id | name  |                  structarr                  |  birthday  | skillNum |     created_at      |\n+----+-------+---------------------------------------------+------------+----------+---------------------+\n|  1 | alice | [{\"key\":\"profile\",\"value\":\"{\\\"age\\\": 10}\"}] | 2012-01-01 |        3 | 2022-01-01 12:00:00 |\n+----+-------+---------------------------------------------+------------+----------+---------------------+\n```\n\n## How to use from python client\n\n### 1. Start the standalone server\n\n```console\n$ ./bigquery-emulator --project=test --dataset=dataset1\n[bigquery-emulator] REST server listening at 0.0.0.0:9050\n[bigquery-emulator] gRPC server listening at 0.0.0.0:9060\n```\n\n### 2. Call endpoint from python client\n\nCreate ClientOptions with api_endpoint option and use AnonymousCredentials to disable authentication.\n\n```python\nfrom google.api_core.client_options import ClientOptions\nfrom google.auth.credentials import AnonymousCredentials\nfrom google.cloud import bigquery\nfrom google.cloud.bigquery import QueryJobConfig\n\nclient_options = ClientOptions(api_endpoint=\"http://0.0.0.0:9050\")\nclient = bigquery.Client(\n  \"test\",\n  client_options=client_options,\n  credentials=AnonymousCredentials(),\n)\nclient.query(query=\"...\", job_config=QueryJobConfig())\n```\n\nIf you use a DataFrame as the download destination for the query results,\nYou must either disable the BigQueryStorage client with `create_bqstorage_client=False` or\ncreate a BigQueryStorage client that references the local grpc port (default 9060).\n\nhttps://cloud.google.com/bigquery/docs/samples/bigquery-query-results-dataframe?hl=en\n\n```python\nresult = client.query(sql).to_dataframe(create_bqstorage_client=False)\n```\n\nor\n\n```python\nfrom google.cloud import bigquery_storage\n\nclient_options = ClientOptions(api_endpoint=\"0.0.0.0:9060\")\nread_client = bigquery_storage.BigQueryReadClient(client_options=client_options)\nresult = client.query(sql).to_dataframe(bqstorage_client=read_client)\n``` \n\n# Synopsis\n\nIf you use the Go language as a BigQuery client, you can launch the BigQuery emulator on the same process as the testing process.  \nPlease imports `github.com/goccy/bigquery-emulator/server` ( and `github.com/goccy/bigquery-emulator/types` ) and you can use `server.New` API to create the emulator server instance.\n\nSee the API reference for more information: https://pkg.go.dev/github.com/goccy/bigquery-emulator\n\n```go\npackage main\n\nimport (\n  \"context\"\n  \"fmt\"\n\n  \"cloud.google.com/go/bigquery\"\n  \"github.com/goccy/bigquery-emulator/server\"\n  \"github.com/goccy/bigquery-emulator/types\"\n  \"google.golang.org/api/iterator\"\n  \"google.golang.org/api/option\"\n)\n\nfunc main() {\n  ctx := context.Background()\n  const (\n    projectID = \"test\"\n    datasetID = \"dataset1\"\n    routineID = \"routine1\"\n  )\n  bqServer, err := server.New(server.TempStorage)\n  if err != nil {\n    panic(err)\n  }\n  if err := bqServer.Load(\n    server.StructSource(\n      types.NewProject(\n        projectID,\n        types.NewDataset(\n          datasetID,\n        ),\n      ),\n    ),\n  ); err != nil {\n    panic(err)\n  }\n  if err := bqServer.SetProject(projectID); err != nil {\n    panic(err)\n  }\n  testServer := bqServer.TestServer()\n  defer testServer.Close()\n\n  client, err := bigquery.NewClient(\n    ctx,\n    projectID,\n    option.WithEndpoint(testServer.URL),\n    option.WithoutAuthentication(),\n  )\n  if err != nil {\n    panic(err)\n  }\n  defer client.Close()\n  routineName, err := client.Dataset(datasetID).Routine(routineID).Identifier(bigquery.StandardSQLID)\n  if err != nil {\n    panic(err)\n  }\n  sql := fmt.Sprintf(`\nCREATE FUNCTION %s(\n  arr ARRAY\u003cSTRUCT\u003cname STRING, val INT64\u003e\u003e\n) AS (\n  (SELECT SUM(IF(elem.name = \"foo\",elem.val,null)) FROM UNNEST(arr) AS elem)\n)`, routineName)\n  job, err := client.Query(sql).Run(ctx)\n  if err != nil {\n    panic(err)\n  }\n  status, err := job.Wait(ctx)\n  if err != nil {\n    panic(err)\n  }\n  if err := status.Err(); err != nil {\n    panic(err)\n  }\n\n  it, err := client.Query(fmt.Sprintf(`\nSELECT %s([\n  STRUCT\u003cname STRING, val INT64\u003e(\"foo\", 10),\n  STRUCT\u003cname STRING, val INT64\u003e(\"bar\", 40),\n  STRUCT\u003cname STRING, val INT64\u003e(\"foo\", 20)\n])`, routineName)).Read(ctx)\n  if err != nil {\n    panic(err)\n  }\n\n  var row []bigquery.Value\n  if err := it.Next(\u0026row); err != nil {\n    if err == iterator.Done {\n        return\n    }\n    panic(err)\n  }\n  fmt.Println(row[0]) // 30\n}\n```\n\n# Debugging\n\nIf you have specified a database file when starting `bigquery-emulator`, you can check the status of the database by using the `zetasqlite-cli` tool. See [here](https://github.com/goccy/go-zetasqlite/tree/main/cmd/zetasqlite-cli#readme) for details.\n\n# How it works\n\n## BigQuery Emulator Architecture Overview\n\nAfter receiving ZetaSQL Query via REST API from bq or Client SDK for each language, go-zetasqlite parses and analyzes the ZetaSQL Query to output AST. After generating a SQLite query from the AST, go-sqite3 is used to access the SQLite Database.\n\n\u003cimg width=\"600px\" src=\"https://user-images.githubusercontent.com/209884/196145011-e35c2df4-5f5d-43ce-b7df-08cd130b5d31.png\"\u003e\u003c/img\u003e\n\n\n\n## Type Conversion Flow\n\nBigQuery has a number of types that do not exist in SQLite (e.g. ARRAY and STRUCT).\nIn order to handle them in SQLite, go-zetasqlite encodes all types except `INT64` / `FLOAT64` / `BOOL` with the type information and data combination and stores them in SQLite.\nWhen using the encoded data, decode the data via a custom function registered with go-sqlite3 before use.\n\n\u003cimg width=\"600px\" src=\"https://user-images.githubusercontent.com/209884/196145033-aa032878-7e01-4ec7-9a23-b174b87e1a24.png\"\u003e\u003c/img\u003e\n\n\n# Reference\n\nRegarding the story of bigquery-emulator, there are the following articles.\n- [How to create a BigQuery Emulator](https://docs.google.com/presentation/d/1j5TPCpXiE9CvBjq78W8BWz-cGxU8djW1qy9Y6eBHso8/edit?usp=sharing) ( Japanese )\n\n\n# License\n\nMIT\n","funding_links":["https://github.com/sponsors/goccy"],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoccy%2Fbigquery-emulator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoccy%2Fbigquery-emulator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoccy%2Fbigquery-emulator/lists"}