{"id":18686733,"url":"https://github.com/wayfair-incubator/bigquery-buildkite-plugin","last_synced_at":"2025-08-21T02:09:03.292Z","repository":{"id":36959635,"uuid":"329311230","full_name":"wayfair-incubator/bigquery-buildkite-plugin","owner":"wayfair-incubator","description":"Buildkite Plugin to create/update structures on BigQuery","archived":false,"fork":false,"pushed_at":"2025-08-20T23:08:04.000Z","size":262,"stargazers_count":8,"open_issues_count":18,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-08-21T01:09:56.262Z","etag":null,"topics":["bigquery","buildkite","buildkite-plugin","gbq","google","google-bigquery","google-cloud-platform","hacktoberfest"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wayfair-incubator.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-01-13T13:07:02.000Z","updated_at":"2024-08-22T12:17:09.000Z","dependencies_parsed_at":"2023-02-12T21:32:02.953Z","dependency_job_id":"374a7cf0-d6c1-4702-8ece-68a36545338f","html_url":"https://github.com/wayfair-incubator/bigquery-buildkite-plugin","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/wayfair-incubator/bigquery-buildkite-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wayfair-incubator%2Fbigquery-buildkite-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wayfair-incubator%2Fbigquery-buildkite-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wayfair-incubator%2Fbigquery-buildkite-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wayfair-incubator%2Fbigquery-buildkite-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wayfair-incubator","download_url":"https://codeload.github.com/wayfair-incubator/bigquery-buildkite-plugin/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wayfair-incubator%2Fbigquery-buildkite-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271415496,"owners_count":24755639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","buildkite","buildkite-plugin","gbq","google","google-bigquery","google-cloud-platform","hacktoberfest"],"created_at":"2024-11-07T10:28:47.808Z","updated_at":"2025-08-21T02:09:03.274Z","avatar_url":"https://github.com/wayfair-incubator.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Actions Status](https://github.com/wayfair-incubator/bigquery-buildkite-plugin/workflows/Lint/badge.svg?branch=main)](https://github.com/wayfair-incubator/bigquery-buildkite-plugin/actions)\n[![Actions Status](https://github.com/wayfair-incubator/bigquery-buildkite-plugin/workflows/Unit%20Tests/badge.svg?branch=main)](https://github.com/wayfair-incubator/bigquery-buildkite-plugin/actions)\n![Version](https://img.shields.io/static/v1.svg?label=Version\u0026message=1.2.2\u0026color=lightgrey\u0026?link=http://left\u0026link=https://github.com/wayfair-incubator/bigquery-buildkite-plugin/tree/v1.2.2)\n![Plugin Status](https://img.shields.io/static/v1.svg?label=\u0026message=Buildkite%20Plugin\u0026color=blue\u0026logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAGAAAABgCAMAAADVRocKAAAAk1BMVEX///+DrRik1Cb+/vyErhin1yeAqhfJ5n+dzCO50X2VwiC2zna020vg8LSo1i+q1zTN3qL7/fav2UDT4q33++yXuj3z+eLO6IqMuByJsSPM54XX5bWcvUbH5Xrg6sWiwVHB1ovk7cymxFnq8deOtC2xzG7c6L6/1YfU7JbI2pj2+e+jzzPd763K3J3j8ryRtDbY7aOqCe3pAAACTElEQVRoge3Z11KDQBSAYUpItSRqVGwxEVOwxPd/OndhCbA5BTaLM87sueb83+yMStHz3Lhx48bN/5iw+2VjImy0fDvzQiNCLKVrdjn0nq++QwNCLMy+9uOzc2Y59AZBwF4F55NefxxxyxK4aEvI/C7x/QxglrMTBNxVej6V+QNALh+AhkRY5isAsVwBxFWfM/rnLsu/qnwNQIkawBBy/abMawBCaEAQXGFElt/Evo8CIHEEIESWH9XyAAAQACCIH40A8yBwRICARiB5BNAIBKgQ8tLbCZBHgRqBAgWB5wmgQhCAIt7ekTwJ5AQHSGKC1TkgiM7WIs8A0bBvCETR8H7CA4EhIPP9fgPA7AR53u91BBR53+8EKPPdACLvq3wXQC1vH9DytoGjvF0gGo71vE0gCoC8PQDJ2wJEvgfmbQFo3g5A5HNA3K+eL0yBePdO5AtAEAOUoIB4c+ONiHwBZHddjMCB5DUV6yOqfzgBQWCAzMu9ZoAiHgACBpJdmj3SNAdQAgKSXfFQ1gZQz28PlxxQ5tsCIKED+6/qU2tbQBF3lxgwn9YfitsDR0QVmE9D7bHeBNCIEphf63lToEYUAJQ3BxSxlUQGPD3Cr5DmwIGQJ8DypwHqlXX7gec5oMcAXv5OT31OYU6weuM/9tDv/iSwWnJxfghA5s0+QzUCFi828iiwWNvJI4C9PAg8WcwDAPVLYwGwndeAufV8DZB/bm3nK0A3+RyI81tdF3l1gn1neQlM4qnpp+9ms0xP/P8AP/8778aNGzdu/mh+AQp1NCB/JInXAAAAAElFTkSuQmCC)\n\n\n# BigQuery Buildkite Plugin\n\nThis Buildkite plugin can be used to deploy tables/views schemas to BigQuery\n\nA [Buildkite plugin](https://buildkite.com/docs/agent/v3/plugins) for deploying tables/views schemas to BigQuery.\n\n## Using the plugin\n\nIf the version number is not provided then the most recent version of the plugin will be used. Do not use version number as `master` or any branch names.\n\n### Simple\n\n```yaml\nsteps:\n  - plugins:\n      - wayfair-incubator/bigquery#v1.2.2:\n          gcp_project: gcp-us-project\n          dataset_schema_directory: schemas/gcp-us-project/dataset\n```\n\n## Configuration\n\n### Required\n\n### `gcp_project` (required, string)\n\nThe full name of the GCP project you want to deploy.\n\nExample: `gcp-us-project`\n\n### `dataset_schema_directory` (required, string)\n\nThe directory in your repository where are you storing the schemas for your tables and views.\n\nExample: `gcp-us-project/dataset_name`\n\n### `prod_build_branch` (optional, string)\n\nDefault branch for the repo.\n\nExample: `main`\n\nDefault: `master`\n\n### `execute_only_changed_files` (optional, boolean)\n\nWhether to execute just the files that changed in a directory or not.\n\nExample: `true`\n\nDefault: `true`\n\n### `fail_pipeline_on_first_exception` (optional, boolean)\n\nWhether to fail pipeline as soon as there is one failure.\n\nExample: `true`\n\nDefault: `true`\n\n## Secret\n\nThis plugin expects `GCP_SERVICE_ACCOUNT` is placed as environment variable. Make sure to store it [securely](https://buildkite.com/docs/pipelines/secrets)!\n\n```yaml\nenv:\n  gcp_service_account: '{\"email\": \"\"}'\n```\n\n## Example\n\n### Basic\n\nThe following pipeline will deploy all the schemas to the `gcp-us-project` living under `directory/project/`\n\n## Schemas\n\nThis plugin uses [GBQ](https://github.com/wayfair-incubator/gbq) to deploy to Google BigQuery.\n[GBQ](https://github.com/wayfair-incubator/gbq) now supports specifying partitions with the schema as well.\n\nTo leverage this you need to nest your JSON table schema in a dictionary. An example for the same is given below. Library supports Time and Range based partitioning along with Clustering.\n\nAll the configuration options can be found [here](https://github.com/wayfair-incubator/gbq/blob/main/gbq/dto.py).\n\n```json\n{\n  \"partition\": {\n    \"type\": \"range\",\n    \"definition\": {\n      \"field\": \"ID\",\n      \"range\": {\n        \"start\": 1,\n        \"end\": 100000,\n        \"interval\": 10\n      }\n    }\n  },\n  \"clustering\": [\n    \"ID\"\n  ],\n  \"schema\": [\n    {\n      \"name\": \"ID\",\n      \"type\": \"INTEGER\",\n      \"mode\": \"REQUIRED\"\n    }\n  ]\n}\n```\n\n## Contributing\n\nSee the [Contributing Guide](CONTRIBUTING.md) for additional information.\n\nTo execute tests locally (requires that `docker` and `docker-compose` are installed):\n\n```bash\ndocker-compose run test\n```\n\n## Credits\n\nThis plugin was originally written by [Jash Parekh](https://github.com/jashparekh) for Wayfair.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwayfair-incubator%2Fbigquery-buildkite-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwayfair-incubator%2Fbigquery-buildkite-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwayfair-incubator%2Fbigquery-buildkite-plugin/lists"}