{"id":18508951,"url":"https://github.com/lightspeed/bigquery-uniview","last_synced_at":"2025-05-14T10:15:05.326Z","repository":{"id":98938325,"uuid":"283556864","full_name":"lightspeed/bigquery-uniview","owner":"lightspeed","description":"Create BigQuery views that unify sets of table with the same prefix and different versions.","archived":false,"fork":false,"pushed_at":"2020-07-29T17:17:38.000Z","size":7,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-17T02:25:01.630Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lightspeed.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-29T17:06:35.000Z","updated_at":"2022-07-28T14:26:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"09d7fc1c-57b9-4e27-8469-edff307e7b4c","html_url":"https://github.com/lightspeed/bigquery-uniview","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightspeed%2Fbigquery-uniview","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightspeed%2Fbigquery-uniview/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightspeed%2Fbigquery-uniview/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightspeed%2Fbigquery-uniview/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lightspeed","download_url":"https://codeload.github.com/lightspeed/bigquery-uniview/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254120180,"owners_count":22017953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T15:15:57.143Z","updated_at":"2025-05-14T10:15:05.281Z","avatar_url":"https://github.com/lightspeed.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BigQuery UniView\n\nCreate BigQuery views that unify sets of table with the same prefix and\ndifferent versions. The view returns a union of all records from the\ndifferent table versions and cast columns to a common data type.\n\nThis tool was written to be deployed as a Cloud Function. The function is\ntriggered by a Pub/Sub topic feeded by our BigQuery sink. The dispatched\nmessages contain information about table that are created by our process.\n\ne.g:\n```\n{\"dataset\":\"...\",\"table\":{\"name\":\"customer\",\"version\":\"5b6ffe5fb\"}}\n```\n\n## Rules\n\nColumns that conflict in their data type are cast to a common representation.\nThe specific cast used depends on the conflicting data types. Basic idea is\nto extend the target data type. We go from a more specific to more generic\ntype. This mitigates any information loss. As such, STRING represents our most\ngeneric type, since it can represent any other BQ type.\n\n| Conflict Types     | Cast To   |\n|--------------------|-----------|\n| DATETIME TIMESTAMP | TIMESTAMP |\n| ANY ANY            | STRING    |\n\n## Variables\n\nThe `PROJECT_ID` environment variable must be set to a valid Google Project ID.\nThe service account must have READ access to the source dataset and WRITE access\nto the output dataset.\n\n## Deployment\n\nThe following example is the command that we use to deploy on GCP:\n\n```\ngcloud functions deploy --project \u003cPROJECT_ID\u003e --runtime python37 --retry \\\n    --entry-point handler --source ./ --trigger-topic bigquery-table-create \\\n    bigquery-uniview --set-env-vars PROJECT_ID=\u003cPROJECT_ID\u003e \\\n    --service-account bigquery-uniview@\u003cPROJECT_ID\u003e.iam.gserviceaccount.com\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightspeed%2Fbigquery-uniview","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flightspeed%2Fbigquery-uniview","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightspeed%2Fbigquery-uniview/lists"}