{"id":30296001,"url":"https://github.com/thesis/mezo-dbt","last_synced_at":"2026-02-26T03:45:41.293Z","repository":{"id":305901065,"uuid":"1024249251","full_name":"thesis/mezo-dbt","owner":"thesis","description":null,"archived":false,"fork":false,"pushed_at":"2025-08-08T12:42:44.000Z","size":357,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-08T14:37:59.772Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thesis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-22T12:06:54.000Z","updated_at":"2025-08-04T19:47:53.000Z","dependencies_parsed_at":"2025-08-08T14:24:52.690Z","dependency_job_id":"107da8bc-370f-4fed-bdad-1ea3cb0703f5","html_url":"https://github.com/thesis/mezo-dbt","commit_stats":null,"previous_names":["thesis/mezo-dbt"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/thesis/mezo-dbt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesis%2Fmezo-dbt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesis%2Fmezo-dbt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesis%2Fmezo-dbt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesis%2Fmezo-dbt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thesis","download_url":"https://codeload.github.com/thesis/mezo-dbt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesis%2Fmezo-dbt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270799867,"owners_count":24648130,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-17T02:00:09.016Z","response_time":129,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-17T02:39:28.491Z","updated_at":"2026-02-26T03:45:41.288Z","avatar_url":"https://github.com/thesis.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DBT DataWarhouse Transformations for Mezo\n\n## Setup the dbt project locally\n\n### Prerequisites\n\n- Install [gcloud](https://cloud.google.com/sdk/docs/install)\n- VSCode or any other code editor\n- Optional: [dbt power user for vscode](https://marketplace.visualstudio.com/items?itemName=innoverio.vscode-dbt-power-user)\n\n### Clone the Repository\n\n```sh\n   git clone https://github.com/thesis/mezo-dbt\n   cd mezo-dbt\n```\n\n### Install Dependencies\n\n- Install [uv](https://docs.astral.sh/uv/getting-started/installation/#installing-uv)\n- Install Python dependencies (with uv):\n\n```sh\n   uv sync\n   source .venv/bin/activate #Activate the venv\n```\n\n### Configure dbt [profile.yml](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) locally\n\n- Create a .dbt folder in your home directory if it doesn’t exist:\n\n```sh\n   mkdir ~/.dbt\n   touch ~/.dbt/profiles.yml\n   code ~/.dbt/profiles.yml ## https://code.visualstudio.com/docs/configure/command-line#_launching-from-command-line\n   ##Or open with vim if you know how to close it.\n   vim ~/.dbt/profiles.yml ## or\n```\n\n- Edit profiles.yml file inside it with your BigQuery configuration:\n\n```yml\n   mezo:\n      outputs:\n         dev:\n            type: bigquery\n            method: oauth\n            project: \u003cyour-gcp-project-id\u003e\n            dataset: dbt_yourname\n            location: EU\n            threads: 4\n   target: dev\n```\n\n- Authenticate with gcloud (creates local credentials JSON automatically):\n\n```sh\n   gcloud auth login --enable-gdrive-access --update-adc\n```\n\n### Test your setup\n\n```sh\n   dbt debug\n```\n\n### Install DBT Dependencies\n\n```sh\n   dbt deps\n```\n\nFor other dbt commands check:\n[https://docs.getdbt.com/reference/dbt-commands](https://docs.getdbt.com/reference/dbt-commands)\n\n### This projects uses [pre-commit](https://pre-commit.com/)\n\nTo run checks locally use:\n\n```sh\n   pre-commit run --all-files --config .pre-commit-config_local.yaml\n```\n\nUse the following hook to run checks before commit:\n\n```bash\nINSTALL_PYTHON=/Users/benedikt/Documents/gitrepos/crfe-orc-cloud-composer/.venv/bin/python3\nARGS=(hook-impl --config=.pre-commit-config_local.yaml --hook-type=pre-commit)\n\n# end templated\n\nHERE=\"$(cd \"$(dirname \"$0\")\" \u0026\u0026 pwd)\"\nARGS+=(--hook-dir \"$HERE\" -- \"$@\")\n\nif [ -x \"$INSTALL_PYTHON\" ]; then\n    exec \"$INSTALL_PYTHON\" -mpre_commit \"${ARGS[@]}\"\nelif command -v pre-commit \u003e /dev/null; then\n    exec pre-commit \"${ARGS[@]}\"\nelse\n    echo '`pre-commit` not found.  Did you forget to activate your virtualenv?' 1\u003e\u00262\n    exit 1\nfi\n```\n\n## How to Set Up a Goldsky Table\n\nTo set up a new table using Goldsky data in BigQuery:\n\nContact Goldsky Support: Email [Goldsky](support@goldsky.com) to request the setup of a new table to be imported into the `mezo-prod-dp-dwh-lnd-goldsky-cs-0` Google Cloud Storage (GCS) bucket. As of this writing, the [Goldsky documentation](https://docs.goldsky.com/mirror/extensions/channels/aws-s3) is limited, and self-service setup is not available—you must contact support to establish the connection.\n\n### Organize Data in GCS\n\n- For each import, create a separate folder in the GCS bucket.\n  - The folder structure should follow this pattern: `event_type=\u003cevent_type\u003e/event_date=\u003cYYYY-MM-DD\u003e/` (e.g., `event_type=donated/event_date=2025-05-22/`).\n  - This structure enables Hive partitioning of the table. For more details, see the [BigLake partitioned data documentation](https://cloud.google.com/bigquery/docs/create-cloud-storage-table-biglake#create-biglake-partitioned-data).\n\n### Update dbt Source Configuration\n\n- Edit the [models/00_sources/goldsky.yml](models/00_sources/goldsky.yml) file to add the new table definition.\n- Use the existing configurations in the file as a template for your new entry as a reference.\n- Ensure all relevant metadata, columns, and partitioning information are included.\n\n### Register the Table in BigQuery\n\n- The table will be created in BigQuery using the [dbt-external-tables](https://github.com/dbt-labs/dbt-external-tables) package.\n- After updating the YAML file, run the following dbt command to create the external tables:\n\n     ```sh\n     dbt run-operation stage_external_sources\n     ```\n\n- This command will register the external tables in BigQuery based on your configuration. This is automatically run during deployment and CI Process.\n\n## Update the External Table in BigQuery\n\nIf the source file (e.g., Google Sheet) changes structure:\n- Edit the corresponding YAML file in models/00_sources/\n- Adjust schema, columns, or partitioning as needed.\n- Re-stage the external table\n\n     ```sh\n     dbt run-operation stage_external_sources\n     ```\n\n- These steps are also run automatically via GitHub Actions, but for local testing, you must run them manually.\n\n## 📖 Documentation\n\nThis project includes autogenerated dbt documentation, hosted with GitHub Pages.\n👉 [View the dbt docs](https://thesis.github.io/mezo-dbt/#!/overview)\nThe documentation site is automatically updated via GitHub Actions when changes are merged into the repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthesis%2Fmezo-dbt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthesis%2Fmezo-dbt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthesis%2Fmezo-dbt/lists"}