{"id":15060752,"url":"https://github.com/modataconsulting/dbt_ga4_project","last_synced_at":"2025-04-10T06:12:04.042Z","repository":{"id":57859175,"uuid":"522072574","full_name":"modataconsulting/dbt_ga4_project","owner":"modataconsulting","description":"This project uses Google Analytics 4 BigQuery Exports as its source data, and offers useful base transformations to provide report-ready dimension \u0026 fact models that can be used for reporting purposes, blending with other data, and/or feature engineering for ML models.","archived":false,"fork":false,"pushed_at":"2022-09-30T20:25:19.000Z","size":144,"stargazers_count":18,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-24T07:22:00.026Z","etag":null,"topics":["bigquery","bq","data-build-tool","dbt","ga4","google-analytics-4","sql"],"latest_commit_sha":null,"homepage":"https://modataconsulting.github.io/docsite/docs/category/dbt-ga4","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/modataconsulting.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-06T23:10:44.000Z","updated_at":"2025-02-11T17:57:02.000Z","dependencies_parsed_at":"2023-01-18T19:54:25.405Z","dependency_job_id":null,"html_url":"https://github.com/modataconsulting/dbt_ga4_project","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modataconsulting%2Fdbt_ga4_project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modataconsulting%2Fdbt_ga4_project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modataconsulting%2Fdbt_ga4_project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modataconsulting%2Fdbt_ga4_project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/modataconsulting","download_url":"https://codeload.github.com/modataconsulting/dbt_ga4_project/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248166925,"owners_count":21058481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","bq","data-build-tool","dbt","ga4","google-analytics-4","sql"],"created_at":"2024-09-24T23:04:01.136Z","updated_at":"2025-04-10T06:12:04.010Z","avatar_url":"https://github.com/modataconsulting.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"***NOTE: This project is still very much a work in progress, with much of the larger model restructuring still to come, see the [TODO](/TODO.md) file for more info.***\n\n# dbt GA4 Project\nFirst and foremost, this project is based off of the dbt [GA4 Package by Velir](https://hub.getdbt.com/velir/ga4/latest), but has been modified and refactored for internal purposes.\nThis project uses [Google Analytics 4 BigQuery Exports](https://support.google.com/analytics/topic/9359001) as its source data, and offers useful base transformations to provide report-ready dimension \u0026 fact models that can be used for reporting purposes, blending with other data, and/or feature engineering for ML models.\n\nFind more info about Google Analytics 4 BigQuery Exports [here](https://developers.google.com/analytics/bigquery).\n\n**Features Overview:**\n\n- Four final tables—`ga4__events`, `ga4__pages`, `ga4__sesssions`, and `ga4__users`—that are completed unnested to be wide \u0026 denomalized for easy querying by the end-user.\n- Conversion of the the day-shared `events_YYYYMMDD` \u0026 `events_intraday_YYYYMMDD` tables into singular date-partitionioned incremental base models.\n- Dynamically flattens `event_params` into their own individual columns.\n- Dynamically flattens `user_props` into their own individual columns.\n- Dynamically extracts \u0026 flattens URL `query_params` (e.g., `gclid`, `fbclid`, `_ga`) into their own individual columns.\n- Custom `Variables`. See [here](#optional-variables) for more info.\n- Custom `Marcros`. See [here](#macros) for more info.\n\n## Style Guide:\nThis project and any future projects that may be based off of this intial `dbt_ga4_project`, will be following [This Project's Style Guide...IN PROGRESS](/STYLEGUIDE.md), which borrows ideals from the following Style Guides:\n\n- [dbt's Style Guide](https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md)\n- [GitLab's SQL Style Guide](https://about.gitlab.com/handbook/business-technology/data-team/platform/sql-style-guide)\n\n# Models\n\n***DAG Overview***\n\n***NOTE: This DAG Image is NOT current \u0026 will continue to CHANGE until all models are finalized.***\n\n![DAG Overview](assets/DAG.png)\n\n## Mart Models\n\n| Model Name | Description |\n|------------|-------------|\n| ga4__events | This is the table for event-level metrics \u0026 dimensions, that has been transformed to be wide \u0026 denomalized for easier quering. |\n| ga4__pages | This is the table for page-level metrics \u0026 dimensions, such as `page_views`, `exits`, and `users`. This table is grouped by `page_title`, `event_date`, and `page_path`. |\n| ga4__sessions | This is the table for session-level metrics \u0026 dimensions, such as `is_engaged_session`, `engagement_duration`, and `page_views`. This table is grouped by both `session_key` and `user_key`. |\n| ga4__users | This is the table for user-level metrics \u0026 dimensions, such as `first` \u0026 `last_seen_date`, `geo`, and `traffic_source`. This table is grouped by the hashed `user_key` dimension, which is based on `user_id`, or `user_pseudo_id` if one doesn't exist. |\n\n## Staging \u0026 Intermediate Models\n\n| Model Name | Description |\n|------------|-------------|\n| stg_ga4__events | Creates a table with event data that is enhanced with useful `event_keys`, `page_keys`, `session keys`, and `user_keys`. |\n| stg_ga4__event_params | Creates a table that unnests all of the event parameters specific to each event (e.g. `page_view`, `click`, or `scroll`), except for those marked in the `dbt_project.yml` file. |\n| stg_ga4__traffic_sources | Creates a table that designates a `default_channel_grouping` via the `source`, `medium`, `campaign` columns. |\n| stg_ga4__user_props | Creates a table that unnests the `user_properties`, except for those marked in the `dbt_project.yml` file. |\n| stg_ga4__query_params | Maps any and all query parameters (e.g. `gclid`, `fbclid`, etc.) contained in each event's `page_location`. |\n| stg_ga4__conversions | Creates a table for the events that you mark as a `conversion_event` in the `dbt_project.yml` file. |\n| int_ga4__events_joined | ...[TO DO]... |\n| int_ga4__pages_grouped | ...[TO DO]... |\n| int_ga4__sessions_grouped | ...[TO DO]... |\n| int_ga4__users_grouped | ...[TO DO]... |\n\n# Macros\n***NOTE: These Macros are also not finalized \u0026 are likely to change.***\n\n### get_first(`by_column_name`,`from_column_name`) *[source](macros/get_positions.sql)*\nThis macro returns the `FIRST` position of a specified `from_column_name`, which is partioned by the `by_column_name`.\n\n**Args:**\n\n- `by_column_name` (required): The name of the column which you want to partition your selction by.\n- `from_column_name` (required): The name of the column to get the first value of. \n\n**Usage:**\n\n```sql\n{{ get_first('\u003cby_column_name\u003e', '\u003cfrom_column_name\u003e') }}\n```\n\n**Example:** Get the landing_page of a corresponding Session by selecting the first `page_path` using that Session's `session_key`.\n\n```sql\nSELECT\n  {{ get_first('session_key', 'page_path') }} AS landing_page\n  ...\n```\n\n### get_last(`by_column_name`,`from_column_name`) *[source](macros/get_positions.sql)*\nThis macro returns the `LAST` position of a specified `from_column_name`, which is partioned by the `by_column_name`.\n\n**Args:**\n\n- `by_column_name` (required): The name of the column which you want to partition your selction by.\n- `from_column_name` (required): The name of the column to get the last value of. \n\n**Usage:**\n\n```sql\n{{ get_last('\u003cby_column_name\u003e', '\u003cfrom_column_name\u003e') }}\n```\n\n**Example:** Get the last `event_key` for a corresponding Session using that Session's `session_key`.\n\n```sql\nSELECT\n  {{ get_last('session_key', 'event_key') }} AS last_session_event_key,\n  ...\n```\n\n### extract_hostname_from_url(`url`) *[source](macros/parse_url.sql)*\nThis macro extracts the `hostname` from a column containing a `url`.\n\n**Args:**\n\n- `url` (required): The column containting URLs.\n\n**Usage:**\n\n```sql\n{{ extract_hostname_from_url('\u003curl\u003e') }}\n```\n\n**Example:** Extract the `hostname` from the `page_location` column.\n\n```sql\nSELECT\n  {{ extract_hostname_from_url('page_location') }} AS page_hostname,\n  ...\n```\n\n### extract_query_string_from_url(`url`) *[source](macros/parse_url.sql)*\nThis macro extracts the `query_string` from a column containing a `url`.\n\n**Args:**\n\n- `url` (required): The column containting URLs.\n\n**Usage:**\n\n```sql\n{{ extract_query_string_from_url('\u003curl\u003e') }}\n```\n\n**Example:** Extract the `query_string` from the `page_location` column.\n\n```sql\nSELECT\n  {{ extract_query_string_from_url('page_location') }} AS page_query_string,\n  ...\n```\n\n### remove_query_parameters(`url`, `[parameters]`) *[source](macros/parse_url.sql)*\nThis macro removes the specified `parameters` from a column containing a `url`.\n\n**Args:**\n\n- `url` (required): The column containting URLs.\n- `parameters` (required, default=`[]`): A list of query parameters to remove from the URL.\n\n**Usage:**\n\n```sql\n{{ remove_query_parameters('\u003curl\u003e', '[parameters]')  }}\n```\n\n**Example:** Remove the parameters: `gclid`, `fbclid`, and `_ga` from the `page_location` column.\n\n```sql\n{% set parameters = ['gclid','fbclid','_ga'] %}\n\nSELECT\n  {{ remove_query_parameters('page_location', parameters) }} AS clean_page_location,\n  ...\n```\n\n### unnest_by_key(`column_to_unnest`, `key_to_extract`, `value_type` = \"string\") *[source](macros/unnest_by_keys.sql)*\nThis macro unnests a single key's value from an array. This macro will dynamically alias the sub-query with the name of the `column_to_unnest`.\n\n**Args:**\n\n- `column_to_unnest` (required): The array column to unnest the key's value from.\n- `key_to_extract` (required): The key by which to get the corresponding value for.\n- `value_type` (optional, default=\"string\"): The data type of the key's value column.\n\n**Usage:**\n\n```sql\n{{ unnest_by_key('\u003ccolumn_to_unnest\u003e', '\u003ckey_to_extract\u003e', '\u003cvalue_type\u003e') }}\n```\n\n**Example:** Unnest the corresponding values for the keys: `page_location` and `ga_session_number` from the nested `event_params` column.\n\n```sql\nSELECT\n  -- Unnest the default STRING value type\n  {{ unnest_by_key('event_params', 'page_location') }},\n  -- Unnest the INT value type\n  {{ unnest_by_key('event_params', 'ga_session_number',  'int') }},\n  ...\n```\n\n### unnest_by_key_alt(`column_to_unnest`, `key_to_extract`, `value_type` = \"string\") *[source](macros/unnest_by_keys.sql)*\nThis macro unnests a single key's value from an array. This macro allows for a custom alias named sub-query.\n\n**Args:**\n\n- `column_to_unnest` (required): The array column to unnest the key's value from.\n- `key_to_extract` (required): The key by which to get the corresponding value for.\n- `value_type` (optional, default=\"string\"): The data type of the key's value column.\n\n**Usage:**\n\n```sql\n{{ unnest_by_key_alt('\u003ccolumn_to_unnest\u003e', '\u003ckey_to_extract\u003e', '\u003cvalue_type\u003e') }} AS \u003ccustom_alias_name\u003e,\n```\n\n**Example:** Unnest the corresponding values for the keys: `page_location` and `ga_session_number` from the nested `event_params` column. \n\n```sql\nSELECT\n  -- Unnest the default STRING value type \u0026 use a custom alias\n  {{ unnest_by_key_alt('event_params', 'page_location') }} AS url, \n  -- Unnest the INT value type \u0026 use a custom alias\n  {{ unnest_by_key_alt('event_params', 'ga_session_number',  'int') }} AS session_number,\n  ...\n```\n\n### get_event_params() *[source](macros/unnest_by_keys.sql)*\nThis macro will dynamically return all of the `keys` and their corresponding `value_types` found in the `event_params` array column.\n\n- This macro will exclude event_params added to the `excluded_event_params` variable, which is specified in the `dbt_project.yml` file.\n\n**Usage / Example:**\n\n```sql\nSELECT\n  {% for event_param in get_event_params() -%}\n\n  {{ unnest_by_key('event_params', event_param['event_param_key'], event_param['event_param_value']) }}\n    \n  {{- \",\" if not loop.last }}\n  {% endfor %}\n  ...\n```\n\n### default_channel_grouping(`source`, `medium`, `source_category`) *[source](macros/default_channel_groupings.sql)*\nThis macro determines the `default_channel_grouping` and will result in one the following classifications: \n\n- `Direct`\n- `Paid Social`\n- `Oraginc Social`\n- `Email`\n- `Affiliates`\n- `Paid Shopping`\n- `Paid Search`\n- `Display`\n- `Other Advertising`\n- `Organic Search`\n- `Organic Video`\n- `Organic Shopping`\n- `Audio`\n- `SMS`\n- `(Other)`\n\n**Args:**\n\n- `source` (required): The source column used in determining the default channel grouping.\n- `medium` (required): The medium column used in determining the default channel grouping.\n- `source_category` (required): The source category column used in determining the default channel grouping. These are desiganted in the `ga4_source_categories.csv` seed file.\n\n**Usage:**\n\n```sql\n{{ default_channel_grouping('\u003csource\u003e', '\u003cmedium\u003e', '\u003csource_category\u003e') }}\n```\n\n**Example:** \n\n```sql\nSELECT\n  {{ default_channel_grouping('source', 'medium', 'source_category') }} AS default_channel_grouping,\n  ...\n```\n\n# Seeds\n| Seed File | Description |\n|-----------|-------------|\n| ga4_source_categories.csv| Google's mapping between `source` and `source_category`. More info and the download can be found [here](https://support.google.com/analytics/answer/9756891?hl=en). |\n\nMake sure to run `dbt seed` before running `dbt run`.\n\n# Installation \u0026 Configuration\n## Setup\n...[TO DO]...\n\n## Required Variables\nThis package assumes that you have an existing DBT project with a BigQuery profile and a BigQuery GCP instance available with GA4 event data loaded. Source data is located using the following variables which must be set in your `dbt_project.yml` file.\n\n```yaml\nvars:\n  project: '\u003cgcp_project\u003e' # Set your Project ID here.\n  dataset: '\u003cga4_dataset\u003e' # Set your Dataset name here.\n  start_date: 'YYYYMMDD'   # Set the start date that you want to retrieve data from.\n  frequency: 'daily'       # daily|streaming|daily+streaming Match to the type of export configured in GA4; daily+streaming appends today's intraday data to daily data.\n```\n\nIf you don't have any GA4 data of your own, you can connect to Google's public data set with the following settings:\n\n```yaml\nvars:\n  project: 'bigquery-public-data'\n  dataset: 'ga4_obfuscated_sample_ecommerce'\n  start_date: '20210120'\n```\nFind more info about the GA4 obfuscated dataset [here](https://developers.google.com/analytics/bigquery/web-ecommerce-demo-dataset). \n\n## Optional Variables\n\n***NOTE: These Variables are also NOT finalized \u0026 are LIKELY to change.***\n\n### Query Parameter Exclusions\nSetting any `query_parameter_exclusions` will remove query string parameters from the `page_location` field for all downstream processing. Original parameters are captured in a new `original_page_location` field. Ex:\n\n```yaml\nvars: \n  query_parameter_exclusions: ['gclid', 'fbclid', '_ga'] \n```\n\n### Conversion Events\nSpecific events can be set as conversions with the `conversion_events` variable in your `dbt_project.yml` file. These events will be counted against each session and included in the final mart models. Ex:\n\n```yaml\nvars:\n  conversion_events: ['purchase', 'download']\n```\n\n### Consideration Events\nSpecific events can be set as considerations with the `conversion_events` variable in your `dbt_project.yml` file. These events will be counted against each session and included in the final mart models. Ex:\n\n```yaml\nvars:\n  consideration_events: ['cta_click', 'view_search_results']\n```\n\n### Funnel Stages [TO DO]\nSet specific events to be stages in a funnel.\n\n```yaml\nvars:\n  funnel_stages: ['begin_checkout', 'add_shipping_info', 'add_payment_info', 'purchase']\n```\n\n### Excluded Events [TO DO]\nExclude specific events from the final tables.\n\n```yaml\nvars:\n  excluded__events: ['session_start']\n```\n\n### Excluded Event parameters [TO DO]\nExclude specific event parameters from the final tables.\n\n```yaml\nvars:\n  excluded__event_params: ['ga_session_id', 'page_location', 'ga_session_number', 'session_engaged', 'engagement_time_msec', 'entrances', 'page_title', 'page_referrer', 'source', 'medium', 'campaign', 'debug_mode', 'term', 'clean_event', 'value', 'tax', 'coupon', 'promotion_name', 'transaction_id']\n```\n\n### Excluded Columns [TO DO]\nExclude specific default columns from the final tables.\n\n```yaml\nvars:\n  excluded__columns: ['event_previous_timestamp', 'event_bundle_sequence_id', 'event_server_timestamp_offset', 'user_id', 'user_pseudo_id', 'stream_id', 'ga_session_id', 'privacy_info', 'event_dimensions', 'app_info']\n```\n\n### Excluded User Properties [TO DO]\nExclude specific user properties from the final tables.\n\n```yaml\nvars:\n  excluded__user_props: ['logged_in']\n```\n\n### Included Query Parameters [TO DO]\nInclude specific query parameters to be in the final tables.\n\n```yaml\nvars:\n  included__query_params: ['utm_source', 'utm_medium', 'utm_campaign', 'utm_content', 'utm_term', 'gclid', 'fbclid', 'gclsrc', '_ga']\n```\n\n## Resources \u0026 References:\n\n  - GA4 Resources:\n    - [GA4 BigQuery Export schema](https://support.google.com/analytics/answer/7029846?hl=en\u0026ref_topic=9359001)\n    - [Intro To GA4 in BQ](https://www.ga4bigquery.com/introduction-to-google-analytics-4-ga4-export-data-in-bigquery/)\n  - SQL \u0026 BigQuery Resources: \n    - [BigQuery Docs](https://cloud.google.com/bigquery/docs)\n    - [BigQuery: Functions, Operators, and Conditionals](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators)\n    - [BigQuery: Query Syntax](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax)\n    - [BigQuery: Best Practices](https://cloud.google.com/bigquery/docs/best-practices-performance-overview)\n    - [SQL Formatter](https://smalldev.tools/sql-formatter-online)\n  - dbt Resources:\n    - [Getting Started with dbt Cloud](https://docs.getdbt.com/guides/getting-started)\n      - [Getting Started with dbt Core](https://docs.getdbt.com/guides/getting-started/learning-more/getting-started-dbt-core)\n      - [Refactoring legacy SQL to dbt](https://docs.getdbt.com/guides/getting-started/learning-more/refactoring-legacy-sql)\n    - [Best Practices](https://docs.getdbt.com/guides/best-practices)\n    - [GitLab's dbt Guide](https://about.gitlab.com/handbook/business-technology/data-team/platform/dbt-guide/)\n    - [Jinja Template Designer Documentation](https://jinja.palletsprojects.com/en/3.1.x/templates)\n  - Project References:\n    - [GA4 dbt Package](https://github.com/Velir/dbt-ga4.git)\n    - [Stacktonic dbt Example Project](https://github.com/stacktonic-com/stacktonic-dbt-example-project)\n    - Also inspired by [this](https://github.com/llooker/ga_four_block_dev/blob/master/views/sessions.view.lkml)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodataconsulting%2Fdbt_ga4_project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodataconsulting%2Fdbt_ga4_project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodataconsulting%2Fdbt_ga4_project/lists"}