{"id":13703421,"url":"https://github.com/sutrolabs/dbt_census_utils","last_synced_at":"2025-05-05T07:30:57.974Z","repository":{"id":160507752,"uuid":"602260610","full_name":"sutrolabs/dbt_census_utils","owner":"sutrolabs","description":"Utility functions for data activation.","archived":false,"fork":false,"pushed_at":"2023-06-03T15:46:42.000Z","size":1147,"stargazers_count":19,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-01-28T23:08:40.175Z","etag":null,"topics":["census","dbt","dbt-packages","ga4"],"latest_commit_sha":null,"homepage":"https://www.getcensus.com/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sutrolabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-02-15T20:45:57.000Z","updated_at":"2023-10-13T01:07:46.000Z","dependencies_parsed_at":"2024-01-06T13:56:08.515Z","dependency_job_id":"8f0cd97b-ba59-4ded-be99-635c88cf8d91","html_url":"https://github.com/sutrolabs/dbt_census_utils","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sutrolabs%2Fdbt_census_utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sutrolabs%2Fdbt_census_utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sutrolabs%2Fdbt_census_utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sutrolabs%2Fdbt_census_utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sutrolabs","download_url":"https://codeload.github.com/sutrolabs/dbt_census_utils/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252458357,"owners_count":21751023,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["census","dbt","dbt-packages","ga4"],"created_at":"2024-08-02T21:00:54.610Z","updated_at":"2025-05-05T07:30:54.717Z","avatar_url":"https://github.com/sutrolabs.png","language":null,"funding_links":[],"categories":["Packages"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003ca alt=\"License\"\n        href=\"https://github.com/census/dbt_census_utils/blob/main/LICENSE\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/License-Apache%202.0-blue.svg\" /\u003e\u003c/a\u003e\n    \u003ca alt=\"dbt-core\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/dbt_Core™_version-\u003e=1.3.0_,\u003c2.0.0-orange.svg\" /\u003e\u003c/a\u003e\n    \u003ca alt=\"Maintained?\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Maintained%3F-yes-green.svg\" /\u003e\u003c/a\u003e\n    \u003ca alt=\"PRs\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Contributions-welcome-blueviolet\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n# Census Utils dbt Package ([Docs](https://sutrolabs.github.io/dbt_census_utils/#!/overview/census_utils))\n# 🎁 What does this dbt package do?\n- Adds a number of macros that are useful when transforming data to be synced via reverse ETL (with Census or your own pipelines).\n- Adds documentation for the macros at: [dbt docs site](https://sutrolabs.github.io/dbt_census_utils/#!/overview/census_utils).\n\n# 👩‍💻 How do I use the dbt package?\n\n## Step 1: Prerequisites\nTo use this dbt package, you must have the following:\n\n- A **BigQuery**, **Snowflake**, or **Redshift** data warehouse.\n\n## Step 2: Install the package\nInclude the following census_utils package version in your `packages.yml` file, then run 'dbt deps':\n\u003e TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.\n```yml\npackages:\n  - package: sutrolabs/census_utils\n    version: [\"\u003e=0.1.0\", \"\u003c1.0.0\"]\n\n```\n\n## Step 3: Run dbt seed\nThis package uses seeds for macros such as converting country codes to country names.  Run 'dbt seed' after 'dbt deps' to materialize these seeds in your data warehouse.\n\n## (Optional) Step 4: Define internal user variables\nThe [is_internal macro (\u003ca href=\"macros/is_internal.sql\"\u003esource\u003c/a\u003e)](#is_internal-source) identifies internal users based off of several potential methods: their email address domain, an existing list of email addresses, or an existing list of IP Addresses. If you want to use the `is_internal` macro, you'll need to specify at least one of these approaches. Add variables to your root `dbt_project.yml` file to reflect the domain of your company and the relations and columns where internal users are tracked.  These relations can be a dbt seed or a dbt model. Common methods of maintaining this relation include: a seed file of internal users and IP address, a dbt model that identifies internal users directly from your application database, or a dbt model referencing a Google Sheet of internal users.  For example, if your company used the domains sawtelleanalytics.com and sawtelleanalytics.co.uk, and you have a dbt seed called 'my_internal_users' with an email_address column for the emails of internal users and an ip_address column for the IPs of internal users, you would add this to your vars:\n\n```yml\nvars:\n  internal_domain: ('sawtelleanalytics.com', 'sawtelleanalytics.co.uk')\n  internal_email_relation: 'my_internal_users'\n  internal_email_column: 'email_address'\n  internal_ip_relation: 'my_internal_users'\n  internal_ip_column: 'ip_address'\n```\n\n## (Optional) Step 5: Sync your dbt models to destinations with Census\n\u003cdetails\u003e\u003csummary\u003eExpand for details\u003c/summary\u003e\n\u003cbr\u003e\n    \nCensus lets you sync your dbt models to destinations such as Salesforce, Hubspot, Zendesk, Facebook, and Google.  Learn how to [sync your data](https://www.getcensus.com/demo).\n\u003c/details\u003e\n\n# List of macros:\n* [parse_ga4_client_id (\u003ca href=\"macros/parse_ga4_client_id.sql\"\u003esource\u003c/a\u003e)](#parse_ga4_client_id-source)\n* [clean (\u003ca href=\"macros/clean.sql\"\u003esource\u003c/a\u003e)](#clean-source)\n* [is_internal (\u003ca href=\"macros/is_internal.sql\"\u003esource\u003c/a\u003e)](#is_internal-source)\n* [extract_email_domain (\u003ca href=\"macros/extract_email_domain.sql\"\u003esource\u003c/a\u003e)](#extract_email_domain-source)\n* [is_personal_email (\u003ca href=\"macros/is_personal_email.sql\"\u003esource\u003c/a\u003e)](#is_personal_email-source)\n* [get_country_code (\u003ca href=\"macros/get_country_code.sql\"\u003esource\u003c/a\u003e)](#get_country_code-source)\n\n\n\n## parse_ga4_client_id ([source](macros/parse_ga4_client_id.sql))\n\nThis macro takes a Google Analytics 4 client ID and returns either the unique ID part, the timestamp part, or the entire ID without any 'GA1.2' type prefix before it.\n\n**Args:**\n\n- `client_id` (required):  The raw GA 4 client ID to parse.\n- `extract_value` (required): Specifies what to extract from the raw client ID.  Should be either 'unique_id', 'timestamp', or 'client_id'.\n\n**Usage:**\n\n```sql\nselect \n    ga4_client_id,\n    {{ census_utils.parse_ga4_client_id('ga4_client_id', 'unique_id') }} as unique_id,\n    {{ census_utils.parse_ga4_client_id('ga4_client_id', 'timestamp') }} as timestamp,\n    {{ census_utils.parse_ga4_client_id('ga4_client_id', 'client_id') }} as client_id\nfrom ga4_client\n```\n\n## clean ([source](macros/clean.sql))\n\nThis macro cleans fields so that they will be accepted by APIs such as [Facebook Ads](https://developers.facebook.com/docs/marketing-api/audiences/guides/custom-audiences/#hash) or [Google Ads](https://developers.google.com/google-ads/api/docs/conversions/enhance-conversions#normalization_and_hashing).\n\n**Args:**\n\n- `field` (required): The name to be cleaned.\n- `destination` (required): The destination the field will be sent to.  Currently supports 'facebook' or 'google'.\n- `type` (required): The type of field, currently supports 'name' (for first or last name), 'email', 'city', 'country', or 'zip'.  Non-US postal codes should not be formatted by this macro as requirements for those vary.\n\n**Usage:**\n\n```sql\nselect \n    {{ census_utils.clean('fn',destination='facebook',type='name') }} as cleaned_facebook_fn,\n    {{ census_utils.clean('ln',destination='facebook',type='name') }} as cleaned_facebook_ln,\n    {{ census_utils.clean('country',destination='facebook',type='country') }} as cleaned_facebook_country,\n    {{ census_utils.clean('city',destination='facebook',type='city') }} as cleaned_facebook_city,\n    case when {{ census_utils.clean('country_to_clean','facebook','country') }} = 'us' then {{ census_utils.clean('zip','facebook','zip') }} else zip end as cleaned_facebook_zip,\n    {{ census_utils.clean('email_address','facebook','email') }} as cleaned_facebook_email\n```\n\n## is_internal ([source](macros/is_internal.sql))\n\nThis macro reports whether a user is an internal user based on their email domain, email address, or IP address.  Relies on at least one variable being set in [dbt_project.yml](https://github.com/sutrolabs/dbt_census_utils#optional-step-4-define-internal-user-variables).\n\n**Args:**\n\n- `email` (optional):  The email address of the user.\n- `ip_address` (optional): The IP address of the user.\n\n**Usage:**\n\n```sql\nselect \n    email_address\n    , ip_address\n    , {{ census_utils.is_internal(email='email_address',ip_address='ip_address') }} as is_internal_user\n    , {{ census_utils.is_internal(email='email_address') }} as is_internal_email\n    , {{ census_utils.is_internal(ip_address='ip_address') }} as is_internal_ip\nfrom\n    users\n```\n\n## extract_email_domain ([source](macros/extract_email_domain.sql))\n\nThis macro extracts the domain from an email address.\n\n**Args:**\n\n- `email` (required):  The email address of the user.\n\n**Usage:**\n\n```sql\nselect \n    email_address,\n    {{ census_utils.extract_email_domain('email_addresses') }} as email_domain\n```\n\n## is_personal_email ([source](macros/is_personal_email.sql))\n\nThis macro determines whether an email address is personal, based on a list of common personal email domains.\n\n**Args:**\n\n- `email` (required):  The email address of the user.\n\n**Usage:**\n\n```sql\nselect \n    email_address,\n    {{ census_utils.is_personal_email('email_addresses') }} as is_personal_email\n```\n\n## get_country_code ([source](macros/get_country_code.sql))\n\nThis macro converts a country name to a [ISO 3166](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) country code.\n**Args:**\n\n- `country_name` (required):  The country name to be converted to a code.\n\n**Usage:**\n\n```sql\nselect \n    country_name,\n    {{ census_utils.get_country_code('country_name') }} as country_code\n```\n\n# 🎢 Does this package have dependencies?\nThis dbt package is dependent on the following dbt packages. Please be aware that these dependencies are installed by default within this package. For more information on the following packages, refer to the [dbt hub](https://hub.getdbt.com/) site.\n\u003e IMPORTANT: If you have any of these dependent packages in your own `packages.yml` file, we recommend that you remove them from your root `packages.yml` to avoid package version conflicts.\n    \n```yml\npackages:\n    - package: dbt-labs/dbt_utils\n      version: [\"\u003e=.9.0\", \"\u003c2.0.0\"]\n```\n# 🤝 How is this package maintained and can I contribute?\n## Package Maintenance\nThe Census team maintaining this package _only_ maintains the latest version of the package. We highly recommend you stay consistent with the [latest version](https://hub.getdbt.com/census/census_utils/) of the package and refer to the changelog and release notes for more information on changes across versions.\n\n## Contributions\nWe highly encourage and welcome contributions to this package. Check out [this dbt Discourse article](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) on the best workflow for contributing to a package!\n\n# 🧭 How can I get help or make suggestions?\n- If you have questions or want to reach out for help, please refer to the [GitHub Issue](https://github.com/sutrolabs/dbt_census_utils/issues/new/choose) section to find the right avenue of support for you.\n- If you would like to provide feedback to the dbt package team at Census or would like to request a new dbt package, please join the [Operational Analytics Slack](https://www.operationalanalytics.club/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsutrolabs%2Fdbt_census_utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsutrolabs%2Fdbt_census_utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsutrolabs%2Fdbt_census_utils/lists"}