Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/fivetran/dbt_twilio

Fivetran data models for Twilio using dbt.
https://github.com/fivetran/dbt_twilio

dbt dbt-packages fivetran twilio

Last synced: 5 months ago
JSON representation

Fivetran data models for Twilio using dbt.

Awesome Lists containing this project

README 

          

# Twilio dbt Package





    

        

    

        

    

        

    

        

    

        




This dbt package transforms data from Fivetran's Twilio connector into analytics-ready tables.



## Resources



- Number of materialized models¹: 20

- Connector documentation

  - [Twilio connector documentation](https://fivetran.com/docs/connectors/applications/twilio)

  - [Twilio ERD](https://fivetran.com/docs/connectors/applications/twilio#schemainformation)

- dbt package documentation

  - [GitHub repository](https://github.com/fivetran/dbt_twilio)

  - [dbt Docs](https://fivetran.github.io/dbt_twilio/#!/overview)

  - [DAG](https://fivetran.github.io/dbt_twilio/#!/overview?g_v=1)

  - [Changelog](https://github.com/fivetran/dbt_twilio/blob/main/CHANGELOG.md)



## What does this dbt package do?

This package enables you to produce modeled tables that leverage Twilio data. It creates enriched models with metrics focused on messaging information and account-level aggregations.



### Output schema

Final output tables are generated in the following target schema:



```

._twilio

```



### Final output tables



By default, this package materializes the following final tables:



| Table | Description |

| :---- | :---- |

| [twilio__message_enhanced](https://fivetran.github.io/dbt_twilio/#!/model/model.twilio.twilio__message_enhanced) | Tracks details for every SMS and MMS message sent or received through Twilio including status, direction, content, pricing, and delivery information to analyze messaging activity and performance. 
**Example Analytics Questions:**
    

  • What is the delivery success rate by message status (sent, delivered, failed) and direction (inbound vs outbound)?
    • 

  • Which phone numbers or messaging campaigns generate the most messages and highest costs?
    • 

  • What are the most common error codes and messages for failed deliveries?
    • 

|

| [twilio__number_overview](https://fivetran.github.io/dbt_twilio/#!/model/model.twilio.twilio__number_overview) | Provides aggregate messaging metrics for each phone number including total messages by status, inbound/outbound volumes, and total spend to understand performance at the number level. 
**Example Analytics Questions:**
    

  • Which phone numbers send or receive the most messages?
    • 

  • What are the delivery rates (delivered vs failed messages) and total costs by phone number?
    • 

  • How do inbound versus outbound message volumes compare across phone numbers?
    • 

|

| [twilio__account_overview](https://fivetran.github.io/dbt_twilio/#!/model/model.twilio.twilio__account_overview) | Summarizes total messaging activity and costs across all Twilio accounts to monitor overall communication volume, spending, and performance. 
**Example Analytics Questions:**
    

  • What is our total messaging volume and spend across all accounts?
    • 

  • How are messaging costs and volumes trending over time by account?
    • 

  • Which accounts have the highest message volumes or spending?
    • 

|



¹ Each Quickstart transformation job run materializes these models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`.



---



## Prerequisites

To use this dbt package, you must have the following:



- At least one Fivetran Twilio connection syncing data into your destination.

- A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination.



## How do I use the dbt package?

You can either add this dbt package in the Fivetran dashboard or import it into your dbt project:



- To add the package in the Fivetran dashboard, follow our [Quickstart guide](https://fivetran.com/docs/transformations/data-models/quickstart-management).

- To add the package to your dbt project, follow the setup instructions in the dbt package's [README file](https://github.com/fivetran/dbt_twilio/blob/main/README.md#how-do-i-use-the-dbt-package) to use this package.



### Install the package

Include the following Twilio package version in your `packages.yml` file:

> TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.

```yaml

packages:

  - package: fivetran/twilio

    version: [">=1.3.0", "<1.4.0"]

```



> All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/twilio_source` in your `packages.yml` since this package has been deprecated.



#### Databricks Dispatch Configuration

If you are using a Databricks destination with this package you will need to add the below (or a variation of the below) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils` then the `dbt-labs/dbt_utils` packages respectively.

```yml

dispatch:

  - macro_namespace: dbt_utils

    search_order: ['spark_utils', 'dbt_utils']

```



### Define database and schema variables



#### Option A: Single connection

By default, this package runs using your [destination](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile) and the `twilio` schema. If this is not where your Twilio data is (for example, if your Twilio schema is named `twilio_fivetran`), add the following configuration to your root `dbt_project.yml` file:



```yml

vars:

  twilio:

    twilio_database: your_database_name

    twilio_schema: your_schema_name

```



#### Option B: Union multiple connections

If you have multiple Twilio connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. For each source table, the package will union all of the data together and pass the unioned table into the transformations. The `source_relation` column in each model indicates the origin of each record.



To use this functionality, you will need to set the `twilio_sources` variable in your root `dbt_project.yml` file:



```yml

# dbt_project.yml



vars:

  twilio:

    twilio_sources:

      - database: connection_1_destination_name # Required

        schema: connection_1_schema_name # Required

        name: connection_1_source_name # Required only if following the step in the following subsection



      - database: connection_2_destination_name

        schema: connection_2_schema_name

        name: connection_2_source_name

```



##### Recommended: Incorporate unioned sources into DAG

> *If you are running the package through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore), the below step is necessary in order to synchronize model runs with your Twilio connections. Alternatively, you may choose to run the package through Fivetran [Quickstart](https://fivetran.com/docs/transformations/quickstart), which would create separate sets of models for each Twilio source rather than one set of unioned models.*



By default, this package defines one single-connection source, called `twilio`, which will be disabled if you are unioning multiple connections. This means that your DAG will not include your Twilio sources, though the package will run successfully.



To properly incorporate all of your Twilio connections into your project's DAG:

1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_twilio.yml` [file](https://github.com/fivetran/dbt_twilio/blob/main/models/staging/src_twilio.yml).



```yml

# a .yml file in your root project



version: 2



sources:

  - name:  # ex: Should match name in twilio_sources

    schema: 

    database: 

    loader: fivetran

    config:

      loaded_at_field: _fivetran_synced

      freshness: # feel free to adjust to your liking

        warn_after: {count: 72, period: hour}

        error_after: {count: 168, period: hour}



    tables: # copy and paste from twilio/models/staging/src_twilio.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once

```



> **Note**: If there are source tables you do not have (see [Enabling/Disabling Models](https://github.com/fivetran/dbt_twilio?tab=readme-ov-file#enablingdisabling-models)), you may still include them, as long as you have set the right variables to `False`.



2. Set the `has_defined_sources` variable (scoped to the `twilio` package) to `True`, like such:

```yml

# dbt_project.yml

vars:

  twilio:

    has_defined_sources: true

```



### Enabling/Disabling Models



Your Twilio connection might not sync every table that this package expects, for example if you are not using the Twilio messaging service feature. If your syncs exclude certain tables, it is either because you do not use that functionality in Twilio or have actively excluded some tables from your syncs. In order to enable or disable the relevant tables in the package, you will need to add the following variable(s) to your `dbt_project.yml` file.



By default, all variables are assumed to be `true`.



```yml

vars:

  using_twilio_call: False # Disable this if not using call

  using_twilio_messaging_service: False # Disable this if not using messaging_service

  using_twilio_usage_record: False # Disable this if not using usage_record



```



### (Optional) Additional configurations



Expand/Collapse configurations



#### Changing the Build Schema



By default, this package will build the Twilio final models within a schema titled (`` + `_twilio`), intermediate models in (`` + `_int_twilio`), and staging models within a schema titled (`` + `_stg_twilio`) in your target database. If this is not where you would like your modeled Twilio data to be written to, add the following configuration to your `dbt_project.yml` file:



```yml

# dbt_project.yml



...

models:

    twilio:

      +schema: my_new_schema_name # Leave +schema: blank to use the default target_schema.

      staging:

        +schema: my_new_schema_name # Leave +schema: blank to use the default target_schema.

```



> Note that if your profile does not have permissions to create schemas in your warehouse, you can set each `+schema` to blank. The package will then write all tables to your pre-existing target schema.



#### Change the source table references

If an individual source table has a different name than the package expects (but is in the same schema and database as the other tables), add the table name as it appears in your destination to the respective variable:



> IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_twilio/blob/main/dbt_project.yml) variable declarations to see the expected names.



```yml

vars:

    twilio__identifier: your_table_name 

```



### (Optional) Orchestrate your models with Fivetran Transformations for dbt Core™

Expand for more details





Fivetran offers the ability for you to orchestrate your dbt project through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore). Learn how to set up your project for orchestration through Fivetran in our [Transformations for dbt Core setup guides](https://fivetran.com/docs/transformations/dbt/setup-guide#transformationsfordbtcoresetupguide).



## Does this package have dependencies?

This dbt package is dependent on the following dbt packages. These dependencies are installed by default within this package. For more information on the following packages, refer to the [dbt hub](https://hub.getdbt.com/) site.

> IMPORTANT: If you have any of these dependent packages in your own `packages.yml` file, we highly recommend that you remove them from your root `packages.yml` to avoid package version conflicts.



```yml

packages:

    - package: fivetran/fivetran_utils

      version: [">=0.4.0", "<0.5.0"]



    - package: dbt-labs/dbt_utils

      version: [">=1.0.0", "<2.0.0"]



    - package: dbt-labs/spark_utils

      version: [">=0.3.0", "<0.4.0"]

```



## How is this package maintained and can I contribute?



### Package Maintenance

The Fivetran team maintaining this package only maintains the [latest version](https://hub.getdbt.com/fivetran/twilio/latest/) of the package. We highly recommend you stay consistent with the latest version of the package and refer to the [CHANGELOG](https://github.com/fivetran/dbt_twilio/blob/main/CHANGELOG.md) and release notes for more information on changes across versions.



### Contributions

A small team of analytics engineers at Fivetran develops these dbt packages. However, the packages are made better by community contributions.



We highly encourage and welcome contributions to this package. Learn how to contribute to a package in dbt's [Contributing to an external dbt package article](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657).



## Are there any resources available?

- If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_twilio/issues/new/choose) section to find the right avenue of support for you.

- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).