{"id":24320901,"url":"https://github.com/matt-strautmann/dbt-bigquery-ecommerce-quickstart","last_synced_at":"2025-07-30T17:34:53.880Z","repository":{"id":270420198,"uuid":"910170473","full_name":"matt-strautmann/dbt-bigquery-ecommerce-quickstart","owner":"matt-strautmann","description":"Welcome to the dbt-BigQuery Quickstart Project! 🎉 This repository is designed as a hands-on guide to help you build a modern data stack leveraging powerful tools like Airbyte for ingestion, dbt for transformation, and BigQuery for storage and analytics. ","archived":false,"fork":false,"pushed_at":"2025-01-16T15:11:43.000Z","size":48,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-17T23:49:40.620Z","etag":null,"topics":["bigquery","dbt","e-commerce","quickstarts"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matt-strautmann.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-30T17:03:19.000Z","updated_at":"2025-01-16T15:11:44.000Z","dependencies_parsed_at":"2024-12-31T03:20:43.485Z","dependency_job_id":"dba18880-c163-4d03-abce-63e845e05bb1","html_url":"https://github.com/matt-strautmann/dbt-bigquery-ecommerce-quickstart","commit_stats":null,"previous_names":["matt-strautmann/dbt-bq-getting-started-template","matt-strautmann/dbt-bigquery-quickstart-project","matt-strautmann/dbt-bigquery-ecommerce-quickstart"],"tags_count":0,"template":true,"template_full_name":null,"purl":"pkg:github/matt-strautmann/dbt-bigquery-ecommerce-quickstart","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matt-strautmann%2Fdbt-bigquery-ecommerce-quickstart","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matt-strautmann%2Fdbt-bigquery-ecommerce-quickstart/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matt-strautmann%2Fdbt-bigquery-ecommerce-quickstart/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matt-strautmann%2Fdbt-bigquery-ecommerce-quickstart/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matt-strautmann","download_url":"https://codeload.github.com/matt-strautmann/dbt-bigquery-ecommerce-quickstart/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matt-strautmann%2Fdbt-bigquery-ecommerce-quickstart/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267911563,"owners_count":24164462,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","dbt","e-commerce","quickstarts"],"created_at":"2025-01-17T16:19:20.910Z","updated_at":"2025-07-30T17:34:53.828Z","avatar_url":"https://github.com/matt-strautmann.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🚀 Getting Started with dbt and Airbyte\n\nWelcome to your modern data stack template! This project demonstrates how to build a scalable data warehouse using:\n- **Airbyte** for data ingestion\n- **dbt** for transformation\n- **BigQuery** for storage and computing\n\n## Prerequisites\n\n1. **VSCode** installed ([Download here](https://code.visualstudio.com/download))\n2. **Turntable.so extension** installed in VSCode ([Install here](https://marketplace.visualstudio.com/items?itemName=Turntable.turntable))\n3. **Python** installed (3.8 or higher)\n4. **Google Cloud account** with BigQuery enabled\n5. **Airbyte** instance set up with sources configured\n\n## 🏗️ Project Structure\n```\n📂 models/\n├── 📁 staging/              # 🛠️ Raw data standardization\n│   ├── 📁 stg_stripe/       # 💳 Payment processing\n│   │   ├── 📁 base/         # 📜 Raw JSON parsing\n│   │   │   └── 📄 base_stripe__customers.sql\n│   │   ├── 📄 stg_stripe__customers.sql\n│   │   └── 📄 _stripe_sources.yml\n│   │\n│   ├── 📁 stg_hubspot/      # 📈 Marketing automation\n│   │   ├── 📁 base/\n│   │   └── 📄 _hubspot_sources.yml\n│   │\n│   └── 📁 stg_shopify/      # 🛒 E-commerce platform\n│       ├── 📁 base/\n│       └── 📄 _shopify_sources.yml\n│\n├── 📁 intermediate/         # 🔍 Business logic layer\n│   ├── 📁 finance/\n│   ├── 📁 marketing/\n│   └── 📁 sales/\n│\n└── 📁 marts/                # 📊 Business-specific models\n    ├── 📁 core/             # 🔑 Core business entities\n    ├── 📁 finance/          # 💰 Finance-specific models\n    ├── 📁 marketing/        # 📣 Marketing-specific models\n    └── 📁 sales/            # 🛒 Sales-specific models\n```\n\n## 🔄 Setup Instructions\n\n1. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/yourusername/dbt-bigquery-quickstart-project.git\n   cd dbt-bigquery-quickstart-project\n   ```\n\n2. **Set Up Environment Variables**:\n   ```bash\n   cp .env.example .env\n   # Edit .env with your configurations\n   ```\n\n3. **Configure dbt Profile**:\n   ```bash\n   cp profiles.yml.example ~/.dbt/profiles.yml\n   # Edit profiles.yml with your BigQuery details\n   ```\n\n4. **Install Dependencies**:\n   ```bash\n   pip install dbt-core dbt-bigquery\n   dbt deps\n   ```\n\n## 🔌 Airbyte Integration\n\nThis template is designed to work with Airbyte's BigQuery destination. Key points:\n\n1. **Raw Data Structure**:\n   - Airbyte creates tables with prefix `_airbyte_raw_`\n   - Data is stored in JSON format in `_airbyte_data` column\n   - Each record has `_airbyte_emitted_at` timestamp\n\n2. **Base Models**:\n   ```sql\n   -- Example: models/staging/stg_stripe/base/base_stripe__customers.sql\n   select \n       JSON_EXTRACT_SCALAR(_airbyte_data, '$.id') as customer_id,\n       JSON_EXTRACT_SCALAR(_airbyte_data, '$.email') as email,\n       _airbyte_emitted_at as ingested_at\n   from {{ source('stripe', '_airbyte_raw_customers') }}\n   ```\n\n3. **Source Configuration**:\n   ```yaml\n   # Example: models/staging/stg_stripe/_stripe_sources.yml\n   version: 2\n   sources:\n     - name: stripe\n       database: \"{{ env_var('DBT_PROJECT_ID') }}\"\n       schema: \"{{ env_var('AIRBYTE_SCHEMA', 'raw') }}\"\n       loader: airbyte\n       loaded_at_field: _airbyte_emitted_at\n       tables:\n         - name: _airbyte_raw_customers\n   ```\n\n## 🏭 Development Workflow\n\n1. **Set Up Airbyte Source**:\n   - Configure source in Airbyte UI\n   - Set destination to BigQuery\n   - Note the destination schema\n\n2. **Update Environment Variables**:\n   ```bash\n   DBT_PROJECT_ID=your-project-id\n   AIRBYTE_SCHEMA=raw\n   DBT_STAGING_SCHEMA=staging\n   ```\n\n3. **Create Base Models**:\n   - Parse JSON data from Airbyte\n   - Use `JSON_EXTRACT_SCALAR` for BigQuery\n   - Add basic data type conversions\n\n4. **Create Staging Models**:\n   - Add business logic and cleaning\n   - Implement standard naming\n   - Add data quality tests\n\n5. **Build Marts**:\n   - Combine data from multiple sources\n   - Create business-specific views\n   - Optimize for analysis\n\n## 📊 Data Quality\n\n1. **Source Freshness**:\n   ```yaml\n   sources:\n     - name: stripe\n       freshness:\n         warn_after: {count: 12, period: hour}\n         error_after: {count: 24, period: hour}\n   ```\n\n2. **Data Tests**:\n   ```yaml\n   models:\n     - name: stg_stripe__customers\n       columns:\n         - name: customer_id\n           tests:\n             - unique\n             - not_null\n   ```\n\n## 🔍 Monitoring\n\n1. **Airbyte Sync Status**:\n   - Check Airbyte UI for sync status\n   - Monitor `_airbyte_emitted_at` for freshness\n\n2. **dbt Run Status**:\n   - Use `dbt source freshness`\n   - Check model test results\n\n## 📚 Learning Resources\n\n1. **Airbyte Resources**:\n   - [Airbyte Docs](https://docs.airbyte.io/)\n   - [BigQuery Setup Guide](https://docs.airbyte.io/integrations/destinations/bigquery)\n\n2. **dbt Resources**:\n   - [dbt Docs](https://docs.getdbt.com/)\n   - [BigQuery Specific Guides](https://docs.getdbt.com/reference/resource-configs/bigquery-configs)\n\n3. **Community**:\n   - [dbt Slack](https://community.getdbt.com/)\n   - [Airbyte Slack](https://slack.airbyte.io/)\n\n## 🆘 Need Help?\n\n1. Check error messages in Turntable.so\n2. Review Airbyte logs for sync issues\n3. Visit [dbt Discourse](https://discourse.getdbt.com/)\n4. Create an issue in this repository\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Submit a pull request\n\n## 📝 License\n\nMIT License - see [LICENSE](LICENSE) file\n\n## BigQuery Setup Guide\n\nFor detailed instructions on setting up your BigQuery connection, including OAuth authentication and testing, see our [BigQuery Setup Guide](docs/bigquery_setup.md).\n\n---\n# ⭐ Credits \u0026 Connect  \n\n## 🚀 About This Repository  \nThis repository is maintained by [Matt Strautmann](https://www.linkedin.com/in/mattstrautmann), an experienced **is working closely with Founder/CEOs to use your Data to improve your bottom line. Period.** Let me help you **trust your data. know your customer. improve your bottom line.**  \n\n### Why Star This Repository?  \nStarring this repository helps me understand which tools, templates, and projects bring the most value to the community. Your support motivates me to keep producing high-quality content and maintain these resources for everyone!  \n\n## 🌟 Support This Project  \nIf this repository has helped you:  \n1. Give it a ⭐ to show your appreciation!  \n2. Share it with others who might find it useful.  \n\n## 🤝 Connect with Me  \nI’d love to hear how you’re using this repository or discuss how I can help with your next project. Let’s connect:  \n- **LinkedIn**: [Matt Strautmann](https://www.linkedin.com/in/mattstrautmann)  \n- **GitHub**: [Matt Strautmann](https://github.com/matt-strautmann)\n\n---\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatt-strautmann%2Fdbt-bigquery-ecommerce-quickstart","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatt-strautmann%2Fdbt-bigquery-ecommerce-quickstart","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatt-strautmann%2Fdbt-bigquery-ecommerce-quickstart/lists"}