{"id":30175314,"url":"https://github.com/datacoves/workshop__snowflake_dynamic_tables","last_synced_at":"2025-08-23T09:09:16.012Z","repository":{"id":301256834,"uuid":"1008641974","full_name":"datacoves/workshop__snowflake_dynamic_tables","owner":"datacoves","description":"This repository contains the examples shared during the Datacoves Webinar","archived":false,"fork":false,"pushed_at":"2025-06-25T23:19:42.000Z","size":22,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-17T23:44:02.532Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datacoves.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-25T21:38:40.000Z","updated_at":"2025-06-25T23:19:45.000Z","dependencies_parsed_at":"2025-06-26T00:38:13.561Z","dependency_job_id":null,"html_url":"https://github.com/datacoves/workshop__snowflake_dynamic_tables","commit_stats":null,"previous_names":["datacoves/workshop__snowflake_dynamic_tables"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/datacoves/workshop__snowflake_dynamic_tables","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datacoves%2Fworkshop__snowflake_dynamic_tables","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datacoves%2Fworkshop__snowflake_dynamic_tables/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datacoves%2Fworkshop__snowflake_dynamic_tables/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datacoves%2Fworkshop__snowflake_dynamic_tables/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datacoves","download_url":"https://codeload.github.com/datacoves/workshop__snowflake_dynamic_tables/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datacoves%2Fworkshop__snowflake_dynamic_tables/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271746300,"owners_count":24813556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-12T01:30:45.373Z","updated_at":"2025-08-23T09:09:16.005Z","avatar_url":"https://github.com/datacoves.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Snowflake Dynamic Tables with dbt Workshop\n\n\u003e **Hosted by [Datacoves](https://datacoves.com)** - Enterprise DataOps platform with managed dbt Core and Airflow for data transformation and orchestration.\n\nThis repository contains the materials and code examples from the **Snowflake Dynamic Tables with dbt** workshop held on **June 25, 2025**.\n\n## Overview\n\nThis workshop demonstrates how to implement and work with Snowflake Dynamic Tables using dbt, showcasing the differences between standard tables and dynamic tables, along with advanced monitoring and alerting capabilities.\n\n## Project Structure\n\n### 📁 `/load`\nContains data loading scripts using dlt (data load tool):\n- **`dlt/loans_data.py`** - Python script that loads personal loans data from a public S3 CSV file into Snowflake using dlt\n- **`dlt/utils/datacoves_utils.py`** - Utility functions for dlt pipeline configuration\n\n### 📁 `/transform`\nContains the dbt project for data transformation:\n- **dbt Version**: Requires dbt \u003e= 1.8.0\n- **Project Name**: `balboa`\n- **Profile**: Uses `default` profile for Snowflake connection\n\n#### Key dbt Components:\n\n**Models Structure:**\n- **`L1_inlets/loans/`** - Staging models for loan data\n  - `stg_personal_loans.sql` - Staging table for personal loans data, target lag = downstream\n- **`L3_coves/loan_analytics/`** - Analytics models demonstrating dynamic vs standard tables\n  - `loans_by_state__dynamic.sql` - Dynamic table implementation with 1-minute target lag\n  - `loans_by_state__standard.sql` - Standard table implementation for comparison\n- **`L1_inlets/observe/`** - Observability models\n  - `stg_test_failures.sql` - View for tracking dbt test failures\n\n**Seeds:**\n- **`state_codes.csv`** - Reference data mapping state codes to state names\n\n**Macros:**\n- **`create_test_failure_view.sql`** - Creates a view that aggregates dbt test results for monitoring dynamic table data quality. This macro runs automatically after each dbt execution when `persist_tests` variable is set to true.\n- **`generate_schema_name.sql`** - Custom schema naming logic that uses custom schema names in production but defaults to user schema in development.\n\n### 📁 `/visualize`\nContains Streamlit application for real-time visualization:\n- **`streamlit/loans-example/loans.py`** - Streamlit app that displays side-by-side comparison of standard vs dynamic tables with auto-refresh every 5 seconds\n- **`streamlit/loans-example/database_connection.py`** - Snowflake connection utilities\n- **`streamlit/loans-example/environment.yml`** - Conda environment specification\n\n### 📁 Root Files\n- **`script.sql`** - Basic workshop script demonstrating:\n  - Enabling change tracking on source tables\n  - Data manipulation and dbt builds\n  - Monitoring dynamic table refresh behavior\n- **`script2_advanced.sql`** - Advanced features including:\n  - Creating streams on dynamic tables\n  - Automated error capture using Snowflake tasks\n  - Email alerting when data quality issues are detected\n\n## Key Features Demonstrated\n\n### 1. Dynamic Tables vs Standard Tables\n- **Dynamic Table**: Automatically refreshes based on changes to upstream data with configurable target lag (1 minute in this example)\n- **Standard Table**: Traditional materialized table that requires manual refresh\n- **Warehouse**: Uses dedicated `wh_transforming_dynamic_tables` warehouse for dynamic table refreshes\n\n### 2. Data Quality Monitoring\n- Automated test failure tracking using custom macro\n- Real-time monitoring of data quality issues\n- Integration with Snowflake streams and tasks for proactive alerting\n\n### 3. Advanced Snowflake Features\n- **Change Tracking**: Enabled on source tables to support dynamic table refreshes\n- **Streams**: Track changes in dynamic tables\n- **Tasks**: Automated error capture and processing\n- **Alerts**: Email notifications when data quality issues are detected\n\n### 4. Real-time Visualization\n- Streamlit application showing live comparison between table types\n- Auto-refreshing charts and error monitoring\n- Side-by-side visualization of data differences\n\n## Getting Started\n\n### Prerequisites\n- Snowflake account with appropriate permissions\n- dbt installed (version \u003e= 1.8.0)\n- uv installed for Python dependency management\n\n### Setup Instructions\n\n1. **Configure Snowflake Connection**\n   - Set up your dbt profile for Snowflake connection\n   - Ensure you have permissions to create dynamic tables\n\n2. **Load Data**\n   ```bash\n   # The script is executable with uv run shebang\n   ./load/dlt/loans_data.py\n   ```\n\n3. **Enable Change Tracking**\n   ```sql\n   ALTER TABLE RAW.LOANS.PERSONAL_LOANS SET CHANGE_TRACKING = true;\n   ```\n\n4. **Run dbt Build**\n   ```bash\n   cd transform\n   dbt build --vars '{\"persist_tests\": \"true\", \"tests_model\": \"test_failures\"}'\n   ```\n\n5. **Launch Streamlit App**\n   ```bash\n   # Using the provided shell script\n   ./visualize/streamlit/start_app.sh\n\n   # Or directly with uvx\n   uvx --with \"schedule,snowflake-connector-python,snowflake-snowpark-python\" streamlit run visualize/streamlit/loans-example/loans.py\n   ```\n\n## Workshop Scenarios\n\nThe workshop walks through several scenarios:\n\n1. **Basic Setup**: Loading data and creating both standard and dynamic tables\n2. **Data Changes**: Demonstrating how dynamic tables automatically refresh while standard tables remain static\n3. **Error Detection**: Introducing data quality issues and showing automated detection\n4. **Advanced Monitoring**: Setting up streams, tasks, and alerts for production-ready monitoring\n\n## Configuration Notes\n\n- **Target Lag**: Dynamic tables are configured with 1-minute target lag for demonstration purposes\n- **Schema Strategy**: Uses custom schema names in production, user schemas in development\n- **Test Storage**: Test failures are stored as views in `DBT_TEST__AUDIT` schema\n- **Time Zone**: Configured for America/Los_Angeles timezone\n\n## Workshop Takeaways\n\n- Dynamic tables provide automatic refresh capabilities based on upstream changes\n- Proper monitoring and alerting are crucial for production dynamic table implementations\n- dbt macros can be used to create sophisticated monitoring solutions\n- Snowflake's native features (streams, tasks, alerts) integrate well with dbt workflows\n\n---\n\n*This repository serves as a reference implementation for the concepts covered in the Snowflake Dynamic Tables with dbt workshop. Feel free to explore the code and adapt it for your own use cases.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatacoves%2Fworkshop__snowflake_dynamic_tables","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatacoves%2Fworkshop__snowflake_dynamic_tables","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatacoves%2Fworkshop__snowflake_dynamic_tables/lists"}