{"id":49540437,"url":"https://github.com/subhamay-bhattacharyya/aws-snowflake-e2e-project","last_synced_at":"2026-05-02T15:05:32.712Z","repository":{"id":335306995,"uuid":"1144623359","full_name":"subhamay-bhattacharyya/aws-snowflake-e2e-project","owner":"subhamay-bhattacharyya","description":"End-to-end Snowflake data engineering project: layered stage/raw/curated design, automated ingestion + DAG, dynamic tables, Snowpark Python, GitHub Actions, and in-Snowflake visualization (Streamlit).","archived":false,"fork":false,"pushed_at":"2026-04-10T02:44:16.000Z","size":186,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-10T04:27:03.733Z","etag":null,"topics":["data-engineering","data-ingestion","data-pipeline","elt","layered-architecture","medallion-architecture","snowflake","streamlit"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/subhamay-bhattacharyya.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-28T21:31:22.000Z","updated_at":"2026-04-10T02:44:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/subhamay-bhattacharyya/aws-snowflake-e2e-project","commit_stats":null,"previous_names":["subhamay-bhattacharyya/aws-snowflake-e2e-project"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/subhamay-bhattacharyya/aws-snowflake-e2e-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subhamay-bhattacharyya%2Faws-snowflake-e2e-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subhamay-bhattacharyya%2Faws-snowflake-e2e-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subhamay-bhattacharyya%2Faws-snowflake-e2e-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subhamay-bhattacharyya%2Faws-snowflake-e2e-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/subhamay-bhattacharyya","download_url":"https://codeload.github.com/subhamay-bhattacharyya/aws-snowflake-e2e-project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subhamay-bhattacharyya%2Faws-snowflake-e2e-project/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32538279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T12:25:33.646Z","status":"ssl_error","status_checked_at":"2026-05-02T12:24:51.733Z","response_time":132,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-ingestion","data-pipeline","elt","layered-architecture","medallion-architecture","snowflake","streamlit"],"created_at":"2026-05-02T15:05:23.805Z","updated_at":"2026-05-02T15:05:32.703Z","avatar_url":"https://github.com/subhamay-bhattacharyya.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Snowflake Lakehouse\n\n![Built with Kiro](https://img.shields.io/badge/Built_with-Kiro-8845f4?logo=robot\u0026logoColor=white)\u0026nbsp;![Commit Activity](https://img.shields.io/github/commit-activity/t/subhamay-bhattacharyya/aws-snowflake-e2e-project)\u0026nbsp;![Last Commit](https://img.shields.io/github/last-commit/subhamay-bhattacharyya/aws-snowflake-e2e-project)\u0026nbsp;![Release Date](https://img.shields.io/github/release-date/subhamay-bhattacharyya/aws-snowflake-e2e-project)\u0026nbsp;![Repo Size](https://img.shields.io/github/repo-size/subhamay-bhattacharyya/aws-snowflake-e2e-project)\u0026nbsp;![File Count](https://img.shields.io/github/directory-file-count/subhamay-bhattacharyya/aws-snowflake-e2e-project)\u0026nbsp;![Issues](https://img.shields.io/github/issues/subhamay-bhattacharyya/aws-snowflake-e2e-project)\u0026nbsp;![Top Language](https://img.shields.io/github/languages/top/subhamay-bhattacharyya/aws-snowflake-e2e-project)\u0026nbsp;![Custom Endpoint](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/bsubhamay/afb632c4d78d83fbc1e6b4486d5720a4/raw/aws-snowflake-e2e-project.json?)\n\nA Snowflake Lakehouse implementation with AWS and Infrastructure as Code (Terraform), automated deployment using GitHub Actions.\n\n## Overview\n\nThis repository tracks the build of an end-to-end Snowflake data engineering solution—from source data analysis and ingestion design to layered stage/raw/curated modeling, automation with DAG + GitHub Actions, dynamic tables, and Streamlit dashboards—using Snowpark Python and marketplace datasets.\n\nThe project demonstrates a complete data lakehouse implementation with:\n\n- **Infrastructure as Code**: Terraform configurations for AWS (S3, IAM) and Snowflake resources\n- **Layered Data Architecture**: Stage → Raw → Curated data modeling pattern\n- **Automated Ingestion**: Snowpipe for real-time data loading from S3\n- **Data Transformation**: Snowpark Python for ETL/ELT processing\n- **Orchestration**: DAG-based workflows with GitHub Actions CI/CD\n- **Dynamic Tables**: Incremental data processing with automatic refresh\n- **Visualization**: Streamlit dashboards for data exploration\n- **Marketplace Integration**: Leveraging Snowflake marketplace datasets\n\n## Repository Structure\n\n```\n.\n├── infra/                          # Infrastructure as Code (Terraform)\n│   ├── platform/tf/                # Root orchestration module (entry point)\n│   │   ├── main.tf                 # Orchestrates AWS + Snowflake modules\n│   │   ├── locals.tf               # Configuration parsing from JSON\n│   │   ├── variables.tf            # Input variables\n│   │   ├── outputs.tf              # Module outputs\n│   │   ├── versions.tf             # Provider version constraints\n│   │   ├── backend.tf              # Terraform Cloud backend\n│   │   ├── providers-aws.tf        # AWS provider configuration\n│   │   └── providers-snowflake.tf  # Snowflake provider configuration\n│   ├── aws/tf/                     # AWS child module\n│   │   ├── main.tf                 # S3 bucket + IAM role orchestration\n│   │   ├── modules/                # Nested modules (s3, iam, iam_role_final, s3_event_notification)\n│   │   └── templates/              # Bucket policy templates\n│   └── snowflake/tf/               # Snowflake child module\n│       ├── main.tf                 # Warehouses, databases, stages, pipes\n│       └── modules/                # Nested modules (warehouse, database, stage, etc.)\n├── input-jsons/                    # Configuration files\n│   ├── aws/config.json             # AWS resource configuration\n│   └── snowflake/config.json       # Snowflake resource configuration\n├── snowflake-ddl/                  # Snowflake DDL Scripts\n│   ├── 00_account/                 # Account-level objects (resource monitors, network policies)\n│   ├── 01_security/                # Roles, users, grants\n│   ├── 02_warehouses/              # Virtual warehouses\n│   ├── 03_databases/               # Database definitions\n│   ├── 04_storage/                 # Storage integrations \u0026 external stages\n│   ├── 05_schemas/                 # Schema-level objects (tables, views)\n│   ├── 06_pipes/                   # Snowpipe definitions\n│   ├── 07_tasks/                   # Task definitions\n│   ├── 08_functions/               # UDFs and UDTFs\n│   ├── 09_procedures/              # Stored procedures\n│   ├── environments/               # Environment configs (dev/staging/prod)\n│   └── scripts/                    # Utility scripts (deploy, rollback, validate)\n├── .github/\n│   └── workflows/                  # GitHub Actions workflows (ci.yaml, etc.)\n├── .devcontainer/                  # Dev container configuration\n├── cliff.toml                      # git-cliff changelog configuration\n└── utils/                          # Utility scripts\n```\n\n## Architecture\n\nThis project uses a **4-phase architecture**:\n\n### Phase 1: AWS Resources\n- S3 Bucket for data storage\n- IAM Role with placeholder trust policy\n\n### Phase 2: Snowflake Resources\n- Warehouses, Databases, Schemas\n- File Formats, Storage Integration\n- External Stages, Tables, Snowpipes\n\n### Phase 3: AWS Trust Policy Update\n- Update IAM Role trust policy with Snowflake's IAM User ARN and External ID\n\n### Phase 4: S3 Event Notifications\n- Configure S3 bucket notifications to trigger Snowpipe auto-ingest\n\n## Getting Started\n\n### Prerequisites\n\n- **Terraform** \u003e= 1.0\n- **Snowflake Account** with appropriate permissions\n- **AWS Account** with IAM permissions\n- **GitHub Repository** with Actions enabled\n\n#### One-Time Snowflake Setup\n\nBefore using this action, run the following SQL script in Snowflake to create the utility infrastructure (only needs to be run once):\n\n**Step 1: Create Utility Infrastructure**\n\n```sql\n-- =========================================================\n-- Snowflake Utility Setup for DDL Migrations\n-- =========================================================\n-- This script creates:\n--   1. A dedicated warehouse for CI/CD metadata operations\n--   2. Utility database (UTIL_DB)\n--   3. Utility schema (UTIL_SCHEMA)\n--   4. DDL migration history table\n--\n-- Safe to re-run (idempotent)\n-- =========================================================\n\n-- -----------------------------------------------------------\n-- 1. Create and use a dedicated warehouse\n-- -----------------------------------------------------------\nCREATE WAREHOUSE IF NOT EXISTS UTIL_WH\n  WAREHOUSE_SIZE = 'XSMALL'\n  WAREHOUSE_TYPE = 'STANDARD'\n  AUTO_SUSPEND = 60\n  AUTO_RESUME = TRUE\n  INITIALLY_SUSPENDED = TRUE\n  COMMENT = 'Warehouse for CI/CD utility operations and DDL migration tracking';\n\nUSE WAREHOUSE UTIL_WH;\n\n-- -----------------------------------------------------------\n-- 2. Create utility database and schema\n-- -----------------------------------------------------------\nCREATE DATABASE IF NOT EXISTS UTIL_DB\n  COMMENT = 'Utility database for CI/CD metadata and migration tracking';\n\nCREATE SCHEMA IF NOT EXISTS UTIL_DB.UTIL_SCHEMA\n  COMMENT = 'Utility schema for migration and operational tables';\n\n-- -----------------------------------------------------------\n-- 3. Create DDL migration history table\n-- -----------------------------------------------------------\nCREATE TABLE IF NOT EXISTS UTIL_DB.UTIL_SCHEMA.DDL_MIGRATION_HISTORY (\n  script_name    STRING        NOT NULL,\n  script_path    STRING        NOT NULL,\n  checksum       STRING        NOT NULL,\n  applied_at     TIMESTAMP_LTZ NOT NULL DEFAULT CURRENT_TIMESTAMP(),\n  status         STRING        NOT NULL,\n  error_message  STRING,\n  run_id         STRING,\n  actor          STRING\n) COMMENT = 'Tracks executed Snowflake DDL migration scripts for CI/CD pipelines';\n\n-- -----------------------------------------------------------\n-- 4. (Optional) Verify creation\n-- -----------------------------------------------------------\nSELECT\n  'UTIL_DB.UTIL_SCHEMA.DDL_MIGRATION_HISTORY created successfully' AS status,\n  CURRENT_TIMESTAMP() AS verified_at;\n```\n\n**Step 2: Grant MANAGE GRANTS Privilege to SYSADMIN**\n\nSYSADMIN needs the MANAGE GRANTS privilege to grant permissions to other roles like PUBLIC. Run this as ACCOUNTADMIN:\n\n```sql\nUSE ROLE ACCOUNTADMIN;\n\n-- Grant MANAGE GRANTS privilege to SYSADMIN\n-- This allows SYSADMIN to grant privileges on objects it owns\nGRANT MANAGE GRANTS ON ACCOUNT TO ROLE SYSADMIN;\n\n-- Verify the grant\nSHOW GRANTS TO ROLE SYSADMIN;\n```\n\n**Note:** With this setup, SYSADMIN can both create objects and manage their permissions, simplifying the deployment process.\n\n**Note:** If you want to use a different database/schema/table name, you can customize it using the `migrations_table` input parameter in the GitHub Actions workflow.\n\n### 1. Create Dedicated Service Account\n\nFor security best practices, create a dedicated service account for GitHub Actions instead of using your personal account.\n\n#### Step 1: Generate Key Pair\n\nOn your local machine, generate an RSA key pair:\n\n**Option A: Without Passphrase (Recommended for CI/CD)**\n```bash\n# Generate unencrypted PKCS8 private key (no passphrase)\nopenssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out snowflake_key.p8 -nocrypt\n\n# Generate public key\nopenssl rsa -in snowflake_key.p8 -pubout -out snowflake_key.pub\n```\n\n**Option B: With Passphrase (For enhanced security)**\n```bash\n# Generate encrypted PKCS8 private key (with passphrase)\nopenssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out snowflake_key.p8 -v2 aes-256-cbc\n\n# Generate public key\nopenssl rsa -in snowflake_key.p8 -pubout -out snowflake_key.pub\n```\n\n**Extract public key value** (for both options):\n```bash\n# Remove header/footer and newlines for Snowflake\ngrep -v \"BEGIN PUBLIC\" snowflake_key.pub | grep -v \"END PUBLIC\" | tr -d '\\n'\n```\n\n**Save the output** - you'll need it for the next step.\n\n**Note:** If using a passphrase, you'll need to provide `SNOWFLAKE_PRIVATE_KEY_PASSPHRASE` as an additional secret.\n\n#### Step 2: Create Service Account in Snowflake\n\nRun this SQL in Snowflake (replace `YOUR_PUBLIC_KEY_HERE` with the output from Step 1):\n\n```sql\n-- =========================================================\n-- Create Service Account for GitHub Actions\n-- =========================================================\n\n-- Create dedicated service account\nCREATE USER IF NOT EXISTS GH_ACTIONS_USER\n  RSA_PUBLIC_KEY = 'YOUR_PUBLIC_KEY_HERE'\n  DEFAULT_ROLE = SYSADMIN\n  DEFAULT_WAREHOUSE = COMPUTE_WH\n  MUST_CHANGE_PASSWORD = FALSE\n  COMMENT = 'Service account for GitHub Actions CI/CD deployments';\n\n-- Grant SYSADMIN role (for DDL and grant operations)\nGRANT ROLE SYSADMIN TO USER GH_ACTIONS_USER;\n\n-- Grant usage on warehouses\nGRANT USAGE ON WAREHOUSE UTIL_WH TO ROLE SYSADMIN;\nGRANT USAGE ON WAREHOUSE COMPUTE_WH TO ROLE SYSADMIN;\n\n-- Grant usage on the utility database\nGRANT USAGE ON DATABASE UTIL_DB TO ROLE SYSADMIN;\nGRANT USAGE ON SCHEMA UTIL_DB.UTIL_SCHEMA TO ROLE SYSADMIN;\n\n-- Grant create privileges for the migration table\nGRANT CREATE TABLE ON SCHEMA UTIL_DB.UTIL_SCHEMA TO ROLE SYSADMIN;\n\n-- Grant all privileges on the migration table (if it already exists)\nGRANT ALL PRIVILEGES ON TABLE UTIL_DB.UTIL_SCHEMA.DDL_MIGRATION_HISTORY TO ROLE SYSADMIN;\n\n-- If the user needs to create the database/schema (first run)\nGRANT CREATE DATABASE ON ACCOUNT TO ROLE SYSADMIN;\n\n-- Verify the user's role\nDESC USER GH_ACTIONS_USER;\n\n-- See what roles the user has\nSHOW GRANTS TO USER GH_ACTIONS_USER;\n\n-- See what the SYSADMIN role can do\nSHOW GRANTS TO ROLE SYSADMIN;\n\n```\n\n**Security Notes:**\n- ✅ Use `SYSADMIN` role for all DDL and grant operations\n- ✅ Grant `MANAGE GRANTS` privilege to SYSADMIN for permission management\n- ✅ Key-pair authentication is more secure than passwords\n- ✅ Service accounts provide better audit trails\n- ✅ Never commit private keys to the repository\n\n### 2. Configure GitHub Secrets and Variables\n\nSet up GitHub Actions authentication. Navigate to **Settings → Secrets and variables → Actions**.\n\n#### Required Repository Variables\n\n| Variable Name | Description | Example |\n|---------------|-------------|---------|\n| `SNOWFLAKE_ORGANIZATION_NAME` | Snowflake organization name | `AGXUOKJ` |\n| `SNOWFLAKE_ACCOUNT_NAME` | Snowflake account name | `JKC15404` |\n| `SNOWFLAKE_USER` | Service account username | `GH_ACTIONS_USER` |\n| `SNOWFLAKE_ROLE` | Snowflake role for deployments | `SYSADMIN` |\n| `AWS_REGION` | AWS region for resources | `us-east-1` |\n| `TF_LINT_VER` | TFLint version (optional) | `v0.50.0` |\n\n#### Required Repository Secrets\n\n| Secret Name | Description |\n|-------------|-------------|\n| `SNOWFLAKE_PRIVATE_KEY` | Content of `snowflake_key.p8` file (including `-----BEGIN/END PRIVATE KEY-----` headers) |\n| `TF_TOKEN_APP_TERRAFORM_IO` | Terraform Cloud API token for remote backend |\n| `AWS_OIDC_ROLE_ARN` | AWS IAM role ARN for OIDC authentication (e.g., `arn:aws:iam::123456789012:role/github-actions-role`) |\n\n#### How to Get These Values\n\n**Snowflake Variables:**\n1. Log into Snowflake\n2. Organization name: Found in your account URL (`https://\u003corg\u003e-\u003caccount\u003e.snowflakecomputing.com`)\n3. Account name: Same as above\n4. User/Role: Created in the service account setup (Step 1)\n\n**Snowflake Private Key:**\n1. Generated in Step 1 (`snowflake_key.p8`)\n2. Copy the entire file content including headers\n\n**Terraform Cloud Token:**\n1. Go to [Terraform Cloud](https://app.terraform.io)\n2. Navigate to **User Settings → Tokens**\n3. Create a new API token\n\n**AWS OIDC Role ARN:**\n1. Set up OIDC in AWS (see [AWS OIDC Setup](#3-aws-oidc-setup-optional-but-recommended))\n2. Copy the IAM role ARN\n\n### 2a. Configure Codespaces Secrets (For Terraform Development)\n\nIf you're running Terraform from GitHub Codespaces, you need to configure Codespaces secrets for authentication.\n\n**Quick setup:**\n\nNavigate to: **Settings → Secrets and variables → Codespaces**\n\nAdd these secrets:\n\n**Snowflake Authentication:**\n| Secret Name | Description |\n|-------------|-------------|\n| `TF_VAR_snowflake_organization_name` | Snowflake organization name |\n| `TF_VAR_snowflake_account_name` | Snowflake account name |\n| `TF_VAR_snowflake_user` | Snowflake username |\n| `TF_VAR_snowflake_private_key` | Content of `snowflake_key.p8` |\n| `TF_VAR_snowflake_role` | Set to `SYSADMIN` |\n\n**AWS Authentication:**\n| Secret Name | Description |\n|-------------|-------------|\n| `AWS_ACCESS_KEY_ID` | From AWS IAM |\n| `AWS_SECRET_ACCESS_KEY` | From AWS IAM |\n| `AWS_DEFAULT_REGION` | e.g., `us-east-1` |\n\n**Note:** GitHub Actions secrets and Codespaces secrets are stored separately. You need to configure both, but you can use the same values.\n\n### 3. AWS OIDC Setup (Optional but Recommended)\n\nFor secure GitHub Actions authentication with AWS without long-lived credentials, set up OIDC (OpenID Connect). This eliminates the need to store AWS access keys in GitHub Secrets.\n\n**See detailed setup instructions:** [infra/aws/README.md](infra/aws/README.md)\n\n**Benefits:**\n- ✅ No AWS access keys stored in GitHub Secrets\n- ✅ Short-lived tokens that expire automatically\n- ✅ Improved security posture\n- ✅ Recommended by AWS and GitHub\n\n## Snowflake Object Organization\n\nScripts are organized by execution order:\n\n1. **00_account**: Resource monitors, network policies\n2. **01_security**: Roles, users, grants\n3. **02_warehouses**: Virtual warehouses\n4. **03_databases**: Database creation\n5. **04_storage**: Storage integrations and external stages\n6. **05_schemas**: Tables, views, streams\n7. **06_pipes**: Snowpipe for automated ingestion\n8. **07_tasks**: Scheduled tasks\n9. **08_functions**: User-defined functions\n10. **09_procedures**: Stored procedures\n\n## Sample Implementation\n\nThe repository includes sample implementations:\n\n- **Warehouse**: `COMPUTE_WH` (small, auto-suspend)\n- **Database**: `RAW_DB` with sales, marketing, finance schemas\n- **Tables**: \n  - `customer_orders` - Order transactions\n  - `customer_master` - Customer data\n  - `product_catalog` - Product information\n\n## GitHub Actions Workflow\n\nThe deployment workflow (`snowflake-deploy.yaml`) automatically:\n\n- Discovers all SQL files in the repository\n- Deploys them in dependency order\n- Runs files in parallel within each stage\n- Uses the reusable action: `subhamay-bhattacharyya-gha/snowflake-run-ddl-action`\n\n**Triggers**:\n- Push to `main` or `develop` branches (when `snowflake/**` files change)\n- Pull requests to `main` or `develop`\n- Manual workflow dispatch\n\n## Best Practices\n\n### Migration Tracking\n\nBy default, the action tracks which scripts have been applied using a migrations table. This enables:\n\n- **Idempotent execution**: Scripts are only run once (based on path + checksum)\n- **Change detection**: If a script's content changes, it will be re-run\n- **Audit trail**: Complete history of what was applied, when, and by whom\n\n#### Migration Table Schema\n\nThe default table `UTIL_DB.UTIL_SCHEMA.DDL_MIGRATION_HISTORY` contains:\n\n- `script_name` - Filename of the script\n- `script_path` - Full path to the script\n- `checksum` - SHA-256 hash of the script content\n- `applied_at` - Timestamp when applied\n- `status` - SUCCESS or FAILED\n- `error_message` - Error details if failed\n- `run_id` - GitHub Actions run ID\n- `actor` - GitHub user who triggered the run\n\n#### Baseline Mode\n\nUse baseline mode to mark existing scripts as applied without executing them. This is useful when:\n\n- Adopting this action in an existing environment\n- Scripts have already been manually applied\n- You want to start tracking from a known state\n\nTo enable baseline mode in the workflow:\n\n```yaml\n- name: Deploy with baseline\n  uses: subhamay-bhattacharyya-gha/snowflake-run-ddl-action@v1\n  with:\n    baseline: true\n    # ... other parameters\n```\n\n#### Disabling Migration Tracking\n\nTo run scripts without tracking (not recommended for production):\n\n```yaml\n- name: Deploy without tracking\n  uses: subhamay-bhattacharyya-gha/snowflake-run-ddl-action@v1\n  with:\n    track_migrations: false\n    # ... other parameters\n```\n\n### SQL Scripts\n- Use `CREATE OR REPLACE` or `CREATE IF NOT EXISTS` for idempotency\n- Add meaningful comments to all objects\n- Number files for execution order (01_, 02_, etc.)\n- Test in dev before promoting to staging/prod\n\n### Security\n- Never commit credentials or private keys\n- Use service accounts for automation\n- Implement least privilege access\n- Rotate keys regularly\n\n### Infrastructure\n- Use remote state storage for Terraform\n- Enable state locking\n- Tag all resources consistently\n- Use separate environments (dev/staging/prod)\n\n## Documentation\n\n- [Infrastructure Setup](infra/README.md)\n- [Snowflake DDL Scripts](snowflake/README.md)\n- [GitHub Actions Setup](.github/SETUP.md)\n- [Deployment Scripts](snowflake/scripts/README.md)\n\n## Contributing\n\n### Commit Message Convention\n\nThis project uses [Conventional Commits](https://www.conventionalcommits.org/) for automated changelog generation. Please format your commit messages as follows:\n\n```\n\u003ctype\u003e: \u003cdescription\u003e\n\n[optional body]\n```\n\n#### Commit Types\n\n| Type | Description | Example |\n|------|-------------|---------|\n| `feat` | New feature or functionality | `feat: add Azure storage integration support` |\n| `fix` | Bug fix | `fix: correct IAM trust policy condition` |\n| `docs` | Documentation changes | `docs: update README with setup instructions` |\n| `style` | Code style changes (formatting, whitespace) | `stylhttps://agxuokj-jkc15404.snowflakecomputing.com/console/login?activationToken=ver%3A1-hint%3A344489740-ETMsDgAAAZuzoPggABRBRVMvQ0JDL1BLQ1M1UGFkZGluZwEAABAAEBldmu8VANRBCTUgQE%2F7RGgAAABg%2Bi1xEnXGEcqx%2BVMauNO9GmzhCnHTRbWhExX%2Ftsk%2BfZHPKbTjNV61u9%2B%2BjuAiPOgpm%2FYk6MsqkwrbcUM5%2F9LYDHnEoUuMjYN5A7MZDQWpWfx2y6ERIZO3Uq1CuKFbCZbEABTZyEHS0WcfOoqbc3Dw6%2FyEs1zyow%3D%3De: fix indentation in main.tf` |\n| `refactor` | Code refactoring without feature changes | `refactor: simplify locals.tf configuration` |\n| `perf` | Performance improvements | `perf: optimize S3 bucket policy lookup` |\n| `test` | Adding or updating tests | `test: add validation for warehouse config` |\n| `chore` | Maintenance tasks, dependencies | `chore: update Terraform provider versions` |\n| `ci` | CI/CD configuration changes | `ci: add changelog generation to workflow` |\n\n#### Examples\n\n```bash\n# Feature\ngit commit -m \"feat: add Snowpipe auto-ingest configuration\"\n\n# Bug fix\ngit commit -m \"fix: resolve storage integration ARN reference\"\n\n# Documentation\ngit commit -m \"docs: add commit message guidelines to README\"\n\n# With scope (optional)\ngit commit -m \"feat(snowflake): add file format support for Parquet\"\n\n# With breaking change\ngit commit -m \"feat!: change storage integration naming convention\"\n```\n\n#### Why This Matters\n\n- Commits are automatically categorized in the changelog\n- Release notes are generated from commit messages\n- Makes it easier to understand project history\n- Enables semantic versioning automation\n\n### Development Workflow\n\n1. Create a feature branch from `main`\n2. Make your changes\n3. Test in dev environment\n4. Create a pull request with description\n5. Wait for approval and automated deployment\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.\n\n## License\n\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Support\n\nFor issues and questions:\n- Open an issue in this repository\n- Check existing documentation in the `docs/` folder\n- Review [Snowflake documentation](https://docs.snowflake.com/)\n\n## Roadmap\n\n- [ ] Add data quality checks\n- [ ] Implement dbt integration\n- [ ] Add monitoring and alerting\n- [ ] Create CI/CD for data pipelines\n- [ ] Add Streamlit dashboards\n- [ ] Implement dynamic tables","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubhamay-bhattacharyya%2Faws-snowflake-e2e-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsubhamay-bhattacharyya%2Faws-snowflake-e2e-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubhamay-bhattacharyya%2Faws-snowflake-e2e-project/lists"}