{"id":31482446,"url":"https://github.com/snowflake-labs/sfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks","last_synced_at":"2026-07-04T00:31:39.946Z","repository":{"id":258360988,"uuid":"872545089","full_name":"Snowflake-Labs/sfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks","owner":"Snowflake-Labs","description":"Multiclass defect detection on PCB images using distributed PyTorch training","archived":false,"fork":false,"pushed_at":"2026-02-17T23:42:30.000Z","size":1700,"stargazers_count":3,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-02-17T23:57:54.838Z","etag":null,"topics":["snowflake-certified"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Snowflake-Labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-14T16:10:48.000Z","updated_at":"2026-02-17T23:42:36.000Z","dependencies_parsed_at":"2024-11-11T05:22:51.571Z","dependency_job_id":"3935813c-156a-40bf-9781-1e7da6690689","html_url":"https://github.com/Snowflake-Labs/sfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks","commit_stats":null,"previous_names":["snowflake-labs/sfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Snowflake-Labs/sfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fsfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fsfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fsfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fsfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Snowflake-Labs","download_url":"https://codeload.github.com/Snowflake-Labs/sfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fsfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35106552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["snowflake-certified"],"created_at":"2025-10-02T07:50:57.570Z","updated_at":"2026-07-04T00:31:39.934Z","avatar_url":"https://github.com/Snowflake-Labs.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Snowflake - Certified](https://img.shields.io/badge/Snowflake-Certified-2ea44f?style=for-the-badge\u0026logo=snowflake)](https://developers.snowflake.com/solutions/)\n\n# PCB Defect Detection for Electronics Manufacturing: Achieve Zero-Defect Quality with Snowflake\n\n## Overview\n\nThis guide demonstrates end-to-end computer vision for manufacturing quality control using **YOLOv12** object detection on **Snowflake Container Runtime with GPU**. Train a state-of-the-art defect detection model using GPU-accelerated PyTorch on Snowflake's Container Runtime, log models to the Snowflake Model Registry, and visualize results in an interactive Streamlit dashboard—all within Snowflake.\n\n**Use Case:** Detect and classify 6 types of PCB (Printed Circuit Board) defects: open circuits, shorts, mousebite, spur, copper defects, and pin-holes.\n\n**Dataset:** [Deep PCB Dataset](https://github.com/tangsanli5201/DeepPCB) - 1,500 image pairs (template + defect images) with bounding box annotations.\n\n## What You'll Learn\n\n- **GPU-Accelerated Training**: Use `PyTorchDistributor` with Snowflake Container Runtime for GPU-accelerated YOLOv12 training\n- **Model Registry**: Log trained models to Snowflake Model Registry for versioning and SQL-based inference\n- **Computer Vision Pipeline**: Convert datasets to YOLO format, train with data augmentation, and perform batch inference\n- **Streamlit Dashboards**: Build production-ready analytics dashboards with real-time quality metrics\n- **Snowflake ML Ops**: Manage the full ML lifecycle (data → training → inference → visualization) in one platform\n\n## What You'll Build\n\n1. **GPU-Accelerated Training Pipeline**: YOLOv12 training notebook running on Snowflake Container Runtime with GPU compute\n2. **Defect Detection Model**: Fine-tuned YOLOv12 model registered in Snowflake Model Registry\n3. **Data Warehouse**: Structured tables (`DEFECT_LOGS`, `PCB_METADATA`) for tracking defects and board metadata\n4. **Executive Dashboard**: Interactive Streamlit app with:\n   - Real-time quality metrics (yield rate, defect rate, false positive rate)\n   - Defect Pareto analysis and factory line performance heatmaps\n   - Model confidence distribution charts\n   - Recent detection timeline\n\n## Repository Structure\n\n```\n.\n├── scripts/\n│   ├── setup.sql              # Infrastructure setup (database, compute pool, tables)\n│   └── teardown.sql           # Cleanup script\n├── notebooks/\n│   └── 0_pcb_defect_detection_yolo.ipynb  # Training \u0026 inference notebook\n├── streamlit/\n│   ├── Executive_Overview.py  # Main dashboard page\n│   ├── pages/\n│   │   ├── 1_Vision_Lab.py    # Interactive defect analysis\n│   │   └── 2_About.py         # Project documentation\n│   └── utils/\n│       ├── data_loader.py     # Data fetching utilities\n│       └── query_registry.py  # SQL query registry\n├── README.md\n└── LEGAL.md\n```\n\n## Prerequisites\n\n### Snowflake Requirements\n- **Account Edition**: Enterprise or higher\n- **Region**: AWS, Azure, or GCP with GPU support\n- **Privileges**: `ACCOUNTADMIN` role or equivalent with:\n  - `CREATE DATABASE`, `CREATE WAREHOUSE`, `CREATE COMPUTE POOL`\n  - `CREATE INTEGRATION` (for external access and git)\n  - `BIND SERVICE ENDPOINT`\n\n## Step-by-Step Guide\n\n### Step 1: Deploy Infrastructure\n\n**1.1 Create a SQL Worksheet:**\n- Navigate to **Projects \u003e Workspaces** in Snowsight\n- Click the **+** sign (top right)\n- Select **Add File \u003e SQL File**\n\n**1.2 Copy the Setup Script:**\n- Open [`scripts/setup.sql`](https://github.com/Snowflake-Labs/sfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks/blob/main/scripts/setup.sql)\n- Copy the entire contents\n- Paste into your new SQL worksheet\n\n**1.3 Execute:**\n- Click **Run All** button\n\n**What this creates:**\n- Database: `PCB_CV`\n- Warehouse: `PCB_CV_WH` (MEDIUM, auto-suspend 5 min)\n- Role: `PCB_CV_ROLE`\n- Compute Pool: `PCB_CV_COMPUTEPOOL` (3x GPU_NV_M nodes)\n- Git Repository: Connected to this GitHub repo\n- Network Rules: External access for downloading datasets/packages\n- Tables: `PCB_METADATA`, `DEFECT_LOGS`\n- Stage: `MODEL_STAGE` for storing models and images\n- Pretrained Weights: Downloads `yolo12n.pt` to stage\n\n### Step 2: Run the Training Notebook\n\nNavigate to: **Projects \u003e Notebooks \u003e PCB_DEFECT_DETECTION**\n\nClick **\"Run All\"** or execute cells sequentially.\n\n**What the Notebook Does:**\n\nThis notebook provides an end-to-end workflow for training and deploying a YOLOv12 defect detection model on Snowflake's Container Runtime:\n\n- **Data Preparation**: Downloads the Deep PCB dataset (~600 MB) and converts annotations from template-based format to YOLO format with normalized bounding boxes\n- **GPU-Accelerated Training**: Trains YOLOv12 on Snowflake's Container Runtime using PyTorchDistributor for GPU compute, with configurable epochs, batch size, and data augmentation\n- **Model Persistence**: Saves trained weights to Snowflake stages and registers the model in Snowflake Model Registry for versioning and governance\n- **Inference Pipeline**: Runs batch predictions on test images and writes structured results (defect class, confidence, bounding boxes) to the DEFECT_LOGS table for dashboard analytics\n\n**Key Output:**\n- Trained model: `@MODEL_STAGE/runs/detect/train/weights/best.pt`\n- Model Registry: `PCB_CV.PUBLIC.YOLO_PCB_DETECTOR` (version 1)\n- Inference logs: 100+ rows in `DEFECT_LOGS` table\n\n### Step 3: Open the Streamlit Dashboard\n\nNavigate to: **Projects \u003e Streamlit \u003e PCB_DEFECT_DETECTION_APP**\n\n**Dashboard Pages:**\n\n**Executive Overview**\n- Yield Rate, Defect Rate, False Positive Rate KPIs\n- Defect Pareto Analysis (which defects occur most)\n- Factory Line Performance heatmap\n- Recent detections timeline\n\n**Vision Lab**\n- Upload PCB images for real-time inference\n- Interactive bounding box visualization\n- Confidence score analysis\n- Defect type filtering\n\n**About**\n- Project architecture diagram\n- Model performance metrics\n- Dataset information\n\n## Key Features\n\n### GPU Training with PyTorchDistributor\n\nThe notebook uses Snowflake's `PyTorchDistributor` for GPU-accelerated training:\n\n```python\nfrom snowflake.ml.modeling.pytorch import PyTorchDistributor\n\ndistributor = PyTorchDistributor(\n    num_nodes=1,\n    instance_family=\"GPU_NV_M\"\n)\n\n# GPU-accelerated training on Snowflake Container Runtime\nresults = distributor.fit(train_function, args=[model_config])\n```\n\n**Benefits:**\n- **No manual cluster setup**: Snowflake handles GPU infrastructure\n- **Auto-scaling**: Compute pool manages GPU resources automatically\n- **Fault tolerance**: Failed jobs are automatically retried\n- **Cost efficiency**: Pay-per-second billing with auto-suspend\n\n### Model Registry Integration\n\nLog models to Snowflake Model Registry for versioning and governance:\n\n```python\nfrom snowflake.ml.registry import Registry\n\nregistry = Registry(session=session)\n\nregistry.log_model(\n    model=yolo_model,\n    model_name=\"YOLO_PCB_DETECTOR\",\n    version_name=\"v1\",\n    conda_dependencies=[\"ultralytics==8.3.0\"],\n    sample_input_data=sample_image\n)\n```\n\n**Access the model:**\n```python\nmodel = registry.get_model(\"YOLO_PCB_DETECTOR\").version(\"v1\")\npredictions = model.run(new_images)\n```\n\n## Cleanup\n\nTo remove all resources created by this guide, execute [`scripts/teardown.sql`](https://github.com/Snowflake-Labs/sfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks/blob/main/scripts/teardown.sql). This will drop the database, warehouse, compute pool, role, and all integrations.\n\n## Conclusion\n\nYou've built a production-ready computer vision pipeline for manufacturing quality control entirely in Snowflake:\n\n- **GPU-accelerated training** - Trained YOLOv12 on Snowflake's Container Runtime with GPU compute  \n- **Model versioning** - Logged models to Snowflake Model Registry  \n- **Data warehouse** - Structured defect logs for analytics  \n- **Executive dashboard** - Real-time quality metrics in Streamlit  \n- **End-to-end ML Ops** - Data → Training → Inference → Visualization in one platform  \n\n**Next Steps:**\n- Customize the model for your manufacturing use case\n- Integrate with your production line systems via Snowflake APIs\n- Scale to multiple factories using Snowflake's data sharing\n\n## Additional Resources\n\n- [Snowflake ML Overview](https://docs.snowflake.com/en/developer-guide/snowflake-ml/overview)\n- [Snowflake Model Registry Documentation](https://docs.snowflake.com/en/developer-guide/snowpark-ml/model-registry/overview)\n- [Snowflake Container Runtime](https://docs.snowflake.com/en/developer-guide/snowpark-container-services/overview)\n- [YOLOv12 Documentation](https://docs.ultralytics.com/)\n- [Deep PCB Dataset Paper](https://arxiv.org/abs/1902.06197)\n\n---\n\n**Built with:** Snowflake ML, YOLOv12, PyTorch, Streamlit\n\n## License\n\nCopyright (c) Snowflake Inc. All rights reserved.\n\nThe code in this repository is licensed under the Apache 2.0 License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowflake-labs%2Fsfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnowflake-labs%2Fsfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowflake-labs%2Fsfguide-defect-detection-using-distributed-pytorch-with-snowflake-notebooks/lists"}