{"id":29804633,"url":"https://github.com/jaganov/google_asl_recognition","last_synced_at":"2026-04-14T03:32:24.606Z","repository":{"id":305841858,"uuid":"1016395600","full_name":"jaganov/google_asl_recognition","owner":"jaganov","description":"Real-time American Sign Language (ASL) recognition system using PyTorch and MediaPipe. Recognizes 25 common ASL gestures with 76.05% accuracy, optimized for RTX4070 GPUs. Features live webcam recognition, hybrid TCN+LSTM+Transformer architecture, and comprehensive training pipeline.","archived":false,"fork":false,"pushed_at":"2025-07-22T07:51:14.000Z","size":22012,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-22T09:46:52.661Z","etag":null,"topics":["accessibility","asl-recognition","computer-vision","deep-learning","gesture-recognition","landmark-detection","lstm","mediapipe","opencv","pytorch","real-time","rtx4070","sign-language","temporal-convolutional-networks","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaganov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-09T00:40:49.000Z","updated_at":"2025-07-22T07:51:18.000Z","dependencies_parsed_at":"2025-07-22T09:57:00.862Z","dependency_job_id":null,"html_url":"https://github.com/jaganov/google_asl_recognition","commit_stats":null,"previous_names":["jaganov/google_asl_recognition"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jaganov/google_asl_recognition","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaganov%2Fgoogle_asl_recognition","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaganov%2Fgoogle_asl_recognition/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaganov%2Fgoogle_asl_recognition/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaganov%2Fgoogle_asl_recognition/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaganov","download_url":"https://codeload.github.com/jaganov/google_asl_recognition/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaganov%2Fgoogle_asl_recognition/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31781292,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T02:24:21.117Z","status":"ssl_error","status_checked_at":"2026-04-14T02:24:20.627Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accessibility","asl-recognition","computer-vision","deep-learning","gesture-recognition","landmark-detection","lstm","mediapipe","opencv","pytorch","real-time","rtx4070","sign-language","temporal-convolutional-networks","transformer"],"created_at":"2025-07-28T13:01:15.490Z","updated_at":"2026-04-14T03:32:24.591Z","avatar_url":"https://github.com/jaganov.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🤟 Google ASL Recognition\n\nA comprehensive American Sign Language (ASL) recognition system based on the winning solution from Google's ASL Signs competition, adapted for PyTorch and optimized for RTX4070 GPUs.\n\n## 🎯 Project Overview\n\nThis project implements a state-of-the-art ASL recognition model that can:\n- **Train** on the Google ASL Signs dataset (25 most common gestures)\n- **Capture** real-time gestures using MediaPipe\n- **Recognize** ASL signs from video input with 76.05% accuracy\n- **Compare** captured gestures with dataset samples\n\nThe model architecture combines TCN, LSTM, and Transformers with adaptive regularization, achieving competitive accuracy while being optimized for modern GPUs.\n\n### 📊 Dataset Source\n\nThis project uses a subset of the **Google ASL Signs** dataset from the Kaggle competition:\n- **Competition**: [ASL Signs - American Sign Language Recognition](https://www.kaggle.com/competitions/asl-signs/overview)\n- **Original Dataset**: Contains 250 ASL signs with hand landmark data\n- **Project Dataset**: Extracted 25 most common ASL signs for focused training\n- **Citation**: [Kaggle Competition Citation](https://www.kaggle.com/competitions/asl-signs/overview/citation)\n- **Format**: Parquet files with MediaPipe landmarks (543 points per frame)\n\n### 🎯 Recognized Gestures (25 ASL Signs)\n\n**Greetings \u0026 Courtesy**: hello, please, thankyou, bye  \n**Family**: mom, dad, boy, girl, man, child  \n**Actions**: drink, sleep, go  \n**Emotions**: happy, sad, hungry, thirsty, sick, bad  \n**Colors**: red, blue, green, yellow, black, white  \n\n## 📈 Project Evolution\n\n### Initial Vision vs. Reality\n\nThe project started with an ambitious 7-day roadmap (see `PREPARE.md`) targeting 80-85% accuracy using Vision Transformers and CNN ensembles. However, reality led to a more focused and practical approach:\n\n**Original Plan:**\n- Vision Transformer + CNN ensemble\n- 250 ASL gestures\n- 80-85% target accuracy\n- 7-day intensive development\n\n**Actual Implementation:**\n- Hybrid TCN + LSTM + Transformer architecture\n- 25 most common ASL gestures\n- 76.05% achieved accuracy\n- Focused, production-ready solution\n\nThis evolution resulted in a more practical, maintainable system optimized for real-world deployment rather than competition-level performance.\n\n## 🏗️ Architecture\n\n### Model Components\n- **TCN Blocks**: 3 layers with dilations (1,2,4) for local temporal patterns\n- **Bidirectional LSTM**: 2 layers for long-term dependencies\n- **Temporal Attention**: 8 heads for focusing on important frames\n- **Conv1D + Transformer**: Hybrid processing for final classification\n- **Adaptive Regularization**: Dropout, LateDropout, AWP for overfitting prevention\n\n### Model Structure\n```\nInput (seq_len, landmarks, 3)\n    ↓\nPreprocessing (normalization, motion features)\n    ↓\nStem (Linear + BatchNorm + AdaptiveDropout)\n    ↓\nTCN Blocks (3 layers, dilations 1,2,4)\n    ↓\nBidirectional LSTM (2 layers)\n    ↓\nTemporal Attention (8 heads)\n    ↓\nConv1D Block × 3 + Transformer Block\n    ↓\nMulti-scale Pooling (avg + max + attention)\n    ↓\nClassifier (25 classes)\n```\n\n## 📊 Results \u0026 Visualizations\n\n### Training Progress\n![Training History](docs/pictures/training_history.png)\n\nThe training history shows stable convergence with the model achieving 76.05% validation accuracy. The loss curves demonstrate effective regularization preventing overfitting while maintaining learning capacity.\n\n### Live Recognition Demo\n![Live Recognition](docs/pictures/happy_20250721_134139_002.jpg)\n\nReal-time ASL recognition in action, demonstrating the system's ability to capture and classify gestures with MediaPipe landmarks and visual feedback.\n\n## 🚀 Quick Start\n\n### Prerequisites\n- Python 3.10+\n- CUDA-compatible GPU (RTX4070 recommended)\n- Webcam for gesture capture\n\n### Installation\n\n1. **Clone the repository**\n```bash\ngit clone \u003crepository-url\u003e\ncd google_asl_recognition\n```\n\n2. **Create and activate virtual environment**\n```bash\n# Create virtual environment\npython -m venv .venv\n\n# Activate on Windows\n.venv\\Scripts\\activate\n\n# Activate on Linux/Mac\nsource .venv/bin/activate\n```\n\n3. **Install PyTorch with CUDA 12.1**\n```bash\n# Install PyTorch with CUDA support\npip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121\n```\n\n4. **Install all dependencies**\n```bash\n# Install all required packages\npip install -r requirements.txt\n```\n\n## 📁 Project Structure\n\n```\ngoogle_asl_recognition/\n├── data/                          # Dataset storage\n│   └── google_asl_signs/         # Original dataset\n├── manual/                        # Main project directory\n│   ├── dataset25/                # Processed dataset (25 signs)\n│   ├── dataset25_split/          # Train/test split\n│   ├── models/                   # Trained models and manifests\n│   ├── utils/                    # Utility scripts\n│   ├── step1_extract_words.py    # Data extraction\n│   ├── step1.2_split_train_test.py # Dataset splitting\n│   ├── step2_prepare_dataset.py  # Dataset preparation\n│   ├── step3_prepare_train.py    # Training script\n│   ├── step4_capture_mediapipe.py # Gesture capture\n│   └── step5_live_recognition.py # Live recognition\n├── docs/                         # Documentation\n│   ├── README.md                 # Documentation overview\n│   ├── training.md               # Training guide\n│   ├── data-preparation.md       # Data preparation guide\n│   ├── live-recognition.md       # Live recognition guide\n│   ├── manifest-system.md        # Manifest system guide\n│   └── pictures/                 # Screenshots and images\n├── demo/                         # Demo scripts\n├── PREPARE.md                    # Original project roadmap\n└── requirements.txt              # Dependencies\n```\n\n## 🎮 Usage\n\n### 1. Data Preparation\n\n```bash\ncd manual\n\n# Extract and prepare the dataset\npython step1_extract_words.py\npython step1.2_split_train_test.py\npython step2_prepare_dataset.py\n```\n\n### 2. Model Training\n\n```bash\n# Start training\npython step3_prepare_train.py\n```\n\n**Training Configuration for RTX4070:**\n- **Epochs**: 300 (full training) or 50 (testing)\n- **Batch Size**: 32 (optimized for RTX4070)\n- **Expected Time**: ~1.75 hours for full training\n- **Expected Accuracy**: ~76.05% validation score\n\n### 3. Live Recognition\n\n```bash\n# Test camera\npython test_camera.py\n\n# Run live ASL recognition\npython step5_live_recognition.py\n```\n\n## 🔧 RTX4070 Optimizations\n\nThe project includes specific optimizations for RTX4070:\n- **TF32**: Enabled for Ampere architecture acceleration\n- **Mixed Precision**: Automatic FP16 usage\n- **cuDNN Benchmark**: Optimized convolution algorithms\n- **Memory Optimization**: Automatic batch size tuning\n- **Adaptive Dropout**: Gradual regularization activation\n\n## 📊 Latest Results\n\n### Model Performance\n- **Model**: `asl_model_v20250720_080209`\n- **Architecture**: Enhanced TCN + LSTM + Transformer v2\n- **Validation Accuracy**: 76.05%\n- **Parameters**: 1.97M\n- **Training Time**: ~1.75 hours\n- **Classes**: 25 ASL gestures\n\n### Training Configuration\n- **Loss**: CrossEntropyLoss with label smoothing (0.05)\n- **Optimizer**: AdamW with weight decay (0.005)\n- **Scheduler**: CosineAnnealingWarmRestarts\n- **Regularization**: Adaptive Dropout, TCN Dropout, Attention Dropout\n\n## 🎥 Live Recognition Features\n\n### MediaPipe Integration\n- **Face Landmarks**: 468 points\n- **Pose Landmarks**: 33 points\n- **Hand Landmarks**: 21 points per hand\n- **Total**: 543 landmarks per frame\n\n### Recognition Controls\n- **'q'**: Stop recognition\n- **'s'**: Save current screenshot\n- **'r'**: Reset frame buffer\n- **'h'**: Show/hide help\n- **Real-time preview**: Visual feedback during recognition\n\n## 📈 Model Versioning\n\nThe project includes a comprehensive manifest system for tracking model versions:\n- **Date-based naming**: `asl_model_vYYYYMMDD_HHMMSS.pth`\n- **Training manifests**: Detailed training history and parameters\n- **Final manifests**: Best model selection with performance metrics\n- **Complete tracking**: All experiments and improvements documented\n\n## 📚 Documentation\n\nComprehensive documentation is available in the `docs/` directory:\n\n- **[Documentation Overview](docs/README.md)** - Complete documentation guide\n- **[Training Guide](docs/training.md)** - How to train the model\n- **[Data Preparation](docs/data-preparation.md)** - Dataset preparation process\n- **[Live Recognition](docs/live-recognition.md)** - Real-time gesture recognition\n- **[Manifest System](docs/manifest-system.md)** - Model versioning and tracking\n- **[RTX4070 Optimizations](docs/rtx4070-optimizations.md)** - GPU-specific optimizations\n- **[Model Architecture](docs/architecture.md)** - Detailed model architecture\n- **[Model Results](docs/model-results.md)** - Performance analysis and results\n- **[Quick Start](docs/quickstart.md)** - Quick start guide\n- **[Installation](docs/installation.md)** - Detailed installation instructions\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests if applicable\n5. Submit a pull request\n\n## 📄 License\n\nThis project is based on the winning solution from Google's ASL Signs competition. Please refer to the original competition terms and conditions.\n\n## 🙏 Acknowledgments\n\n- **Kaggle Competition**: [ASL Signs - American Sign Language Recognition](https://www.kaggle.com/competitions/asl-signs/overview) organizers and participants\n- **Dataset**: Google ASL Signs dataset provided by the competition\n- **MediaPipe team** for hand tracking and landmark extraction\n- **PyTorch community** for the deep learning framework\n- **Original competition winners** for the model architecture inspiration\n\n## 📞 Support\n\nFor issues and questions:\n1. Check the documentation in the `docs/` directory\n2. Review existing issues\n3. Create a new issue with detailed information\n\n---\n\n**Note**: This project requires a CUDA-compatible GPU for optimal performance. CPU-only mode is available but significantly slower.\n\n**Ready to recognize ASL gestures! 🤟**","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaganov%2Fgoogle_asl_recognition","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaganov%2Fgoogle_asl_recognition","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaganov%2Fgoogle_asl_recognition/lists"}