https://github.com/manishms18/sign-language-advance
Contributed to a vision-driven accessibility tool translating sign language into text
https://github.com/manishms18/sign-language-advance
accessibility ai-for-accessibility american-sign-language asl cohens-kappa confusion-matrix data-augmentation deep-learning efficientnet f1-score gru keras multimodal-fusion recurrent-neural-networks resnet50 robustness sign-language-recognition tensorflow transfer-learning
Last synced: 10 months ago
JSON representation
Contributed to a vision-driven accessibility tool translating sign language into text
- Host: GitHub
- URL: https://github.com/manishms18/sign-language-advance
- Owner: Manishms18
- License: mit
- Created: 2025-09-09T20:22:43.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2025-09-26T03:57:22.000Z (10 months ago)
- Last Synced: 2025-09-26T05:45:41.277Z (10 months ago)
- Topics: accessibility, ai-for-accessibility, american-sign-language, asl, cohens-kappa, confusion-matrix, data-augmentation, deep-learning, efficientnet, f1-score, gru, keras, multimodal-fusion, recurrent-neural-networks, resnet50, robustness, sign-language-recognition, tensorflow, transfer-learning
- Language: Python
- Homepage:
- Size: 54.7 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Advancing Sign Language Interpretation with Transfer Learning and Multimodal Features
- Baseline CNN for sign language gesture recognition
- Transfer learning using ResNet-50 and EfficientNet-B0
- Hand landmark extraction via MediaPipe (GRU-based)
- Comprehensive evaluation: accuracy, macro/micro F1, Cohen's kappa, confusion matrix
- Robustness testing (lighting, noise)
The repository includes:
- Scripts for training and evaluation.
- Sample outputs (results.json, plots).
## Quickstart
```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# Train baseline CNN on mini-sample
python -m src.models.train_cnn --config configs/config.yaml
python -m src.eval.evaluate --config configs/config.yaml --weights outputs/baseline_cnn_best.h5
python -m src.eval.robustness --config configs/config.yaml --weights outputs/baseline_cnn_best.h5
# Transfer Learning
python -m src.models.train_transfer --config configs/config.yaml --backbone resnet50
python -m src.models.train_transfer --config configs/config.yaml --backbone efficientnetb0
# Landmark + GRU (stub example)
python -m src.models.train_landmark_gru --config configs/config.yaml
## Demo
### Streamlit (browser)
To run a simple browser-based demo using Streamlit:
```bash
pip install -r requirements.txt
streamlit run demo/streamlit_app.py
```
Upload a hand image (PNG or JPEG) and optionally adjust the weights path. The app displays the predicted letter and confidence as well as per-class probabilities.
### Gradio (browser)
To run a Gradio demo:
```bash
pip install -r requirements.txt
python demo/gradio_app.py
```
This serves a web interface where you can upload an image and see the probability distribution across classes and the top prediction.
### Webcam (local)
To run a live webcam demo (requires a webcam):
```bash
python scripts/webcam_demo.py
```
### Citation
If you are using this work, please cite using below details :
```
@misc{shukla_gupta_2025_advancing_sign_language,
author = {Shukla, Manish and Gupta, Harsh},
title = {Advancing Sign Language Interpretation with Transfer Learning and Multimodal Features},
year = {2025},
month = sep,
note = {Preprint (Version 1), Research Square},
doi = {10.21203/rs.3.rs-7586144/v1},
url = {https://doi.org/10.21203/rs.3.rs-7586144/v1}
}
```
Shukla, M., & Gupta, H. (2025, September 12). Advancing Sign Language Interpretation with Transfer Learning and Multimodal Features (Version 1) [Preprint]. Research Square. https://doi.org/10.21203/rs.3.rs-7586144/v1