https://github.com/guptakushal03/signlangsync

SignLangSunc is a web app for real-time sign language recognition using your webcam, with custom class training, MobileNet for feature extraction, and KNN for classification. It supports saving and loading models and provides spoken predictions.
https://github.com/guptakushal03/signlangsync

javascript knnclassifier machinelearning mobilenet signlanguagerecognition tensorflowjs

Last synced: 3 months ago
JSON representation

SignLangSunc is a web app for real-time sign language recognition using your webcam, with custom class training, MobileNet for feature extraction, and KNN for classification. It supports saving and loading models and provides spoken predictions.

• Host: GitHub
• URL: https://github.com/guptakushal03/signlangsync
• Owner: guptakushal03
• Created: 2024-09-08T16:57:34.000Z (8 months ago)
• Default Branch: main
• Last Pushed: 2025-02-14T12:02:28.000Z (3 months ago)
• Last Synced: 2025-02-14T13:23:26.874Z (3 months ago)
• Topics: javascript, knnclassifier, machinelearning, mobilenet, signlanguagerecognition, tensorflowjs
• Language: CSS
• Homepage: https://guptakushal03.github.io/SignLangSync/
• Size: 1.62 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# SignLangSunc - Webcam Image Classification for Sign Language Recognition

SignLangSunc is a web-based project for real-time sign language recognition using your webcam. It utilizes transfer learning with MobileNet and a K-Nearest Neighbors (KNN) classifier to identify custom sign language gestures. Users can add new signs, capture images to train the model, and make predictions using webcam input.

---

## Features

- **Webcam Input**: Capture real-time images from your webcam for training and classification.

- **Custom Sign Language Class Training**: Add new sign language classes (e.g., "Hello", "Thank you"), capture images, and store them in the model.

- **Transfer Learning with MobileNet**: Leverage MobileNet’s pre-trained model for feature extraction to classify signs.

- **KNN Classifier**: Classify captured images using a KNN classifier that is updated with new sign images.

- **Save and Load Model**: Persist the trained model by saving it into a JSON file and load it for later use.

- **Speech Feedback**: Hear the predicted sign language class spoken aloud using speech synthesis.

---

## Prerequisites

Before running SignLangSunc, ensure that you have the following dependencies:

- **TensorFlow.js**: For machine learning operations and tensor handling.

- **MobileNet**: A pre-trained model for feature extraction.

- **KNN Classifier**: Used to classify images based on the nearest neighbors.

- **Web Speech API**: For text-to-speech functionality.

Install the necessary libraries via npm:

```bash

npm install @tensorflow/tfjs @tensorflow-models/mobilenet @tensorflow-models/knn-classifier

```

---

## How It Works

1. **Webcam Input**:

   - Access your webcam using `navigator.mediaDevices.getUserMedia()` and display the live feed in an HTML element.

2. **MobileNet Model**:

   - The MobileNet model is loaded using `mobilenet.load()`, enabling feature extraction from captured images.

3. **KNN Classifier**:

   - Create a KNN classifier with `knnClassifier.create()`, which uses feature activations from MobileNet for classification.

4. **Add Custom Sign Language Classes**:

   - Users can add new classes (representing sign language gestures) by entering a class name and capturing images.

   - Each new image is added to the corresponding class in the KNN classifier.

5. **Train the Model**:

   - For each captured image, MobileNet generates a feature vector (activation), which is then added to the KNN classifier.

6. **Classify Signs**:

   - When the webcam feed is processed, the classifier predicts the most likely sign class from the current image.

   - The predicted class and confidence score are displayed in the UI.

7. **Model Persistence**:

   - Save the trained model into a JSON file using the "Save Model" button.

   - Load a previously saved model using the "Load Model" button to continue training or make predictions.

8. **Speech Output**:

   - The predicted sign language class can be announced aloud using the Web Speech API when the "Speak" button is clicked.

---

## Setup Instructions

1. Clone or download the repository to your local machine.

2. Open the `index.html` file in a modern browser (Chrome, Firefox, etc.) that supports Web APIs and TensorFlow.js.

3. The application will automatically request permission to access your webcam.

---

## User Interface

- **Input Class Name**: Enter the name of the new sign language class you want to create (e.g., "Hello", "Thank You").

- **Add Class**: Click this button to create a new class and start collecting images for it.

- **Training Cards**: Displays the added classes and their associated image counts.

- **Add New Images**: Capture new images for the selected class.

- **Predictions**: Displays the predicted sign language class for the current webcam frame.

- **Confidence**: Shows the classification confidence in percentage.

- **Save Model**: Save the current trained model to a JSON file.

- **Load Model**: Upload and load a previously saved model to continue training or use for classification.

- **Speak**: Hear the predicted class read aloud by clicking this button.

---

## Running the Application

1. Add a new class by typing a name (e.g., "Hello") and clicking **Add Class**.

2. Begin training the class by capturing images. Once you have enough images, start the classification process.

3. Use the **Start Classifying** button to begin predicting new sign gestures from the webcam.

4. Optionally, save your trained model using **Save Model** and load it again using **Load Model**.

---

## Important Notes

- **Performance**: The application runs in the browser, so performance may vary depending on the device and browser.

- **Compatibility**: Ensure your browser supports Web APIs like SpeechSynthesis, TensorFlow.js, and webcam access.

- **Webcam Permissions**: The app will prompt you to grant permission to access your webcam; please allow it for the app to function correctly.

---

## License

This project is open-source and available under the MIT License.

---

## Acknowledgements

- **TensorFlow.js**: For providing the machine learning framework in the browser.

- **MobileNet**: For the pre-trained feature extraction model.

- **KNN Classifier**: For easy classification of custom datasets.