{"id":23968761,"url":"https://github.com/sign-language-translator/sign-language-translator","last_synced_at":"2025-04-06T22:09:36.912Z","repository":{"id":169428645,"uuid":"626408754","full_name":"sign-language-translator/sign-language-translator","owner":"sign-language-translator","description":"Python library \u0026 framework to build custom translators for the hearing-impaired and translate between Sign Language \u0026 Text using Artificial Intelligence.","archived":false,"fork":false,"pushed_at":"2024-09-23T18:44:04.000Z","size":736,"stargazers_count":137,"open_issues_count":6,"forks_count":24,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-11-05T09:05:05.731Z","etag":null,"topics":["artificial-intelligence","attention-mechanism","computer-vision","deep-learning","encoder-decoder","language-models","machine-learning","motion-transfer","nlp","pose-estimation","python","pytorch","rule-based-nlp","sign-language","sign-language-recognition","sign-language-translation","sign-to-text","text-to-sign","transformers","translation"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/sign-language-translator/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sign-language-translator.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"custom":["https://www.paypal.com/donate/?hosted_button_id=7SNGNSKUQXQW2"]}},"created_at":"2023-04-11T12:12:32.000Z","updated_at":"2024-11-04T14:57:16.000Z","dependencies_parsed_at":"2024-01-15T14:15:09.021Z","dependency_job_id":"e2250bca-f40b-4bee-9be3-e3c73d3d02d6","html_url":"https://github.com/sign-language-translator/sign-language-translator","commit_stats":{"total_commits":68,"total_committers":2,"mean_commits":34.0,"dds":0.02941176470588236,"last_synced_commit":"0a427fb782408bed6588328a78cb3e678d44b314"},"previous_names":["sign-language-translator/sign-language-translator"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sign-language-translator%2Fsign-language-translator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sign-language-translator%2Fsign-language-translator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sign-language-translator%2Fsign-language-translator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sign-language-translator%2Fsign-language-translator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sign-language-translator","download_url":"https://codeload.github.com/sign-language-translator/sign-language-translator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247557767,"owners_count":20958047,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanism","computer-vision","deep-learning","encoder-decoder","language-models","machine-learning","motion-transfer","nlp","pose-estimation","python","pytorch","rule-based-nlp","sign-language","sign-language-recognition","sign-language-translation","sign-to-text","text-to-sign","transformers","translation"],"created_at":"2025-01-07T00:46:20.713Z","updated_at":"2025-04-06T22:09:36.887Z","avatar_url":"https://github.com/sign-language-translator.png","language":"Python","funding_links":["https://www.paypal.com/donate/?hosted_button_id=7SNGNSKUQXQW2"],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# Sign Language Translator ⠎⠇⠞\n\n\u003cimg width=\"61.8%\" alt=\"SLT: Sign Language Translator\" src=\"https://github.com/sign-language-translator/sign-language-translator/assets/118578823/d4723333-3d25-413d-83a1-a4bbdc8da15a\"\u003e\n\n\u003cbr\u003e\n\u003c/br\u003e\n\n***Build Custom Translators and Translate between Sign Language \u0026 Text with AI.***\n\n[![python](https://img.shields.io/pypi/pyversions/sign-language-translator?logo=python)](https://pypi.org/project/sign-language-translator/)\n[![PyPi](https://img.shields.io/pypi/v/sign-language-translator?logo=pypi)](https://pypi.org/project/sign-language-translator/)\n![GitHub branch check runs](https://img.shields.io/github/check-runs/sign-language-translator/sign-language-translator/main?logo=pytest)\n[![codecov](https://codecov.io/gh/sign-language-translator/sign-language-translator/branch/main/graph/badge.svg?precision=1)](https://codecov.io/gh/sign-language-translator/sign-language-translator)\n[![Documentation Status](https://img.shields.io/readthedocs/sign-language-translator?logo=readthedocs\u0026)](https://sign-language-translator.readthedocs.io/)\u003cbr\u003e\n[![GitHub Repo stars](https://img.shields.io/github/stars/sign-language-translator/sign-language-translator?logo=github)](https://github.com/sign-language-translator/sign-language-translator/stargazers)\n[![Repository Views per Month](https://img.shields.io/badge/dynamic/xml?url=https%3A%2F%2Fu8views.com%2Fapi%2Fv1%2Fgithub%2Fprofiles%2F118578823%2Fviews%2Fday-week-month-total-count.svg\u0026query=%2F*%5Blocal-name()%3D'svg'%5D%2F*%5Blocal-name()%3D'g'%5D%5B3%5D%2F*%5Blocal-name()%3D'g'%5D%2F*%5Blocal-name()%3D'text'%5D%5B2%5D\u0026label=%F0%9F%91%81%EF%B8%8F%20views%2Fmonth\u0026color=6d96ff\u0026cacheSeconds=5000)](https://u8views.com/github/mdsrqbl)\n[![Downloads](https://img.shields.io/pepy/dt/sign_language_translator?color=purple\u0026logoColor=white\u0026logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI4MDAiIGhlaWdodD0iODAwIiBmaWxsPSJub25lIiB2aWV3Qm94PSIwIDAgMjQgMjQiPjxwYXRoIGZpbGw9IiMwMDAiIGZpbGwtcnVsZT0iZXZlbm9kZCIgZD0iTTkuMiAyLjhjLS4yLjMtLjIuOC0uMiAxLjZWMTFINy44Yy0uOSAwLTEuMyAwLTEuNS4yYS44LjggMCAwIDAtLjMuNmMwIC4zLjMuNiAxIDEuMmw0LjEgNC40LjcuNmEuNy43IDAgMCAwIC40IDBsLjctLjZMMTcgMTNjLjYtLjYuOS0xIC45LTEuMmEuOC44IDAgMCAwLS4zLS42Yy0uMi0uMi0uNi0uMi0xLjUtLjJIMTVWNC40YzAtLjggMC0xLjMtLjItMS42YTEuNSAxLjUgMCAwIDAtLjYtLjZjLS4zLS4yLS44LS4yLTEuNi0uMmgtMS4yYy0uOCAwLTEuMyAwLTEuNi4yYTEuNSAxLjUgMCAwIDAtLjYuNnpNNSAyMWExIDEgMCAwIDAgMSAxaDEyYTEgMSAwIDEgMCAwLTJINmExIDEgMCAwIDAtMSAxeiIgY2xpcC1ydWxlPSJldmVub2RkIi8+PC9zdmc+)](https://pepy.tech/projects/sign-language-translator/)\u003cbr\u003e\n[![HuggingFace Spaces](https://img.shields.io/badge/%F0%9F%8C%90%20Web%20Demo-%F0%9F%A4%97%20hf.co%2FsltAI-mediumpurple)](https://huggingface.co/sltAI)\n\n| **Support Us** ❤️ | [![PayPal](https://img.shields.io/badge/PayPal-00457C?logo=paypal\u0026logoColor=white)](https://www.paypal.com/donate/?hosted_button_id=7SNGNSKUQXQW2) |\n| - | - |\n\n\u003c/div\u003e\n\n---\n\n1. [Overview](#overview)\n   1. [Solution](#solution)\n   2. [Major Components](#major-components)\n   3. [Goals](#goals)\n2. [**Installation** `🛠️`](#how-to-install-the-package)\n3. [**Usage**](#usage)\n   1. [Web 🌐](#web-gui)\n   2. [Python `🐍`](#python)\n   3. [Command Line \u003cspan style=\"color:green\"\u003e**`\u003e_`**\u003c/span\u003e](#command-line)\n4. [Languages](#languages)\n5. [Models](#models)\n6. [How to Build a Translator for your Sign Language](#how-to-build-a-translator-for-sign-language)\n7. [Module Hierarchy](#module-hierarchy)\n8. [How to Contribute](#how-to-contribute)\n9. [Citation, License \u0026 Research Papers](#citation-licence--research-papers)\n10. [Credits and Gratitude](#credits-and-gratitude)\n\n## Overview\n\nSign language consists of gestures and expressions used mainly by the hearing-impaired to talk. This project is an effort to bridge the communication gap between the hearing and the hearing-impaired community using Artificial Intelligence.\n\n\u003cdetails\u003e\n\u003csummary\u003eThis python library provides a user-friendly translation API and a framework for building sign language translators that can easily adapt to any regional sign language...\u003c/summary\u003e\n\u003c/br\u003e\nA big hurdle is the lack of datasets (global \u0026 regional) and frameworks that deep learning engineers and software developers can use to build useful products for the target community. This project aims to empower sign language translation by providing robust components, tools, datasets and models for both sign language to text and text to sign language conversion. It aims to facilitate the creation of sign language translators for any region, while building the way towards sign language standardization.\n\n\u003c/br\u003e\nUnlike most other projects, this python library can translate full sentences and not just the alphabet.\n\n\u003c/details\u003e\n\n\u003c!-- This is the package that powers the [slt_ai website](https://github.com/mdsrqbl/slt_ai). --\u003e\n\n### Solution\n\nThis package comes with an *extensible rule-based* text-to-sign translation system that can be used to generate training data for *Deep Learning* models for both sign to text \u0026 text to sign translation. \u003c!-- Pipelines for fine-tuning our deep learning models are also available. --\u003e\n\n\u003e [!Tip]\n\u003e To create a rule-based translation system for your regional language, you can inherit the TextLanguage and SignLanguage classes and pass them as arguments to the ConcatenativeSynthesis class. To write sample texts of supported words, you can use our language models. Then, you can use that system to fine-tune our deep learning models.\n\nSee the \u003ckbd\u003e[documentation](https://sign-language-translator.readthedocs.io)\u003c/kbd\u003e and our \u003ckbd\u003e[datasets](https://github.com/sign-language-translator/sign-language-datasets)\u003c/kbd\u003e for details.\n\n### Major Components\n\n\u003col\u003e\n\u003cli\u003e\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e\nSign language to Text\n\u003c/b\u003e\u003c/summary\u003e\n\n1. Extract features from sign language videos\n   1. See the [`slt.models.video_embedding`](https://github.com/sign-language-translator/sign-language-translator/tree/main/sign_language_translator/models/video_embedding) sub-package and the [`$ slt embed`](https://slt.readthedocs.io/en/latest/#embed-videos) command.\n   2. Currently Mediapipe 3D landmarks are being used for deep learning.\n2. Transcribe and translate signs into multiple text languages to generalize the model.\n3. To train for word-for-word gloss writing task, also use a synthetic dataset made by concatenating signs for each word in a text. (See [`slt.models.ConcatenativeSynthesis`](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/text_to_sign/concatenative_synthesis.py))\n4. Fine-tune a neural network, such as one from [`slt.models.sign_to_text`](https://github.com/sign-language-translator/sign-language-translator/tree/main/sign_language_translator/models/sign_to_text) or the encoder of any multilingual seq2seq model, on your dataset.\n\n\u003c/details\u003e\n\u003c/li\u003e\n\n\u003cli\u003e\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e\nText to Sign Language\n\u003c/b\u003e\u003c/summary\u003e\n\nThere are two approaches to this problem:\n\n1. Rule Based Concatenation\n   1. Label a Sign Language Dictionary with all word tokens that can be mapped to those signs. See our mapping format [here](https://github.com/sign-language-translator/sign-language-datasets/blob/main/parallel_texts/pk-dictionary-mapping.json).\n   2. Parse the input text and play appropriate video clips for each token.\n      1. Build a text processor by inheriting `slt.languages.TextLanguage` (see [`slt.languages.text`](https://github.com/sign-language-translator/sign-language-translator/tree/main/sign_language_translator/languages/text) sub-package for details)\n      2. Map the text grammar \u0026 words to sign language by inheriting `slt.languages.SignLanguage` (see [`slt.languages.sign`](https://github.com/sign-language-translator/sign-language-translator/tree/main/sign_language_translator/languages/sign) sub-package for details)\n      3. Use our rule-based model [`slt.models.ConcatenativeSynthesis`](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/text_to_sign/concatenative_synthesis.py) for translation.\n   3. It is faster but the **word sense has to be disambiguated** in the input. See the deep learning approach to automatically handle ambiguous words \u0026 **words not in dictionary**.\n\n2. Deep learning (seq2seq)\n   1. Either generate the sequence of filenames that should be concatenated \u003c!-- TODO: `slt.models.mGlossBART` --\u003e\n      1. you will need a [parallel corpus](https://github.com/sign-language-translator/sign-language-datasets/blob/main/parallel_texts/pk-sentence-mapping.json) of normal text sentences against sign language gloss (sign sequence written word-for-word)\n   2. Or synthesize the signs directly by using a pre-trained multilingual text encoder and\n      1. a GAN or diffusion model or decoder to synthesize a sequence of pose vectors (`shape = (time, num_landmarks * num_coordinates)`) \u003c!-- TODO: `slt.models.SignPoseGAN` --\u003e\n         1. Move an Avatar with those pose vectors (Easy) \u003c!-- TODO: `slt.models.Avatar` --\u003e\n         2. Use motion transfer to generate a video (Medium) \u003c!-- TODO: `slt.models.SignMotionTransfer` --\u003e\n         3. Synthesize a video frame for each vector (Difficult) \u003c!-- TODO: `slt.models.DeepPoseToImage` --\u003e\n      2. a video synthesis model (Very Difficult) \u003c!-- TODO: `slt.models.DeepSignVideoGAN` --\u003e\n\n\u003c/details\u003e\n\u003c/li\u003e\n\n\u003cli\u003e\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e\nLanguage Processing\n\u003c/b\u003e\u003c/summary\u003e\n\n1. Sign Processing\n   - 3D world landmarks extraction with Mediapipe.\n   - Pose Visualization with matplotlib.\n   - Pose transformations (data augmentation) with scipy.\n2. Text Processing\n   - Normalize text input by substituting unknown characters/spellings with supported words.\n   - Disambiguate context-dependent words to ensure accurate translation.\n         \"spring\" -\u003e [\"spring(water-spring)\", \"spring(metal-coil)\"]\n   - Tokenize text (word \u0026 sentence level).\n   - Classify tokens and mark them with Tags.\n\n\u003c/details\u003e\n\u003c/li\u003e\n\n\u003cli\u003e\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e\nDatasets\n\u003c/b\u003e\u003c/summary\u003e\n\nFor our datasets \u0026 conventions, see the [*sign-language-datasets* repo](https://github.com/sign-language-translator/sign-language-datasets) and its [releases](https://github.com/sign-language-translator/sign-language-datasets/releases).\nSee this [documentation](https://slt.readthedocs.io/en/latest/datasets.html) for more on building a dataset of Sign Language videos (or motion capture gloves' output features).\n\n**Your data should include**:\n\n1. A word level Dictionary (Videos of individual signs \u0026 corresponding Text tokens (words \u0026 phrases))\n2. Replications of the dictionary. (Set up multiple syncronized cameras and record random people performing the dictionary videos. ([notebook](https://colab.research.google.com/github/sign-language-translator/notebooks/blob/main/data_collection/clip_extractor.ipynb)))\n3. Parallel sentences\n   1. Normal text language sentences against sign language videos. (Use our Language Models to generate sentences composed of dictionary words.)\n   2. Normal text language sentences against the [text gloss](https://github.com/sign-language-translator/sign-language-datasets#glossary) of the corresponding sign language sentence.\n   3. Sign language sentences against their text gloss\n   4. Sign language sentences against translations in multiple text languages\n4. Grammatical rules of the sign language\n   1. Word order (e.g. SUBJECT OBJECT VERB TIME)\n   2. Meaningless words (e.g. \"the\", \"am\", \"are\")\n   3. Ambiguous words (e.g. spring(coil) \u0026 spring(water-fountain))\n\n**Try to incorporate**:\n\n1. Multiple camera angles\n2. Diverse performers to capture all *accents* of the signs\n3. Uniqueness in labeling of word tokens\n4. Variations in signs for the same concept\n\nTry to capture variations in signs in a scalable and diversity accommodating way and enable advancing sign language standardization efforts.\n\n\u003c/details\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\n### Goals\n\n1. Enable **integration** of sign language into existing applications.\n2. Assist construction of **custom** solutions for resource poor sign langauges.\n3. Improve **education** quality for the deaf and elevate literacy rates.\n4. Promote communication **inclusivity** of the hearing impaired.\n5. Establish a framework for sign language **standardization**.\n\n## How to install the package\n\n```bash\npip install sign-language-translator\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eEditable mode (\u003ccode\u003egit clone\u003c/code\u003e):\u003c/summary\u003e\n\nThe package ships with some optional dependencies as well (e.g. deep_translator for synonym finding and mediapipe for a pretrained pose extraction model). Install them by appending `[all]`, `[full]`, `[mediapipe]` or `[synonyms]` to the project name in the command (e.g `pip install sign-langauge-translator[full]`).\n\n```bash\ngit clone https://github.com/sign-language-translator/sign-language-translator.git\ncd sign-language-translator\npip install -e \".[all]\"\n```\n\n```bash\npip install -e git+https://github.com/sign-language-translator/sign-language-translator.git#egg=sign_language_translator\n```\n\n\u003c/details\u003e\n\n## Usage\n\nHead over to [slt.**readthedocs**.io](https://slt.readthedocs.io) to see the detailed usage in Python, CLI and gradio GUI.\nSee the [*test cases*](https://github.com/sign-language-translator/sign-language-translator/blob/main/tests) or the [*notebooks* repo](https://github.com/sign-language-translator/notebooks) to see the internal code in action.\n\n### Web GUI\n\nIndividual models deployed on HuggingFace Spaces:\n\n[![HuggingFace Spaces](https://img.shields.io/badge/text%20to%20sign-ConcatenativeSynthesis%2BLM-mediumslateblue)](https://huggingface.co/spaces/sltAI/ConcatenativeSynthesis)\n\n\u003c!-- #FF3270 #FFD702 #861FFF #097EFF #10B981 #3B82F6 #6366F1 #F59E0B #EF4444 --\u003e\n\n### Python\n\n```python\nimport sign_language_translator as slt\n\n# The core model of the project (rule-based text-to-sign translator)\n# which enables us to generate synthetic training datasets\nmodel = slt.models.ConcatenativeSynthesis(\n   text_language=\"urdu\", sign_language=\"pk-sl\", sign_format=\"video\" )\n\ntext = \"یہ بہت اچھا ہے۔\" # \"this-very-good-is\"\nsign = model.translate(text) # tokenize, map, download \u0026 concatenate\nsign.show()\n\nmodel.sign_format = slt.SignFormatCodes.LANDMARKS\nmodel.sign_embedding_model = \"mediapipe-world\"\n\n# ==== English ==== #\nmodel.text_language = slt.languages.text.English()\nsign_2 = model.translate(\"This is an apple.\")\nsign_2.save(\"this-is-an-apple.csv\", overwrite=True)\n\n# ==== Hindi ==== #\nmodel.text_language = slt.TextLanguageCodes.HINDI\nsign_3 = model.translate(\"कैसे हैं आप?\") # \"how-are-you\"\nsign_3.save_animation(\"how-are-you.gif\", overwrite=True)\n```\n\n| ![this very good is](https://github.com/sign-language-translator/sign-language-translator/assets/118578823/7f4ff312-df03-4b11-837b-5fb895c9f08e) | \u003cpicture\u003e\u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://github.com/user-attachments/assets/4d54a197-d723-4cc4-a3ba-cae98e681003\" /\u003e\u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://github.com/user-attachments/assets/45e71098-7a94-4a9e-ad24-1773369b65d5\" /\u003e\u003cimg alt=\"how are you (landmark 3d plot)\" src=\"https://github.com/user-attachments/assets/45e71098-7a94-4a9e-ad24-1773369b65d5\" /\u003e\u003c/picture\u003e |\n| :-: | :-: |\n| \"یہ بہت اچھا ہے۔\" (this-very-good-is) | \"कैसे हैं आप?\" (how-are-you) |\n\n\u003c/br\u003e\n\n```python\nimport sign_language_translator as slt\n\n# sign = slt.Video(\"path/to/video.mp4\")\nsign = slt.Video.load_asset(\"pk-hfad-1_aap-ka-nam-kya(what)-hy\")  # your name what is? (auto-downloaded)\nsign.show_frames_grid()\n\n# Extract Pose Vector for feature reduction\nembedding_model = slt.models.MediaPipeLandmarksModel()      # pip install \"sign_language_translator[mediapipe]\"  # (or [all])\nembedding = embedding_model.embed(sign.iter_frames())\n\nslt.Landmarks(embedding.reshape((-1, 75, 5)),\n              connections=\"mediapipe-world\"  ).show()\n\n# # Load sign-to-text model (pytorch) (COMING SOON!)\n# translation_model = slt.get_model(slt.ModelCodes.Gesture)\n# text = translation_model.translate(embedding)\n# print(text)\n```\n\n```python\n# custom translator (https://slt.readthedocs.io/en/latest/#building-custom-translators)\nhelp(slt.languages.SignLanguage)\nhelp(slt.languages.text.Urdu)\nhelp(slt.models.ConcatenativeSynthesis)\n```\n\n### Command Line\n\n```bash\n$ slt\n\nUsage: slt [OPTIONS] COMMAND [ARGS]...\n   Sign Language Translator (SLT) command line interface.\n   Documentation: https://sign-language-translator.readthedocs.io\nOptions:\n  --version  Show the version and exit.\n  --help     Show this message and exit.\nCommands:\n  assets     Assets manager to download \u0026 display Datasets \u0026 Models.\n  complete   Complete a sequence using Language Models.\n  embed      Embed Videos Using Selected Model.\n  translate  Translate text into sign language or vice versa.\n```\n\n**Generate training examples**: write a sentence with a language model and synthesize a sign language video from it with a single command:\n\n```bash\nslt translate --model-code rule-based --text-lang urdu --sign-lang pk-sl --sign-format video \\\n\"$(slt complete '\u003c' --model-code urdu-mixed-ngram --join '')\"\n```\n\n## Languages\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eText Languages\u003c/b\u003e\u003c/summary\u003e\n\nAvailable Functions:\n\n- Text Normalization\n- Tokenization (word, phrase \u0026 sentence)\n- Token Classification (Tagging)\n- Word Sense Disambiguation\n\n| Name | Vocabulary | Ambiguous tokens | Signs |\n| - | - | - | - |\n| [English](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/text/english.py) | 1591 words+phrases | 167 | 776 |\n| [Urdu](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/text/urdu.py)    | 2080 words+phrases | 227 | 776 |\n| [Hindi](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/text/hindi.py)   |  137 words+phrases |   5 |  84 |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eSign Languages\u003c/b\u003e\u003c/summary\u003e\n\nAvailable Functions:\n\n- Word \u0026 phrase mapping to signs\n- Sentence restructuring according to grammar\n- Sentence simplification (drop stopwords)\n\n| Name | Vocabulary | Dataset | Parallel Corpus |\n| - | - | - | :-: |\n| [Pakistan Sign Language](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/sign/pakistan_sign_language.py) | 776 | 23 hours | [details](https://github.com/sign-language-translator/sign-language-datasets#datasets) |\n\n\u003c/details\u003e\n\n## Models\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eTranslation\u003c/b\u003e: Text to sign Language\u003c/summary\u003e\n\n\u003c!-- [Available Trained models]() --\u003e\n\n| Name                                                                                                                                                                              | Architecture        | Description                                                                                                                                            | Input  | Output                     | Web Demo                                                                                                                                                                  |\n| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| [Concatenative Synthesis](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/text_to_sign/concatenative_synthesis.py) | Rules + Hash Tables | The Core Rule-Based translator mainly used to synthesize translation dataset.\u003cbr/\u003eInitialize it using TextLanguage, SignLanguage \u0026 SignFormat objects. | string | slt.Video \\| slt.Landmarks | [![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/spaces/sltAI/ConcatenativeSynthesis) |\n\n\u003c!--                                                                                                                                                                              | [pose-gen]()        | Encoder-Decoder Transformers (Seq2Seq)                                                                                                              | Generates a sequence of pose vectors conditioned on input text. | torch.Tensor\u003cbr/\u003e(batch, token_ids) | torch.Tensor\u003cbr/\u003e(batch, n_frames, n_landmarks*3) |  | --\u003e\n\n\u003c/details\u003e\n\n\u003c!-- \u003cdetails\u003e\n\u003csummary\u003eTranslation: Sign Language to Text\u003c/summary\u003e\n\n[Available Trained models]()\n\n| Name        | Architecture                                | Description                                                                                                  | Input format                                             | Output format                       |\n| ----------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- | ----------------------------------- |\n| [gesture]() | CNN+Encoder - Decoder Transformer (seq2seq) | Encodes the pose vectors depicting sign language sentence and generates text conditioned on those encodings. | torch.Tensor\u003cbr/\u003e(batch, n_frames=1000, n_landmarks * 3) | torch.Tensor\u003cbr/\u003e(batch, token_ids) |\n\u003c/details\u003e --\u003e\n\n\u003c!--\n\u003cdetails\u003e\n\u003csummary\u003eVideo: Synthesis/Generation\u003c/summary\u003e\n\n[Available Trained models]()\n\n| Name        | Architecture                                | Description                                                                                                  | Input format                                             | Output format                       |\n| ----------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- | ----------------------------------- |\n| [gesture]() | CNN+Encoder - Decoder Transformer (seq2seq) | Encodes the pose vectors depicting sign language sentence and generates text conditioned on those encodings. | torch.Tensor\u003cbr/\u003e(batch, n_frames=1000, n_landmarks * 3) | torch.Tensor\u003cbr/\u003e(batch, token_ids) |\n\u003c/details\u003e\n--\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eSign Embedding/Feature extraction\u003c/b\u003e:\u003c/summary\u003e\n\n\u003c!-- [Available Trained models]() --\u003e\n\n| Name                                                                                                                                                                                                 | Architecture                                                                                                               | Description                                                                                       | Input format                                                 | Output format                                |\n| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | -------------------------------------------- |\n| [MediaPipe Landmarks\u003cbr\u003e(Pose + Hands)](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/video_embedding/mediapipe_landmarks_model.py) | CNN based pipelines. See Here: [Pose](https://arxiv.org/pdf/2006.10204.pdf), [Hands](https://arxiv.org/pdf/2006.10214.pdf) | Encodes videos into pose vectors (3D world or 2D image) depicting the movements of the performer. | List of numpy images\u003cbr/\u003e(n_frames, height, width, channels) | torch.Tensor\u003cbr/\u003e(n_frames, n_landmarks * 5) |\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eData generation\u003c/b\u003e: Language Models\u003c/summary\u003e\n\n[Available Trained models](https://github.com/sign-language-translator/sign-language-datasets/releases/tag/v0.0.1)\n\n| Name                                                                                                                                                                                             | Architecture                    | Description                                                                                         | Input format                                                | Output format                                                                 |\n| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------- |\n| [N-Gram Langauge Model](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/language_models/ngram_language_model.py)                  | Hash Tables                     | Predicts the next token based on learned statistics about previous N tokens.                        | List of tokens                                              | (token, probability)                                                          |\n| [Transformer Language Model](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/language_models/transformer_language_model/model.py) | Decoder-only Transformers (GPT) | Predicts next token using query-key-value attention, linear transformations and soft probabilities. | torch.Tensor\u003cbr/\u003e(batch, token_ids)\u003cbr/\u003e\u003cbr/\u003eList of tokens | torch.Tensor\u003cbr/\u003e(batch, token_ids, vocab_size)\u003cbr/\u003e\u003cbr/\u003e(token, probability) |\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eText Embedding\u003c/b\u003e:\u003c/summary\u003e\n\n[Available Trained models](https://github.com/sign-language-translator/sign-language-datasets/releases/)\n\n| Name                                                                                                                                                                  | Architecture | Description                                                                                                             | Input format | Output format             |\n| --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------- | ------------ | ------------------------- |\n| [Vector Lookup](https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/text_embedding/vector_lookup_model.py) | HashTable    | Finds token index and returns the coresponding vector. Tokenizes sentences and computes average vector of known tokens. | string       | torch.Tensor\u003cbr/\u003e(n_dim,) |\n\n\u003c/details\u003e\n\n\u003c!--\n## Servers\n\n| Name (repository) | Framework | Docker  | Status       |\n| ----------------- | --------- | ------- | ------------ |\n| [slt-models]()    | FastAPI   | [url]() | Coming Soon! |\n| [slt-backend]()   | Django    | [url]() | Coming Soon! |\n| [slt-frontend]()  | React     | [url]() | Coming Soon! |\n\nYou can interact with the live version of the above servers at [something.com](https://www.something.com)\n--\u003e\n\n## How to Build a Translator for Sign Language\n\nTo create your own sign language translator, you'll need these essential components:\n\n\u003col\u003e\n\u003cli\u003e\n\u003cdetails\u003e\n\u003csummary\u003eData Collection\u003c/summary\u003e\n\n   1. Gather a collection of [dictionary videos](https://github.com/sign-language-translator/sign-language-datasets/releases/tag/v0.0.2) (word level) featuring individuals performing sign language gestures. These can be obtained from schools \u0026 organizations for the deaf. You should record multiple people perform the same sign to capture various *accents* of the sign. Set up multiple cameras in different locations in parallel to further augment the data.\n   2. Prepare a [JSON file](https://github.com/sign-language-translator/sign-language-datasets/blob/main/parallel_texts/pk-dictionary-mapping.json) that maps dictionary video file names to corresponding text language words \u0026 phrases that are synonymous with the gestures.\n   3. Prepare a synthetic data [parallel corpus](https://github.com/sign-language-translator/sign-language-datasets/blob/main/parallel_texts/pk-synthetic-sentence-mapping.json) containing text language sentences and sequences of sign language video filenames. You can use langauge models to generate these sentences \u0026 sequences.\n   4. Prepare a dataset of sign language [sentence videos](https://github.com/sign-language-translator/sign-language-datasets/releases/tag/v0.0.3) that are labeled with [translations \u0026 glosses](https://github.com/sign-language-translator/sign-language-datasets/blob/main/parallel_texts/pk-sentence-mapping.json) in multiple text languages.\n\n\u003c/details\u003e\n\u003c/li\u003e\n\n\u003cli\u003e\n\u003cdetails\u003e\n\u003csummary\u003eLanguage Processing\u003c/summary\u003e\n\n   1. Implement a subclass of `slt.languages.TextLanguage`:\n      - Tokenize your text language and assign appropriate tags to the tokens for streamlined processing.\n   2. Create a subclass of `slt.languages.SignLanguage`:\n      - Map text tokens to video filenames using the provided JSON data.\n      - Rearrange the sequence of video filenames to align with the grammar and structure of sign language.\n\n\u003c/details\u003e\n\u003c/li\u003e\n\n\u003cli\u003e\n\u003cdetails\u003e\n\u003csummary\u003eRule-Based Translation\u003c/summary\u003e\n\n   1. Pass instances of your classes from the previous step to `slt.models.ConcatenativeSynthesis` class to obtain a rule-based translator object.\n   2. Construct sentences in your text language and use the rule-based translator to generate sign language translations. (You can use our language models to generate such texts.)\n\n\u003c/details\u003e\n\u003c/li\u003e\n\n\u003cli\u003e\n\u003cdetails\u003e\n\u003csummary\u003eDeep Learning Model Fine-Tuning\u003c/summary\u003e\n\n   1. Utilize the (synthetic \u0026 real) sign language videos and corresponding text sentences from the previous step.\n   2. Apply our training pipeline to fine-tune a chosen model for improved accuracy and translation quality.\n\n\u003c/details\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\nRemember to contribute back to the community:\n\n- Share your data, code, and models by creating a pull request, allowing others to benefit from your efforts.\n- Create your own sign language translator (e.g. as your university thesis) and contribute to a more inclusive and accessible world.\n\nSee the `code` at [Build Custom Translator section in ReadTheDocs](https://sign-language-translator.readthedocs.io/en/latest/#building-custom-translators) or in this [notebook](https://github.com/sign-language-translator/notebooks/blob/main/translation/concatenative_synthesis.ipynb). [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/sign-language-translator/notebooks/blob/main/translation/concatenative_synthesis.ipynb)\n\n## Module Hierarchy\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb style=\"font-size:large;\"\u003e\u003ccode\u003esign-language-translator\u003c/code\u003e\u003c/b\u003e (Click to see file descriptions)\u003c/summary\u003e\n\n\u003cpre\u003e\n├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/README.md\"\u003eREADME.md\u003c/a\u003e\n├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/pyproject.toml\"\u003epyproject.toml\u003c/a\u003e\n├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/requirements.txt\"\u003erequirements.txt\u003c/a\u003e\n├── \u003cb\u003edocs\u003c/b\u003e\n│   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/docs\"\u003e*\u003c/a\u003e\n├── \u003cb\u003etests\u003c/b\u003e\n│   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/tests\"\u003e*\u003c/a\u003e\n│\n└── \u003cb style=\"font-size:large;\"\u003esign_language_translator\u003c/b\u003e\n    ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/cli.py\"\u003ecli.py\u003c/a\u003e \u003csub\u003e\u003csup\u003e`\u003e slt` command line interface\u003c/sup\u003e\u003c/sub\u003e\n    ├── \u003ci\u003e\u003cb\u003eassets\u003c/b\u003e\u003c/i\u003e \u003csub\u003e\u003csup\u003e(auto-downloaded)\u003c/sup\u003e\u003c/sub\u003e\n    │   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-datasets\"\u003e*\u003c/a\u003e\n    │\n    ├── \u003cb\u003econfig\u003c/b\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/config/assets.py\"\u003eassets.py\u003c/a\u003e \u003csub\u003e\u003csup\u003edownload, extract and remove models \u0026 datasets\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/config/colors.py\"\u003ecolors.py\u003c/a\u003e \u003csub\u003e\u003csup\u003enamed RGB tuples for visualization\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/config/enums.py\"\u003eenums.py\u003c/a\u003e \u003csub\u003e\u003csup\u003estring short codes to identify models \u0026 classes\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/config/settings.py\"\u003esettings.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eglobal variables in repository design-pattern\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── \u003ci\u003e\u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/config/urls.json\"\u003eurls.json\u003c/a\u003e\u003c/i\u003e\n    │   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/config/utils.py\"\u003eutils.py\u003c/a\u003e\n    │\n    ├── \u003cb\u003elanguages\u003c/b\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/utils.py\"\u003eutils.py\u003c/a\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/vocab.p\"\u003evocab.py\u003c/a\u003e \u003csub\u003e\u003csup\u003ereads word mapping \u003ca href=\"https://github.com/sign-language-translator/sign-language-datasets/tree/main/parallel_texts\"\u003edatasets\u003c/a\u003e\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── sign\n    │   │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/sign/mapping_rules.py\"\u003emapping_rules.py\u003c/a\u003e \u003csub\u003e\u003csup\u003estrategy design-pattern for word to sign mapping\u003c/sup\u003e\u003c/sub\u003e\n    │   │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/sign/pakistan_sign_language.py\"\u003epakistan_sign_language.py\u003c/a\u003e\n    │   │   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/sign/sign_language.py\"\u003esign_language.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eBase class for text to sign mapping and sentence restructuring\u003c/sup\u003e\u003c/sub\u003e\n    │   │\n    │   └── text\n    │       ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/text/english.py\"\u003eenglish.py\u003c/a\u003e\n    │       ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/text/hindi.py\"\u003ehindi.py\u003c/a\u003e\n    │       ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/text/text_language.py\"\u003etext_language.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eBase class for text normalization, tokenization \u0026 tagging\u003c/sup\u003e\u003c/sub\u003e\n    │       └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/languages/text/urdu.py\"\u003eurdu.py\u003c/a\u003e\n    │\n    ├── \u003cb\u003emodels\u003c/b\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/_utils.py\"\u003e_utils.py\u003c/a\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/utils.py\"\u003eutils.py\u003c/a\u003e\n    │   ├── language_models\n    │   │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/language_models/abstract_language_model.py\"\u003eabstract_language_model.py\u003c/a\u003e\n    │   │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/language_models/beam_sampling.py\"\u003ebeam_sampling.py\u003c/a\u003e\n    │   │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/language_models/mixer.py\"\u003emixer.py\u003c/a\u003e \u003csub\u003e\u003csup\u003ewrap multiple language models into a single object\u003c/sup\u003e\u003c/sub\u003e\n    │   │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/language_models/ngram_language_model.py\"\u003engram_language_model.py\u003c/a\u003e \u003csub\u003e\u003csup\u003euses hash-tables \u0026 frequency to predict next token\u003c/sup\u003e\u003c/sub\u003e\n    │   │   └── transformer_language_model\n    │   │       ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/language_models/transformer_language_model/layers.py\"\u003elayers.py\u003c/a\u003e\n    │   │       ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/language_models/transformer_language_model/model.py\"\u003emodel.py\u003c/a\u003e \u003csub\u003e\u003csup\u003edecoder-only transformer with controllable vocabulary\u003c/sup\u003e\u003c/sub\u003e\n    │   │       └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/language_models/transformer_language_model/train.py\"\u003etrain.py\u003c/a\u003e\n    │   │\n    │   ├── sign_to_text\n    │   ├── text_to_sign\n    │   │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/text_to_sign/concatenative_synthesis.py\"\u003econcatenative_synthesis.py\u003c/a\u003e \u003csub\u003e\u003csup\u003ejoin sign clip of each word in text using rules\u003c/sup\u003e\u003c/sub\u003e\n    │   │   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/text_to_sign/t2s_model.py\"\u003et2s_model.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eBase class\u003c/sup\u003e\u003c/sub\u003e\n    │   │\n    │   ├── text_embedding\n    │   │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/text_embedding/text_embedding_model.py\"\u003etext_embedding_model.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eBase class\u003c/sup\u003e\u003c/sub\u003e\n    │   │   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/text_embedding/vector_lookup_model.py\"\u003evector_lookup_model.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eretrieves word embedding from a vector database\u003c/sup\u003e\u003c/sub\u003e\n    │   │\n    │   └── video_embedding\n    │       ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/video_embedding/mediapipe_landmarks_model.py\"\u003emediapipe_landmarks_model.py\u003c/a\u003e \u003csub\u003e\u003csup\u003e2D \u0026 3D coordinates of points on body\u003c/sup\u003e\u003c/sub\u003e\n    │       └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/models/video_embedding/video_embedding_model.py\"\u003evideo_embedding_model.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eBase class\u003c/sup\u003e\u003c/sub\u003e\n    │\n    ├── \u003cb\u003etext\u003c/b\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/text/metrics.py\"\u003emetrics.py\u003c/a\u003e \u003csub\u003e\u003csup\u003enumeric score techniques\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/text/preprocess.py\"\u003epreprocess.py\u003c/a\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/text/subtitles.py\"\u003esubtitles.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eWebVTT\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/text/synonyms.py\"\u003esynonyms.py\u003c/a\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/text/tagger.py\"\u003etagger.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eclassify tokens to assist in mapping\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/text/tokenizer.py\"\u003etokenizer.py\u003c/a\u003e \u003csub\u003e\u003csup\u003ebreak text into words, phrases, sentences etc\u003c/sup\u003e\u003c/sub\u003e\n    │   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/text/utils.py\"\u003eutils.py\u003c/a\u003e\n    │\n    ├── \u003cb\u003eutils\u003c/b\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/utils/archive.py\"\u003earchive.py\u003c/a\u003e \u003csub\u003e\u003csup\u003ezip datasets\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/utils/arrays.py\"\u003earrays.py\u003c/a\u003e \u003csub\u003e\u003csup\u003ecommon interface \u0026 operations for numpy.ndarray and torch.Tensor\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/utils/download.py\"\u003edownload.py\u003c/a\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/utils/parallel.py\"\u003eparallel.py\u003c/a\u003e \u003csub\u003e\u003csup\u003emulti-threading\u003c/sup\u003e\u003c/sub\u003e\n    │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/utils/tree.py\"\u003etree.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eprint file hierarchy\u003c/sup\u003e\u003c/sub\u003e\n    │   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/utils/utils.py\"\u003eutils.py\u003c/a\u003e\n    │\n    └── \u003cb\u003evision\u003c/b\u003e\n        ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/vision/_utils.py\"\u003e_utils.py\u003c/a\u003e\n        ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/vision/utils.py\"\u003eutils.py\u003c/a\u003e\n        ├── landmarks\n        │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/vision/landmarks/connections.py\"\u003econnections.py\u003c/a\u003e \u003csub\u003e\u003csup\u003edrawing configurations for different landmarks models\u003c/sup\u003e\u003c/sub\u003e\n        │   ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/vision/landmarks/display.py\"\u003edisplay.py\u003c/a\u003e \u003csub\u003e\u003csup\u003evisualize points \u0026 lines on 3D plot\u003c/sup\u003e\u003c/sub\u003e\n        │   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/vision/landmarks/landmarks.py\"\u003elandmarks.py\u003c/a\u003e \u003csub\u003e\u003csup\u003ewrapper for sequence of collection of points on body\u003c/sup\u003e\u003c/sub\u003e\n        │\n        ├── sign\n        │   └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/vision/sign/sign.py\"\u003esign.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eBase class to wrap around sign clips\u003c/sup\u003e\u003c/sub\u003e\n        │\n        └── video\n            ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/vision/video/display.py\"\u003edisplay.py\u003c/a\u003e \u003csub\u003e\u003csup\u003ejupyter notebooks inline video \u0026 pop-up in CLI\u003c/sup\u003e\u003c/sub\u003e\n            ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/vision/video/transformations.py\"\u003etransformations.py\u003c/a\u003e \u003csub\u003e\u003csup\u003estrategy design-pattern for image augmentation\u003c/sup\u003e\u003c/sub\u003e\n            ├── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/vision/video/video_iterators.py\"\u003evideo_iterators.py\u003c/a\u003e \u003csub\u003e\u003csup\u003eadapter design-pattern for video reading\u003c/sup\u003e\u003c/sub\u003e\n            └── \u003ca href=\"https://github.com/sign-language-translator/sign-language-translator/blob/main/sign_language_translator/vision/video/video.py\"\u003evideo.py\u003c/a\u003e\n\u003c/pre\u003e\n\n\u003c/details\u003e\n\n## How to Contribute\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eDatasets\u003c/b\u003e:\u003c/summary\u003e\n\nSee our datasets \u0026 conventions [here](https://github.com/sign-language-translator/sign-language-datasets).\n\n- Contribute by scraping, compiling, and centralizing video datasets.\n- Help with labeling [word mapping datasets](https://github.com/sign-language-translator/sign-language-datasets/tree/main/parallel_texts).\n- Establish connections with Academies for the Deaf to collaboratively develop standardized *sign language grammar* and integrate it into the rule-based translators.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eNew Code\u003c/b\u003e:\u003c/summary\u003e\n\n- Create dedicated sign language classes catering to various regions.\n- Develop text language processing classes for diverse languages.\n- Experiment with training models using diverse hyper-parameters.\n- Don't forget to integrate `string short codes` of your classes and models into **`enums.py`**, and ensure to update factory functions like `get_model()` and `get_.*_language()`.\n- Enhance the codebase with comprehensive docstrings, exemplary usage cases, and thorough test cases.\n\n\u003c/details\u003e\n\n\u003cdetails open\u003e\n\u003csummary\u003e\u003cb\u003eExisting Code\u003c/b\u003e:\u003c/summary\u003e\n\n- Implement the `# ToDo` from the code or fix `# bug` / `# type: ignore` or anything from the [roadmap](#upcomingroadmap).\n- Optimize the codebase by implementing techniques like parallel processing and batching.\n- Strengthen the project with clear docstrings containing illustrative examples and with robust test coverage.\n- Contribute to the documentation for [sign-language-translator ReadTheDocs](https://github.com/sign-language-translator/sign-language-translator/blob/main/docs/index.rst) to empower users with comprehensive insights. Currently it needs a **better template** for the auto-generated pages.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eProduct Development\u003c/b\u003e:\n\u003c/summary\u003e\n\n- Engage in the development efforts across [MLOps](https://huggingface.co/sltAI) \u0026 [web-frontend](https://github.com/sign-language-translator/slt-frontend) \u003c!-- [backend](https://github.com/sign-language-translator/slt-backend), and [mobile](https://github.com/sign-language-translator/slt-mobile) --\u003e domains, depending on your expertise and interests.\n\n\u003c/details\u003e\n\n## Upcoming/Roadmap\n\n\u003cdetails open\u003e\n\u003csummary\u003eLANDMARKS_WRAPPER: v0.8\u003c/summary\u003e\n\n```python\n# 0.8.2: landmark augmentation (zoom, rotate, move, noise, duration, rectify, stabilize, __repr__)\n# 0.8.3: trim signs before concatenation, insert transition frames\n# 0.8.4: plotly \u0026 three.js/mixamo display , pass matplotlib kwargs all the way down\n\n# 0.8.5: subtitles/captions\n# 0.8.6: stabilize video batch using landmarks, draw/overlay 2D landmarks on video/image\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eCLEAN_UP: v0.9\u003c/summary\u003e\n\n```python\n# mock test cases which require internet when internet isn't available / test for  dummy languages\n# improve langauge classes architecture (for easy customization via inheritance) | clean-up slt.languages.text.* code\n# ? add a generic SignedTextLanguage class which just maps text lang to signs based on mappinng.json ?\n# add progress bar to slt.models.MediaPipeLandmarksModel\n\n# rename 'country' to 'region' \u0026 rename wordless_wordless to wordless.mp4 # insert video type to archives: .*.videos-`(dictionary|sentences)(-replication)?`-mp4.zip\n# decide mediapipe-all = world \u0026 image concactenated in landmark dim or feature dim?\n# expand dictionary video data by scraping everything\n# upload the 12 person dictionary replication landmark dataset\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eDEEP_TRANSLATION: v0.9 - v1.2\u003c/summary\u003e\n\n```python\n# 0.9.1: TransformerLanguageModel - Drop space tokens \u0026 bidirectional prediction. infer on specific vocab only .... pretrain on max vocab and mixed data. finetune on balanced data (wiki==news==novels==poetry==reviews) .... then RLHF on coherent generations (Comparison data: generate 100 examples (at high temperature) and cut them at random points and regerate the rest and label these pairs for coherence[ and novelity].) (use same model/BERT as reward model with regression head.) (ranking loss with margin) (each token is a time step) (min KL Divergance from base - exploration without mode collapse) ... label disambiguation data and freeze all and finetune disambiguated_tokens_embeddings  (disambiguated embedding: word ± 0.1*(sense1 - sense2).normalize()) .... generate data on broken compound words and finetune their token_embeddings ... generate sentences of supported words and translate to other languages.\n# 0.9.2: sign to text with custom seq2seq transformer\n# 0.9.3: pose vector generation from text with custom seq2seq transformer\n# 0.9.4: sign to text with fine-tuned whisper\n# 0.9.5: pose vector generation with fine-tuned mBERT\n# 0.9.6: custom 3DLandmark model (training data = mediapipe's output on activity recognition or any dataset)\n# 1.0.0: all models trained on custom landmark model\n# 🎉\n# 1.0.1: video to text model (connect custom landmark model with sign2text model and finetune)\n# 1.1.0: motion transfer\n# 1.1.1: custom pose2video: stable diffusion or GAN?\n# 1.2.0: speech to sign\n# 1.2.1: sign to speech\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eMISCELLANEOUS\u003c/summary\u003e\n\nIssues\n\n```python\n# bugfix:      inaccurate num_frames in video file metadata\n# bugfix:      Expression of type \"Literal[False]\" cannot be assigned to member \"SHOW_DOWNLOAD_PROGRESS\" of class \"Settings\"\n# feature:     video transformations (e.g. stabilization with image pose landmarks, watermark text/logo)\n# improvement: SignFilename.parse(\"videos/pk-hfad-1_airplane.mp4\").gloss  # airplane\n```\n\nMiscellaneous\n\n```python\n# parallel text corpus\n# clean demonstration notebooks\n# * host video dataset online, descriptive filenames\n# dataset info table\n# sequence diagram for creating a translator\n# GUI with gradio or something\n```\n\nResearch Papers\n\n```python\n# datasets: clips, text, sentences, disambiguation\n# rule based translation: describe entire repo\n# deep sign-to-text: pipeline + experiments\n# deep text-to-sign: pipeline + experiments\n```\n\nServers / Product\n\n```python\n# ML inference server\n# Django backend server\n# React Native mobile app\n```\n\n[![Total Views](https://u8views.com/api/v1/github/profiles/118578823/views/total-count.svg)](https://u8views.com/github/mdsrqbl)\n\n\u003c/details\u003e\n\n## Citation, Licence \u0026 Research Papers\n\n```bibtex\n@software{mdsr2023slt,\n  author       = {Mudassar Iqbal},\n  title        = {Sign Language Translator: Python Library and AI Framework},\n  year         = {2023},\n  publisher    = {GitHub},\n  howpublished = {\\url{https://github.com/sign-language-translator/sign-language-translator}},\n}\n```\n\nThis project is licensed under the [Apache 2.0 License](https://github.com/sign-language-translator/sign-language-translator/blob/main/LICENSE). You are permitted to use the library, create modified versions, or incorporate pieces of the code into your own work. Your product or research, whether commercial or non-commercial, must provide appropriate credit to the original author(s) by citing this repository.\n\nStay Tuned for research Papers!\n\n## Credits and Gratitude\n\nThis project started in October 2021 as a BS Computer Science final year project with 3 students and 1 supervisor. After 9 months at university, it became a hobby project for [Mudassar](https://github.com/mdsrqbl) who has continued it till at least 2024-09-23.\n\n## Bonus\n\nCount total number of **lines of code** (Package: **14,034** + Tests: **2,928**):\n\n```bash\ngit ls-files | grep '\\.py' | xargs wc -l\n```\n\n**Just for *fun* 🙃**\n\n```text\nQ: What was the deaf student's favorite course?\nA: Communication skills\n```\n\n```text\nQ: Why was the ML engineer sad?\nA: Triplet loss\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eStar History\u003c/b\u003e\u003c/summary\u003e\n\n\u003ca href=\"https://star-history.com/#sign-language-translator/sign-language-translator\u0026Timeline\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://api.star-history.com/svg?repos=sign-language-translator/sign-language-translator\u0026type=Timeline\u0026theme=dark\" /\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://api.star-history.com/svg?repos=sign-language-translator/sign-language-translator\u0026type=Timeline\" /\u003e\n    \u003cimg alt=\"Star History Chart\" src=\"https://api.star-history.com/svg?repos=sign-language-translator/sign-language-translator\u0026type=Timeline\" /\u003e\n  \u003c/picture\u003e\n\u003c/a\u003e\n\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsign-language-translator%2Fsign-language-translator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsign-language-translator%2Fsign-language-translator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsign-language-translator%2Fsign-language-translator/lists"}