{"id":35062157,"url":"https://github.com/mnvoh/cameratokeyboard","last_synced_at":"2026-03-17T07:39:29.339Z","repository":{"id":227954055,"uuid":"772617457","full_name":"mnvoh/cameratokeyboard","owner":"mnvoh","description":"Camera to keyboard:  A computer vision typing experiment that allows you to use your camera as a keyboard (early stage).","archived":false,"fork":false,"pushed_at":"2024-04-03T11:36:42.000Z","size":39883,"stargazers_count":37,"open_issues_count":0,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-04T08:39:19.244Z","etag":null,"topics":["computer-vision","machine-learning","python","virtual-hardware","virtual-keyboard"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mnvoh.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE-dataset.md","code_of_conduct":"docs/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"docs/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-15T14:49:06.000Z","updated_at":"2025-11-20T16:11:33.000Z","dependencies_parsed_at":"2024-03-22T15:32:35.123Z","dependency_job_id":"6e0984eb-36a1-4606-96f4-699f3aa2c537","html_url":"https://github.com/mnvoh/cameratokeyboard","commit_stats":null,"previous_names":["mnvoh/cameratokeyboard"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/mnvoh/cameratokeyboard","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnvoh%2Fcameratokeyboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnvoh%2Fcameratokeyboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnvoh%2Fcameratokeyboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnvoh%2Fcameratokeyboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mnvoh","download_url":"https://codeload.github.com/mnvoh/cameratokeyboard/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnvoh%2Fcameratokeyboard/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30617552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-17T04:46:40.957Z","status":"ssl_error","status_checked_at":"2026-03-17T04:46:32.538Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","machine-learning","python","virtual-hardware","virtual-keyboard"],"created_at":"2025-12-27T10:50:01.436Z","updated_at":"2026-03-17T07:39:29.334Z","avatar_url":"https://github.com/mnvoh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003cbr\u003e\n  \u003ca href=\"#\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/mnvoh/cameratokeyboard/main/docs/images/c2k.png\" alt=\"Camera to Keyboard\" width=\"200\"\u003e\u003c/a\u003e\n  \u003cbr\u003e\n  Camera to Keyboard\n  \u003cbr\u003e\n\n  ![Python Version](https://img.shields.io/badge/Python-3.8..3.11-blue)\n  ![PyPI - Version](https://img.shields.io/pypi/v/c2k)\n  [![Tests](https://img.shields.io/github/actions/workflow/status/mnvoh/cameratokeyboard/test.yml)](https://github.com/mnvoh/cameratokeyboard/actions)\n  ![Codecov](https://codecov.io/gh/mnvoh/cameratokeyboard/graph/badge.svg?token=Q7P8AT45OF)\n  [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)\n\u003c/h1\u003e\n\n\n⭐ Star it, make it visible, make it better!\n\nCamera To Keyboard is a python desktop application that allows you to type without a keyboard\nand with just a camera, or at least that's the plan. \n\n\u003e [!NOTE]\n\u003e Currently the accuracy leaves much to be desired, so it's a PoC and a long way from\n\u003e having optimal accuracy. But hopefully with your support it'll get there.\n\nSome of its possible applications, if desirable accuracy is achieved include:\n- In cell phones, just put your phone down in front of you and type\n- in VR/AR\n- Or print your own keyboard design on a big mouse mat!\n\n\u003cvideo src=\"https://github.com/mnvoh/cameratokeyboard/assets/4628766/c5610879-7e9c-4070-bf76-b513c6b1cb86\" controls\u003e\u003c/video\u003e\n\n## Table of Content\n\n\n- [Installation](#installation)\n- [Training](#training)\n- [App Guide](#app-guide)\n- [Usage](#usage)\n- [Contributing](#contributing)\n- [Roadmap](#roadmap)\n- [Security](#security)\n- [License](#license)\n- [Authors \u0026 Contributors](#authors--contributors)\n- [Support](#support)\n\n## Installation\n\n\u003e [!IMPORTANT] \n\u003e To use GPU, you have to install compatible versions of CUDA and torch.\n\u003e\n\u003e Check out [Get Started Locally](https://pytorch.org/get-started/locally/) for more information.\n\n### Using PIP\n\nRun the following command to install the package from PyPI:\n\n```bash\npython -m pip install --upgrade c2k\n```\n\nRun the project with:\n\n```bash\npython -m c2k\n```\n\n### Using Git\n\n1. Clone the repository\n\n```bash\ngit clone https://github.com/mnvoh/cameratokeyboard.git\n```\n\n2. Create a virtualenv and install the requirements\n\n```bash\ncd cameratokeyboard\npython -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\n3. Run the app. If you have more than one video input device, specify it:\n\n```bash\npython c2k.py -i 1 # or 0, or 2, or ...\n```\n\n## Training\n\n⭐ The model is trained using ultralytics and YOLOv8.\n\nIf you have added additional images to the dataset or just want to retrain the model\nwith different parameters, here's how you do it. But before jumping in, I should mention\n2 things:\n\nFirst, all the modules in `cameratokeyboard/model` are deprecated. One of the very first\nsteps in the roadmap is to establish a descent pipeline (most probably Roboflow).\n\nSecondly, here's how the training works:\n\n1. The images are picked up from `\u003cROOT\u003e/raw_dataset`. This directory has a flat 1-level\nstructure containing all `\u003cname\u003e.jpg` images and `\u003cname\u003e.txt` labels.\n2. The images are paritioned into `train`, `test` and `val` splits. You can customize\nthe ratios with arguments (See Usage)\n3. The images in the train split are augmented. The augmentation strategies are defined\nin `cameratokeyboard/config.py`. You can mix and match the implemented augmenters.\n\nHaving said all that, here's how to train the model\n\n1. Train it:\n\n```bash\npython c2k.py train\n```\n\nThat's it, no second steps. The newly trained model (`best.pt`) is moved to \n`cameratokeyboard/model.pt`, replacing the existing one. \n\n## App Guide\n\nThis app relies on 4 markers (aka control points) to determine the boundries of the keyboard.\nSo first off, print the following image on paper. Print on an A4 paper, it's the same size\nas an actual keyboard.\n\n\u003e [!TIP]\n\u003e The algorithm only needs the markers and everything else is irrelevant to it, you could \n\u003e even just draw the markers yoruself using a marker! (pun probably intended, undecided as of now).\n\u003e But you, as a user, are most probably gonna need to know where the keys are.\n\n![Keyboard Print](https://raw.githubusercontent.com/mnvoh/cameratokeyboard/main/docs/images/keyboard.jpg)\n\nAt the moment, the interface is really simple, just a text box and a preview of what's \ngoing on. \n\n![Camera to Keyboard UI](https://raw.githubusercontent.com/mnvoh/cameratokeyboard/main/docs/images/screenshot.png)\n\n#### The Text Box\n\nWhatever you type will show up here.\n\n#### The Preview\n\nShows the current frame along with the detections made. Some important things to note:\n\n1. At the top left corner you'll see your camera's yaw and pitch angles. Yaw is pretty \naccurate (or at least accurate enough), try to keep that as close to 0 as possible. Pitch\nhowever doesn't seem to be accurate at all (working on fixing this as well). So try to adjust \nyour camera's pitch to get the best results.\n\n2. There's a color coded border around the preview.\n\n- Gray ⬛ (let's pretend this is gray): Initializing, most probably you'll never see this.\n- Red 🟥: Some or all markers are missing.\n- Orange 🟧: Some or all fingers are missing.\n- Yellow 🟨: One or both thumbs are missing.\n- Green 🟩: Everything's A-OK. All objects have been successfully detected.\n\n3. And finally the detected fingers and markers\n\n## Usage\n\n\u003e [!IMPORTANT]\n\u003e If you are using WSL, you cannot access your webcam in that environment. A\n\u003e workaround is to use this [Kernel](https://github.com/PINTO0309/wsl2_linux_kernel_usbcam_enable_conf)\n\u003e but I have not used it and will not take responsibility for any consequences if you decide to\n\u003e use it. Just run the app in Powershell, ez pz.\n\n```\nusage: c2k.py [-h] [-e TRAINING_EPOCHS] [-ts WIDTH HEIGHT] [-b TRAINING_BATCH] [-rp RAW_DATASET_PATH]\n              [-dp DATASET_PATH] [-sp TRAIN TEST VAL] [-sr TRAIN TEST VAL] [-ie IMAGE_EXTENSION] [-p MODEL_PATH]\n              [-r WIDTH HEIGHT] [-f APP_FPS] [-i VIDEO_INPUT_DEVICE] [-d PROCESSING_DEVICE]\n              [-mc MARKERS_MIN_CONFIDENCE] [-fc FINGERS_MIN_CONFIDENCE] [-tc THUMBS_MIN_CONFIDENCE]\n              [-s KEY_DOWN_SENSITIVITY] [-rd REPEATING_KEYS_DELAY]\n              [{train}]\n\nCamera To Keyboard\n\npositional arguments:\n  {train}               Specify the command to execute. Omit to run the main app.\n\noptions:\n  -h, --help            show this help message and exit\n  -e TRAINING_EPOCHS, --training_epochs TRAINING_EPOCHS\n                        Specify the number of training epochs. Default: 40\n  -ts WIDTH HEIGHT, --training_image_size WIDTH HEIGHT\n                        The dimensions of the training images. Default: 640 640\n  -b TRAINING_BATCH, --training_batch TRAINING_BATCH\n                        The batch size for training. Default: -1 (automatic)\n  -rp RAW_DATASET_PATH, --raw_dataset_path RAW_DATASET_PATH\n                        The path to the raw dataset. Default: raw_dataset\n  -dp DATASET_PATH, --dataset_path DATASET_PATH\n                        The path to the partitioned and augmented dataset. Default: ../datasets/c2k\n  -sp TRAIN TEST VAL, --split_paths TRAIN TEST VAL\n                        The paths to the train, test and validation datasets. Default: train test val\n  -sr TRAIN TEST VAL, --split_ratios TRAIN TEST VAL\n                        The ratios for the train, test and validation datasets. Default: 0.7 0.15 0.15\n  -ie IMAGE_EXTENSION, --image_extension IMAGE_EXTENSION\n                        The extension of the images in the dataset. Default: jpg\n  -p MODEL_PATH, --model_path MODEL_PATH\n                        The path to the model. Default: cameratokeyboard/model.pt\n  -r WIDTH HEIGHT, --resolution WIDTH HEIGHT\n                        The resolution of the images taken from the camera. Default: 1280 720\n  -f APP_FPS, --app_fps APP_FPS\n                        The refresh rate of the app. Default: 30\n  -i VIDEO_INPUT_DEVICE, --video_input_device VIDEO_INPUT_DEVICE\n                        The device number of the input camera. Default: 0\n  -d PROCESSING_DEVICE, --processing_device PROCESSING_DEVICE\n                        The device index to use for training and inference or enter 'cpu' to use CPU. Default: 0.\n  -mc MARKERS_MIN_CONFIDENCE, --markers_min_confidence MARKERS_MIN_CONFIDENCE\n                        The minimum confidence for the markers. Default: 0.3\n  -fc FINGERS_MIN_CONFIDENCE, --fingers_min_confidence FINGERS_MIN_CONFIDENCE\n                        The minimum confidence for the fingers. Default: 0.3\n  -tc THUMBS_MIN_CONFIDENCE, --thumbs_min_confidence THUMBS_MIN_CONFIDENCE\n                        The minimum confidence for the thumbs. Default: 0.3\n  -s KEY_DOWN_SENSITIVITY, --key_down_sensitivity KEY_DOWN_SENSITIVITY\n                        The sensitivity for the key down action. Default: 0.75\n  -rd REPEATING_KEYS_DELAY, --repeating_keys_delay REPEATING_KEYS_DELAY\n                        The delay for repeating keys. Default: 0.4\n```\n\n## Contributing\n\nPlease visit [CONTRIBUTING](docs/CONTRIBUTING.md) for more info.\n\n## Roadmap\n\n- [x] **Improve the tests**\n- [x] Establish an actual ML pipeline\n- [ ] Improve the detection accuracy of down fingers\n- [ ] Improve the dataset\n- [ ] Implement the recognition of modifier key presses\n- [ ] Train the model to recognize hand gestures for:\n    - [ ] Running calibration\n    - [ ] Resetting the marker positions\n- [ ] Once a desirable accuracy has been obtained, implement a virtual keyboard driver or\n    accessibility APIs (basically by any means necessary)\n- [ ] Create a settings view/page to make app settings (currently `config.py`) modifiable\n    at runtime.\n\n## Security\n\nCamera to Keyboard follows good security practices, but 100% security cannot be assured.\nCamera to Keyboard is provided **\"as is\"** without any **warranty**. Use at your own risk.\n\n_For more information and to report security issues, please refer to our [security documentation](docs/SECURITY.md)._\n\n## License\n\nThis project's code is licensed under the **GNU General Public License v3**.\nSee [LICENSE](LICENSE.md) for more information.\n\nThe dataset and its derivatives (such as the trained model) are licensed under **CC BY-NC-SA 4.0**. See [LICENSE-dataset](LICENSE-dataset.md)\n\n## Authors \u0026 contributors\n\nThe original setup of this repository is by [mnvoh](https://github.com/mnvoh).\n\nFor a full list of all authors and contributors, see [the contributors page](https://github.com/mnvoh/cameratokeyboard/contributors).\n\n## Support\n\nPlease feel free to use any of these channels to ask for help or report bugs and security\nvulnerabilities:\n\n1. Github Issues to report issues, bugs, feature requests, etc\n2. Github Discussions to ask questions, troubleshoot your installation, ...\n3. Discord Server\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmnvoh%2Fcameratokeyboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmnvoh%2Fcameratokeyboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmnvoh%2Fcameratokeyboard/lists"}