{"id":19815162,"url":"https://github.com/akbartus/handwriting-recognition-in-vr","last_synced_at":"2025-09-18T06:32:22.201Z","repository":{"id":174556535,"uuid":"648015241","full_name":"akbartus/Handwriting-Recognition-in-VR","owner":"akbartus","description":"The demonstration of open-source handwriting recognition in webVR environment, powered by Tensorflow.js, OpenCV.js, A-Frame and Three.js. The repo also contains 2 components for A-Frame.","archived":false,"fork":false,"pushed_at":"2023-10-17T21:26:05.000Z","size":2173,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-19T10:39:30.423Z","etag":null,"topics":["aframe-vr","handwriting-recognition","opencvjs","tensorflowjs","threejs"],"latest_commit_sha":null,"homepage":"https://handwriting-api.glitch.me/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akbartus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-06-01T02:54:58.000Z","updated_at":"2024-11-05T04:58:54.000Z","dependencies_parsed_at":"2023-10-15T00:03:22.213Z","dependency_job_id":null,"html_url":"https://github.com/akbartus/Handwriting-Recognition-in-VR","commit_stats":null,"previous_names":["akbartus/a-frame-component-handwriting-recognition","akbartus/handwriting-recognition-in-vr"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akbartus%2FHandwriting-Recognition-in-VR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akbartus%2FHandwriting-Recognition-in-VR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akbartus%2FHandwriting-Recognition-in-VR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akbartus%2FHandwriting-Recognition-in-VR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akbartus","download_url":"https://codeload.github.com/akbartus/Handwriting-Recognition-in-VR/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233456045,"owners_count":18678964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aframe-vr","handwriting-recognition","opencvjs","tensorflowjs","threejs"],"created_at":"2024-11-12T10:04:56.205Z","updated_at":"2025-09-18T06:32:16.560Z","avatar_url":"https://github.com/akbartus.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Handwriting Recognition in VR\nhttps://github.com/akbartus/Handwriting-Recognition-in-VR/assets/54757270/5067c28b-bcde-4aea-83f2-c8dfa61840a3\n\n### **Description / Rationale**\nThis repository contains various implementations of handwritten text recognition for web virtual reality. The repository was created to show the possibility of doing handwritten text recognition in web virtual reality environment based on API and without API or any server (only front end). I believe the use of handwritten text recognition can greatly enhance the experience and create some unique features. I hope this type of feature will become widely available in the near future.  \n\nThere are two types of handwritten text recognition:\n* \u003cb\u003eHandwritten text recognition based on stroke related data\u003c/b\u003e, which uses simple API access to the incredible handwriting recognition of Google IME and generates the results (i.e. API based).\n* \u003cb\u003eHandwritten text recognition based on image analysis\u003c/b\u003e, which uses ML image classification model powered by Tensorflow.js (i.e. serverless and without any API).\n\nThe first type of handwritten text recognition allows to do the recognition of text, using Google IME API, of the majority of languages of the world. Notably, this API is used for doing handwritten text conversion on \u003ca href=\"https://support.google.com/gboard/answer/9108773?hl=en\u0026co=GENIE.Platform%3DAndroid\"\u003eAndroid Devices\u003c/a\u003e. It is also used as part of INTERPLAY MODE created by Google Creative Lab, which demonstrates \u003ca href=\"https://experiments.withgoogle.com/interplay-mode/view/\"\u003ethis API usage combined with video\u003c/a\u003e.  \n \nThe second type of handwritten text recognition combines machine learning, computer vision and NLP and only recognizes English letters (A-z) and digits (0-9). Here is briefly how everything works in it: \n1. Segmentation is done using OpenCV.js, i.e. bounding box of each element based on contour in an image is calculated, then segmented and placed based on distance between bounding box x position (top left) and left corner of image. It results in several segmentations based on the total number of characters.\n2. Segmentation is then passed over to Tensorflow ML model (image segmentation task), imported and adapted from Keras model, which identifies to which class each segmented image corresponds.\n3. The text string is generated and passed over to words base, which analyzes it for correspondence and divides into meaningful words.\n4. At the end the text is displayed.\n\nThe second type of handwritten text recognition also has the following Tensorflow.js models, which are tiny and robust enough to be run on mobile devices (and therefore very suitable for web experiences):\n* Alphanumeric model (used in all examples).\n* Only letters models (16-bit and 32-bit floating-point types; see: \"serverless\" \u003e \"misc\").\n\n### **Instructions**\nThe repository contains the following: \n* A-Frame based implementation (see: \"serverless\" \u003e \"a-frame-implementation\" folder). It contains the last natural language processing (NLP) step (dividing into meaningful words).\n* Component for A-Frame (see: \"serverless\" \u003e \"a-frame-component\" folder). It does not contain the last natural language processing (NLP) step (dividing into meaningful words).\n* Simple html implementation (see \"serverless\" \u003e \"simple-implementation\" folder). It contains the last natural language processing (NLP) step (dividing into meaningful words).\n* A-Frame component with Google IME API (see \"api\" \u003e \"a-frame-component\" folder). \n\n\u003cb\u003eTo use A-Frame component (serveless one)\u003c/b\u003e, please make sure to attach the following to \u003ca-plane\u003e element: \u003cb\u003ehandwriting-recognition texture-painter id=\"drawingArea\" class=\"clickable\"\u003c/b\u003e. Below sample code is provided:\n```\n\u003chtml\u003e\n    \u003chead\u003e\n    \u003ctitle\u003eHandwriting Recognition in VR: A-Frame Demo\u003c/title\u003e\n    \u003cscript src='https://aframe.io/releases/1.4.2/aframe.min.js'\u003e\u003c/script\u003e\n    \u003cscript src=\"https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@latest/dist/tf.min.js\"\u003e\u003c/script\u003e\n\u003c/head\u003e\n    \u003cbody\u003e\n        \u003ca-scene\u003e\n           \u003ca-plane handwriting-recognition texture-painter id=\"drawingArea\" class=\"clickable\" position=\"0 1.5 -4\" rotation=\"0 0 0\" width=\"5\" height=\"4\"\u003e\u003c/a-plane\u003e\n           \u003ca-entity cursor=\"rayOrigin: mouse\" raycaster=\"objects: .clickable;\"\u003e\u003c/a-entity\u003e\n           \u003ca-entity  button-listener class=\"controller\" laser-controls=\"hand: left\" raycaster=\"objects: .clickable;\" line=\"color: #000000\"\u003e \n           \u003ca-sky color='#ECECEC'\u003e\u003c/a-sky\u003e\n        \u003c/a-scene\u003e\n        \u003cscript src=\"handwriting-recognition.js\"\u003e\u003c/script\u003e\n    \u003c/body\u003e\n\u003c/html\u003e\n```\n\u003cb\u003ePlease note:\u003c/b\u003e This A-Frame component is attached after a-scene element. It does not have recognize and clear buttons for mouse clicks. It only supports VR mode with controllers. A-Frame implementation and component also support Quest 2 buttons: \u003cb\u003ebutton X - recognize, button Y - clear.\u003c/b\u003e\n\n\u003cb\u003eSample usage of A-Frame component (with API)\u003c/b\u003e is provided below:\n```\n\u003c!DOCTYPE html\u003e\n\u003chtml\u003e\n\u003chead\u003e\n    \u003ctitle\u003eHandwriting Recognition in VR: A-Frame Component with API\u003c/title\u003e\n     \u003cmeta charset=\"UTF-8\"\u003e\n    \u003cmeta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\"\u003e\n    \u003cmeta name=\"viewport\" content=\"width=device-width, initial-scale=1\"\u003e\n    \u003cscript src='https://aframe.io/releases/1.4.0/aframe.min.js'\u003e\u003c/script\u003e\n    \u003cscript src=\"https://unpkg.com/aframe-troika-text/dist/aframe-troika-text.min.js\"\u003e\u003c/script\u003e\n\u003c/head\u003e\n\u003cbody\u003e\n    \u003ca-scene\u003e\n        \u003ca-plane id=\"drawingArea\" class=\"clickable\" handwriting-recognition-api=\"handwritingLanguage: en; size: 20\" position=\"0 1.5 -5\" rotation=\"0 0 0\" width=\"5\" height=\"4\"\u003e\u003c/a-plane\u003e\n        \n        \u003ca-entity id=\"outputText\" position=\"0 0.2 -4\" \n        troika-text=\"value: Console; color: black\"\u003e\u003c/a-entity\u003e\n        \u003ca-entity id=\"send\" text=\"value: Send; align: center; width: 3;\" position=\"-2.5 0.2 -4\" class=\"clickable\" geometry=\"primitive: plane; height: 0.3\" material=\"color: black\"\u003e\n        \u003c/a-entity\u003e\n        \u003ca-entity id=\"clear\" text=\"value: Clear; align: center; width: 3;\" position=\"2.5 0.2 -4\" class=\"clickable\" geometry=\"primitive: plane; height: 0.3\" material=\"color: black\"\u003e\n        \u003c/a-entity\u003e\n       \n        \u003ca-entity cursor=\"rayOrigin: mouse\" raycaster=\"objects: .clickable;\"\u003e\u003c/a-entity\u003e\n        \u003ca-entity  class=\"controller\" laser-controls=\"hand: left\" raycaster=\"objects: .clickable;\" line=\"color: #000000\"\u003e\u003c/a-entity\u003e \n        \u003ca-sky color=\"#ECECEC\" rotation=\"0 -90 0\"\u003e\u003c/a-sky\u003e\n    \u003c/a-scene\u003e\n    \u003cscript src='handwriting-recognition-api.js'\u003e\u003c/script\u003e\n\n\u003c/body\u003e\n\n\u003c/html\u003e\n```\nIt has the following attributes/schemas:\n* color: { type: \"color\", default: \"black\" } - Color of stroke.\n* size: { type: \"int\", default: 20 } - Size of stroke\n* background: { type: \"color\", default: \"white\" } - Plane background color. \n* clearAll: { type: \"boolean\", default: false } - Whether clearAll is enabled.\n* language: { type: \"string\", default: \"en\"} = Language in which handwrtitten text should be recognized.\n\n\u003cb\u003ePlease note:\u003c/b\u003e In this example we are using troika text component, which allows to show text in other languages.\n\n### **Language Codes**\nThe following is a list of language codes, which can be used with A-Frame component using Google IME API:\n| Language              | code  |\n|-----------------------|-------|\n| Afrikaans             | af    |\n| Albanian              | sq    |\n| Basque                | eu    |\n| Belarusian            | be    |\n| Bulgarian             | bg    |\n| Catalan               | ca    |\n| Chinese (Simplified)  | zh_CN |\n| Chinese (Traditional) | zh_TW |\n| Croatian              | hr    |\n| Czech                 | cs    |\n| Danish                | da    |\n| Dutch                 | nl    |\n| English               | en    |\n| Estonian              | et    |\n| Filipino              | fil   |\n| Finnish               | fi    |\n| French                | fr    |\n| Galician              | gl    |\n| German                | de    |\n| Greek                 | el    |\n| Haitian               | ht    |\n| Hindi                 | hi    |\n| Hungarian             | hu    |\n| Icelandic             | is    |\n| Indonesian            | id    |\n| Irish                 | ga    |\n| Italian               | it    |\n| Japanese              | ja    |\n| Korean                | ko    |\n| Latin                 | la    |\n| Latvian               | lv    |\n| Lithuanian            | lt    |\n| Macedonian            | mk    |\n| Malay                 | ms    |\n| Norwegian             | no    |\n| Polish                | pl    |\n| Portuguese (Brazil)   | pt_BR |\n| Portuguese (Portugal) | pt_PT |\n| Romanian              | ro    |\n| Russian               | ru    |\n| Serbian               | sr    |\n| Slovak                | sk    |\n| Slovenian             | sl    |\n| Spanish               | es    |\n| Swahili               | sw    |\n| Swedish               | sv    |\n| Thai                  | th    |\n| Turkish               | tr    |\n| Ukranian              | yk    |\n| Vietnamese            | vi    |\n| Welsh                 | cy    |\n\n### **Updates**\nIt is definitely possible to add other ML language models and therefore do handwriting recognition in that language. Soon will add new language model. In addition, will be providing small tutorial on how to train own model.\n\n### **Tech Stack**\nHandwritten text recognition is powered by AFrame, Three.js and OpenCV.js and Tensorflow.js. It uses updated/modified \u003ca href=\"https://github.com/marlon360/whiteboard-vr\"\u003etexture painter component\u003c/a\u003e, which is part of Whiteboard VR by Marlon Lückert. The code related to API was developed based on the example provided in Chen Yu Ho's \u003ca href=\"https://github.com/ChenYuHo/handwriting.js\"\u003eHandwriting.js repository\u003c/a\u003e, and Amit Agarwal's blog post \u003ca href=\"https://www.labnol.org/code/19205-google-handwriting-api\"\u003e\"Google Handwriting IME API Request\"\u003c/a\u003e. It should be noted though there is very little information on the use of this IME API!   \n\nTo learn more about OpenCV.js and its various uses, please refer to: https://github.com/akbartus/OpenCV-Examples-in-JavaScript.\nTo see another creative use of drawing in web VR, please refer to: https://github.com/akbartus/VR-Doodle-Painter.  \nTo see handsfree handwriting recognition, using similar functionality refer to: https://github.com/akbartus/Web-Based-Touchfree-Handwriting-Recognition \n\n### **Demo**\nThe repository contains the following implementations/demos:\n* Serverless:\n  - \u003ca href=\"https://handwriting-vr.glitch.me/\"\u003eA-Frame implementation\u003c/a\u003e.\n  - \u003ca href=\"https://handwriting-component.glitch.me/\"\u003eA-Frame component, serveless\u003c/a\u003e.\n  - \u003ca href=\"https://handwriting-simple.glitch.me/\"\u003eSimple html demo\u003c/a\u003e.\n* API based:\n  - \u003ca href=\"https://handwriting-api.glitch.me/\"\u003eA-Frame Component, API\u003c/a\u003e.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakbartus%2Fhandwriting-recognition-in-vr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakbartus%2Fhandwriting-recognition-in-vr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakbartus%2Fhandwriting-recognition-in-vr/lists"}