{"id":13732708,"url":"https://github.com/MahmudulAlam/Fingertip-Mixed-Reality","last_synced_at":"2025-05-08T09:31:10.098Z","repository":{"id":117898809,"uuid":"178904370","full_name":"MahmudulAlam/Fingertip-Mixed-Reality","owner":"MahmudulAlam","description":"Affine transformation virtual 3D object using a finger gesture-based interactive system in the virtual environment.","archived":false,"fork":false,"pushed_at":"2023-10-21T10:33:03.000Z","size":389479,"stargazers_count":33,"open_issues_count":1,"forks_count":13,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-08-04T03:02:14.582Z","etag":null,"topics":["affine-transformation","finger-gesture","fingertip-detection","fingertip-position","hand-detection","mixed-reality","unity","virtual-object","virtual-reality"],"latest_commit_sha":null,"homepage":"https://www.sciencedirect.com/science/article/pii/S2096579620300917?via%3Dihub","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MahmudulAlam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-04-01T16:33:38.000Z","updated_at":"2024-07-24T03:10:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"94171010-2008-4cd6-a638-c120c70e2cb5","html_url":"https://github.com/MahmudulAlam/Fingertip-Mixed-Reality","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MahmudulAlam%2FFingertip-Mixed-Reality","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MahmudulAlam%2FFingertip-Mixed-Reality/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MahmudulAlam%2FFingertip-Mixed-Reality/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MahmudulAlam%2FFingertip-Mixed-Reality/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MahmudulAlam","download_url":"https://codeload.github.com/MahmudulAlam/Fingertip-Mixed-Reality/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224717457,"owners_count":17357884,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["affine-transformation","finger-gesture","fingertip-detection","fingertip-position","hand-detection","mixed-reality","unity","virtual-object","virtual-reality"],"created_at":"2024-08-03T03:00:32.608Z","updated_at":"2025-05-08T09:31:10.090Z","avatar_url":"https://github.com/MahmudulAlam.png","language":"Python","funding_links":[],"categories":["Unity Resources"],"sub_categories":[],"readme":"\u003ch2 align=\"center\"\u003eAffine Transformation of Virtual Object\u003c/h2\u003e\n\n\u003cp align=\"justify\"\u003e\nA convolutional neural network (CNN) based thumb and index fingertip detection system are presented here for seamless interaction with a virtual 3D object in the virtual environment. First, a two-stage CNN is employed to detect the hand and fingertips, and using the information of the fingertip position, the scale, rotation, translation, and in general, the affine transformation of the virtual object is performed.\n\u003c/p\u003e\n\n## Update \nThis is the version ```2.0``` that includes a more generalized affine transformation of virtual objects in the virtual environment with more experimentation and analysis. Previous versions only include the geometric transformation of a virtual 3D object with respect to a finger gesture. To get the previous version visit [here](https://github.com/MahmudulAlam/Fingertip-Mixed-Reality/releases).\n\n[![GitHub stars](https://img.shields.io/github/stars/MahmudulAlam/Fingertip-Mixed-Reality)](https://github.com/MahmudulAlam/Fingertip-Mixed-Reality/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/MahmudulAlam/Fingertip-Mixed-Reality)](https://github.com/MahmudulAlam/Fingertip-Mixed-Reality/network)\n[![Downloads](https://img.shields.io/badge/version-2.0-orange.svg?longCache=true\u0026style=flat)](https://github.com/MahmudulAlam/Fingertip-Mixed-Reality)\n[![GitHub license](https://img.shields.io/github/license/MahmudulAlam/Fingertip-Mixed-Reality)](https://github.com/MahmudulAlam/Fingertip-Mixed-Reality/blob/master/LICENSE)\n\n## Paper\nPaper for the affine transformation of the virtual 3D object has been published in Virtual Reality \u0026 Intelligent Hardware, Elsevier Science Publishers in 2020. To get more detail, please go through the [```paper```](https://doi.org/10.1016/j.vrih.2020.10.001). Paper for the geometric transformation of the virtual object [```v1.0```](https://github.com/MahmudulAlam/Fingertip-Mixed-Reality/releases/tag/v1.0) has also been published. For more detail, please go through this [```paper```](https://ieeexplore.ieee.org/abstract/document/9035256). If you use the code or data from the project, please cite the following papers: \n\n[![Paper](https://img.shields.io/badge/paper-ScienceDirect-orange.svg?longCache=true\u0026style=flat)](https://doi.org/10.1016/j.vrih.2020.10.001)\n![](https://img.shields.io/badge/-v2.0-brightgreen)\n\n[***```Affine transformation of virtual 3D object using 2D localization of fingertips```***](https://www.sciencedirect.com/science/article/pii/S2096579620300917?via%3Dihub) 🔗\n\n```bibtex\n@article{alam2020affine,\n  title={Affine transformation of virtual 3D object using 2D localization of fingertips},\n  author={Alam, Mohammad Mahmudul and Rahman, SM Mahbubur},\n  journal={Virtual Reality \\\u0026 Intelligent Hardware},\n  volume={2},\n  number={6},\n  pages={534--555},\n  year={2020},\n  publisher={Elsevier}\n}\n```\n\n[![Paper](https://img.shields.io/badge/paper-IeeeXplore-blue.svg?longCache=true\u0026style=flat)](https://ieeexplore.ieee.org/abstract/document/9035256)\n![](https://img.shields.io/badge/-v1.0-brightgreen)\n\n[***```Detection and Tracking of Fingertips for Geometric Transformation of Objects in Virtual Environment```***](https://ieeexplore.ieee.org/abstract/document/9035256) 🔗\n\n```bibtex\n@inproceedings{alam2019detection,\n  title={Detection and Tracking of Fingertips for Geometric Transformation of Objects in Virtual Environment},\n  author={Alam, Mohammad Mahmudul and Rahman, SM Mahbubur},\n  booktitle={2019 IEEE/ACS 16th International Conference on Computer Systems and Applications (AICCSA)},\n  address = {Abu Dhabi, United Arab Emirates},\n  pages={1--8},\n  year={2019},\n  organization={IEEE}\n}\n```\n\n## System Overview\nHere it the real-time demo of the scale, rotation, translation, and overall affine transformation of the virtual object using finger \ninteraction.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/37298971/78501859-96a26b00-777f-11ea-9f33-977ea8feda09.gif\" width=\"640\"\u003e\n\u003c/p\u003e\n\n| \u003cimg src=\"https://user-images.githubusercontent.com/37298971/82154840-6d9eeb00-9892-11ea-986d-6314a21d5283.gif\"\u003e  | \u003cimg src=\"https://user-images.githubusercontent.com/37298971/82154849-7abbda00-9892-11ea-84e9-4e1b5508c1e8.gif\"\u003e  | \u003cimg src=\"https://user-images.githubusercontent.com/37298971/82154850-81e2e800-9892-11ea-8d7f-7d6830978a49.gif\"\u003e  |\n|:-:|:-:|:-:|\n\n## Dataset\nTo train the hand and fingertip detection model two different datasets are used. One is a self-made publicly released dataset called [TI1K Dataset](https://github.com/MahmudulAlam/TI1K-Dataset) which contains 1000 images with the annotations of hand and fingertip position and another one is [Scut-Ego-Gesture Dataset](http://www.hcii-lab.net/data/SCUTEgoGesture/index.htm). \n\n## Requirements\n- [x] TensorFlow-GPU==1.15.0\n- [x] OpenCV==4.2.0\n- [x] Cython==0.29.2\n- [x] ImgAug==0.2.6\n- [x] Weights: [```download```](https://mega.nz/folder/SkklxRAA#Z0p60mPe1BwJ7ZTZniMtDA) the trained weights file for both hand and fingertip detection model and put the weights folder in the working directory. \n\n[![Downloads](https://img.shields.io/badge/download-weights-blue.svg?style=popout-flat\u0026logo=mega)](https://mega.nz/folder/SkklxRAA#Z0p60mPe1BwJ7ZTZniMtDA)\n\n## Experimental Setup\nThe experimental setup has a server and client-side. Fingertip detection and tracking and all other machine learning stuff are programmed in the server-side using Python. On the client-side, the virtual environment is created using Unity along with the Vuforia software development kit (SDK). To locate and track a virtual object using the webcam, Vuforia needs marker assistance. For that purpose, a marker is designed which works as an image target. The [```marker/```](https://github.com/MahmudulAlam/Fingertip-Mixed-Reality/tree/master/marker) folder contains the pdf of the designed marker. To use the system print a copy of the marker.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/37298971/57572552-c9dc5d00-743d-11e9-9f1c-fcf9d97517b5.jpg\" width=\"800\"\u003e\n\u003c/p\u003e\n\n## How to Use\nFirst, to run the server-side directly run ```'server.py'```. It will wait until the client-side (Unity) is starting to send images to the server. \n```\ndirectory \u003e python server.py\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/37298971/57572729-97802f00-7440-11e9-9c5c-fa4d6427d8ec.png\" width=\"700\"\u003e\n\u003c/p\u003e\n\nOpen the ```'Unity Affine Transformation'``` environment using [```Unity```](https://unity3d.com/get-unity/download) and hit the play button. Make sure a webcam is connected. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/37298971/57572793-5b010300-7441-11e9-96cb-07bec8f818b1.png\" width=\"700\"\u003e\n\u003c/p\u003e\n\nBring your hand in front of the webcam and interact with the virtual object using your finger gesture.\n\n\u003c!---\n## Demo\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/37298971/60720908-07e19300-9f4e-11e9-8e38-f322d81a9abd.gif\" width=\"700\"\u003e\n\u003c/p\u003e\n--\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMahmudulAlam%2FFingertip-Mixed-Reality","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMahmudulAlam%2FFingertip-Mixed-Reality","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMahmudulAlam%2FFingertip-Mixed-Reality/lists"}