{"id":29296509,"url":"https://github.com/xverse-engine/xvrfacetracking","last_synced_at":"2025-07-25T02:35:09.204Z","repository":{"id":302867723,"uuid":"983969034","full_name":"xverse-engine/XVRFaceTracking","owner":"xverse-engine","description":"A VRCFaceTracking(VRCFT) based plugin aiming to provide real-time lower-face tracking solution for any VR headsets.","archived":false,"fork":false,"pushed_at":"2025-05-26T02:52:42.000Z","size":77207,"stargazers_count":8,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-07-04T14:31:58.093Z","etag":null,"topics":["computer-vision","vrcfacetracking","vrchat"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xverse-engine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-15T07:36:52.000Z","updated_at":"2025-06-12T21:14:44.000Z","dependencies_parsed_at":"2025-07-04T14:42:06.087Z","dependency_job_id":null,"html_url":"https://github.com/xverse-engine/XVRFaceTracking","commit_stats":null,"previous_names":["xverse-engine/xvrfacetracking"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/xverse-engine/XVRFaceTracking","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xverse-engine%2FXVRFaceTracking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xverse-engine%2FXVRFaceTracking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xverse-engine%2FXVRFaceTracking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xverse-engine%2FXVRFaceTracking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xverse-engine","download_url":"https://codeload.github.com/xverse-engine/XVRFaceTracking/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xverse-engine%2FXVRFaceTracking/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266944694,"owners_count":24010488,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-25T02:00:09.625Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","vrcfacetracking","vrchat"],"created_at":"2025-07-06T16:00:35.241Z","updated_at":"2025-07-25T02:35:09.195Z","avatar_url":"https://github.com/xverse-engine.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Xverse VRFace Tracking\n\n\u003ca href=\"./LICENSE\"\u003e\n        \u003cimg alt=\"License\" src=\"https://img.shields.io/badge/License-Apache_2.0-blue.svg\"\u003e\u003c/a\u003e\n\n\n[\n  \u003cimg src=\"images/XVERSE.jpg\" width=\"800\" /\u003e\n](http://xverse.cn/)\n\n\n\n# Introduction\n\n Xverse VRFace Tracking is a lower face tracking solution developed by XVERSE Technology Inc. (Shenzhen, China), aiming to provide real-time face tracking with any VR headset in different lighting. We have trained a new deep learning model to better detect facial expression changes. Currently, our face tracking project is implemented based on VRface and VRCFaceTracking，leveraging the simplicity and convenience of VRCFT.\n\nWe will actively release new features in this repo, please stay tuned. Some future updates will contain:\n- [ ] Enhance Facial Expression Prediction\n- [ ] Inplement Tongue Expression Prediction\n- [ ] Extend Hardware Support for VR Devices\n\n      \nBelow are examples of our method applied to different VRFace avatars.\n\n\n  \u003cimg src=\"images/VRface_demo3.gif\" width=\"800\" /\u003e\n  \u003cimg src=\"images/VRface_demo4.gif\" width=\"800\" /\u003e\n\n\n# Getting Started\n\nYou can make your own camera hardware following [Hardware](#hardware) or contact us via email xengine@xverse.cn to purchase finished products. Below are visuals of the camera’s appearance, installation, and how it looks in use.\n\n\n \u003cimg src=\"images/camera.jpg\" width=\"800\" /\u003e\n \n \u003cimg src=\"images/hardware_install.gif\" width=\"800\" /\u003e\n \n \u003cimg src=\"images/install_demo.png\" width=\"800\" /\u003e\n\n\n## System Requirements\n- Windows 10 or 11\n- Python 3.12\n- VRface and VRCFaceTracking\n\n\n\n\n\n## Software Install \n\nThe file structure of this project is as below:\n```\n├── software/                        \n│   ├── CameraIPReciver.py              # camera IP reciver\n│   └── XverseVRfaceMouthDetectionUI.py # lower face detection UI\n├── CameraWebServer/                    # Camera web service\n│   ├── app_httpd.cpp                   # HTTP server implementation\n│   ├── CameraWebServer.ino             # Arduino main sketch\n│   ├── camera_index.h                  # Web interface template\n│   ├── camera_pins.h                   # Hardware pin definitions\n│   ├── ci.json                         # CI configuration\n│   ├── partitions.csv                  # ESP32 partition table\n│   ├── README.md                       # README doc\n│   ├── remote_post.cpp                 # Remote communication module\n│   └── remote_post.h                   # Communication header\n├── assets/                             # plugin folder\n│   ├── VRCFaceTracking.Xverse.dll      # VRCFT plugin\n│   └── XverseConfig.json               # plugin json\n├── images/                             # pictures for readme\n│   └── ...\n├── models/                             # onnx model\n│   └── XVRFaceTracking.onnx            # onnx model for lower face tracking         \n├── 3D_hardware/                        # hardware files for 3D print \n│   ├── hardware-led.blend              # blend file for 3D print\n│   └── hardware-led.stl                # stl file for 3D print\n├── requirements.txt                    # Python requriements\n├── README.md                           \n├── LICENSE                          \n\n```\n\n\n0.Install the dependency packages listed in [**requriements.txt**](https://github.com/xverse-engine/XVRFaceTracking/blob/main/requirements.txt).\n\n```\npip install -r requirements.txt\n```\n\n 1.Install [**ardunio**](https://www.arduino.cc/en/software/). \n Connect ESP32S3 to your PC with a micro-USB/mini-USB/USB type-c cable. Open [**CameraWebServer.ino**](https://github.com/xverse-engine/XVRFaceTracking/blob/main/CameraWebServer/CameraWebServer.ino)       in [**CameraWebServer**](https://github.com/xverse-engine/XVRFaceTracking/blob/main/CameraWebServer/) via ardunio. Change board type to XIAO_ESP32S3 and fill your WiFi network name and password.\n\n \u003cimg src=\"images/camerawebserverino.png\" width=\"800\" /\u003e\n\n\n Open [**remote_post.cpp**](https://github.com/xverse-engine/XVRFaceTracking/blob/main/CameraWebServer/remote_post.cpp)       in [**CameraWebServer**](https://github.com/xverse-engine/XVRFaceTracking/blob/main/CameraWebServer/). Fill in your PC IP, verify and upload.\n\n \u003cimg src=\"images/remotepostcpp.png\" width=\"800\" /\u003e\n \n You can run 'ipconfig' in terminal to know your PC IP. \n \n\u003cimg src=\"images/ipconfig.png\" width=\"800\" /\u003e\n\nAfter successful upload, the arduino will display the notification shown in the figure below.\n\n\n\u003cimg src=\"images/ardi_sucess.png\" width=\"800\" /\u003e\n \nAfter uploading [**CameraWebServer**](https://github.com/xverse-engine/XVRFaceTracking/blob/main/CameraWebServer/) to ESP32S3, run  [**CameraIPReciver.py**](https://github.com/xverse-engine/XVRFaceTracking/blob/main/software/CameraIPReciver.py) to recieve Stream URL. Ensure that ports 8889 on your PC is not blocked or already in use.\n\n\n\n\u003cimg src=\"images/cameraIPreciever.png\" width=\"800\" /\u003e\n\n\n\n\n\n\n 2.Install [**VRCFaceTracking**](https://github.com/benaclejames/VRCFaceTracking). \n Drop the VRCFaceTracking.Xverse.dll and XverseConfig.json into AppData\\Roaming\\VRCFaceTracking\\CustomLibs. If you can't find this path, you can use [**Everything**](https://www.voidtools.com/zh-cn/) for search. If this folder does not exist you can create it, VRCFaceTracking will create it on launch.\n\n\n  \u003cimg src=\"images/CustomLibs.png\" width=\"800\" /\u003e\n\n\n\n\n 3.Run [**XverseVRfaceMouthDetectionUI.py**](https://github.com/xverse-engine/XVRFaceTracking/blob/main/software/XverseVRfaceMouthDetectionUI.py). \n Stream Url is from [**CameraIPReciver.py**](https://github.com/xverse-engine/XVRFaceTracking/blob/main/software/CameraIPReciver.py)(step 1). Remeber to keep 'http://' in Stream Url. ONNX file is our pretrained deep learning model in folder [**model**](https://github.com/xverse-engine/XVRFaceTracking/tree/main/models). ONNX Path refers to the file path of the ONNX model on your computer. Generally, the default ONNX path should work. ROI settings can change the region of your capture picture.  For OneEuro filter, you can check the official [**website**](https://gery.casiez.net/1euro/) for reference.\n\n\n\n\n  \u003cimg src=\"images/VRfaceUI.jpg\" width=\"800\" /\u003e\n\n\n \n\n\n 4.Open VRCFaceTracking and VRChat.\n In VRChat, select an avatar that supports VRCFT and enable OCT. If you are unable to animate the avatar's facial expressions, ensure that ports 8888 and 9000 on your local machine are not blocked or already in use. If you are unable to establish an OSC connection between your PC and VR, please refer to [**VRCFT**](https://docs.vrcft.io/docs/vrcft-software/vrcft) website. Remember to open OSC in your VR device.\n\n\n  \u003cimg src=\"images/VRCFT.png\" width=\"800\" /\u003e\n\n  \u003cimg src=\"images/OSCVR.jpg\" width=\"800\" /\u003e\n\n\n\n## Hardware\n\n\n\n \n\nYou will need [**Xiao_ESP32S3**](https://wiki.seeedstudio.com/cn/xiao_esp32s3_getting_started/)(It has compact size with both wireless and wired support and no need for additional  antennas) and OV2640 with a viewing angle of 160 degrees. These two components, when powered through the Type-C port, can stream the signal to the computer. Alternative device for Xiao_ESP32S3 is Freenove ESP 32-S3 WROOM, which is larger and  more expensive than Xiao_ESP32S3, and lacks support for an external antenna. Alternative cameras with a narrower viewing angle may reduce recognition accuracy and demand higher lighting conditions. \n\n\n\n\n  \u003cimg src=\"images/esp32.png\" width=\"800\" /\u003e\n\n  \nOptional components: 3.7V battery, 5mm White Through Hole LED. Either 3.3V or 5V can be used to power the LEDs, however the resistors used will be different. 5V can only be used when powered over USB while 3.3V can be used with both USB and battery power. For 3.3V power: Use 82-ohm resistors in series with the LEDs. For 5V power: Use 160-ohm resistors in series with the LEDs. Connect the GND pin on the ESP32 to the cathode (short leg) of the LED. Add an appropriate resistor (82 ohms for 3.3V, 160 ohms for 5V) in series with the anode (long leg) of the LED. Do not connect LEDs directly to the ESP32 pins without a resistor, as it may damage the LEDs or the microcontroller. Refer to the following figure for LED connection.\n\n\n  \u003cimg src=\"images/lightLED.png\" width=\"800\" /\u003e\n\n\n\n\nIf you want to connect a battery to the Xiao, we recommend that you purchase a qualified 3.7V rechargeable lithium battery. When soldering the battery, be careful to distinguish the positive and negative terminals. The negative terminal of the battery should be on the side closest to the USB port (BAT-), and the positive terminal of the battery should be on the side away from the USB port (BAT+). Refer to the following two figures for battery connection.\n\n\n  \u003cimg src=\"images/battery.png\" width=\"800\" /\u003e\n\n\n\n\n\n\n\n  \u003cimg src=\"images/battery_connect.jpg\" width=\"800\" /\u003e\n\n\n\n\n\nPrint the model inside folder '3D_hardware'. Currently, our 3D model is provided for the Quest 3. If you require hardware-related assistance or wish to purchase finished products, please contact us via email xengine@xverse.cn. \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxverse-engine%2Fxvrfacetracking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxverse-engine%2Fxvrfacetracking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxverse-engine%2Fxvrfacetracking/lists"}