{"id":19302016,"url":"https://github.com/liuzikai/holoscanner","last_synced_at":"2026-05-16T09:31:30.070Z","repository":{"id":72346268,"uuid":"582387751","full_name":"liuzikai/HoloScanner","owner":"liuzikai","description":"In-hand scanning with Microsoft HoloLens 2 for irregular geometries","archived":false,"fork":false,"pushed_at":"2022-12-26T18:01:06.000Z","size":21537,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"MR","last_synced_at":"2025-02-24T01:47:18.059Z","etag":null,"topics":["3d-reconstruction","3d-scanning","hololens","hololens-researchmode","hololens2","unity"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/liuzikai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-26T17:04:57.000Z","updated_at":"2024-10-20T14:11:12.000Z","dependencies_parsed_at":"2023-05-02T01:34:27.054Z","dependency_job_id":null,"html_url":"https://github.com/liuzikai/HoloScanner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/liuzikai/HoloScanner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuzikai%2FHoloScanner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuzikai%2FHoloScanner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuzikai%2FHoloScanner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuzikai%2FHoloScanner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/liuzikai","download_url":"https://codeload.github.com/liuzikai/HoloScanner/tar.gz/refs/heads/MR","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuzikai%2FHoloScanner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33096851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T04:41:52.686Z","status":"ssl_error","status_checked_at":"2026-05-16T04:41:52.009Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","3d-scanning","hololens","hololens-researchmode","hololens2","unity"],"created_at":"2024-11-09T23:19:37.945Z","updated_at":"2026-05-16T09:31:30.053Z","avatar_url":"https://github.com/liuzikai.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"HoloScanner: In-Hand Scanning with HoloLens 2 for Irregular Geometries\n======================================================================\n\nAn end-to-end system for in-hand scanning and near-real-time reconstruction of irregular geometries based on Microsoft\nHoloLens 2. Group project of ETHZ Mixed Reality HS2022 by:\n\n* Zikai Liu (liuzik@student.ethz.ch)\n* Dumeni Manatschal (dumenim@student.ethz.ch)\n* Noureddine Gueddach (ngueddach@student.ethz.ch)\n* Mingyuan Chi (minchi@student.ethz.ch)\n\nSupervised by Andrea Settimi (andrea.settimi@epfl.ch).\n\nThis repo contains the Unity app running on HoloLens 2 and the server app running on PC. The Unity app and the HoloLens\nResearch Mode plugin is based\non [HoloLens2-ResearchMode-Unity](https://github.com/petergu684/HoloLens2-ResearchMode-Unity).\n\n![The scanning process](Doc/Scanning.jpg)\n\nThe post-processing program is a standalone program in another\nrepo: [mr_postprocess](https://github.com/Vellyxenya/mr_postprocess).\n\n![Output of the egg box](Doc/Output.gif)\n\nThe following prototyping repos can also be used as reference:\n\n* [walkerchi/ethz-mixed-reality-project](https://github.com/walkerchi/ethz-mixed-reality-project)\n* [Vellyxenya/mixed_reality_dev_cpp](https://github.com/Vellyxenya/mixed_reality_dev_cpp)\n\n## System Architecture\n\n![System architecture](Doc/System-Architecture.png)\n\n## The Unity App Running on HoloLens\n\nRequirements:\n\n* Unity 2019.4.22f1 on **Windows**.\n    * Unity up to 2021.1 may also work, but 2021.2 or later does not due to the reason explained\n      in [UnitySample/README.md](UnitySample/README.md).\n    * UWP application is not available for Unity on macOS or Linux.\n* Visual Studio **2019** and required components based\n  on [the HoloLens tutorial](https://learn.microsoft.com/en-us/windows/mixed-reality/develop/install-the-tools?tabs=unity).\n    * Up to 2022.10.17, VS 2022 did not work (on Win 11 ARM), showing “missing C++ workload component” error.\n      This [issue](https://github.com/microsoft/MixedRealityToolkit-Unity/issues/10647) might be related.\n* HoloLens 2 with firmware **21H2 - March 2022 (Build 20348.1450)**.\n    * Up to 2022.10.25, the latest firmware of HoloLens 2 had broken Research Mode API, causing the app to crash on\n      start. This [issue](https://github.com/microsoft/HoloLens2ForCV/issues/133) might be related. To downgrade the\n      firmware, follow\n      [this tutorial](https://learn.microsoft.com/en-us/hololens/hololens-recovery#normal-flashing-procedure),\n      using\n      the [Advanced Recovery Companion](https://apps.microsoft.com/store/detail/advanced-recovery-companion/9P74Z35SFRS8?hl=en-us\u0026gl=us).\n      Remember to disable auto update in Settings after downgrading (you can disable update for up to 30 days, so\n      remember to do\n      this regularly).\n\nOpen the Unity project [UnitySample](UnitySample) in the Unity editor. Follow the instruction\nin [UnitySample/README.md](UnitySample/README.md).\n\nBased on the original project, we extend the app with the ability to stream data to the server and to visualize point\ncloud streamed back. We also update the UI (eye tracking button to start/stop scanning, hand mesh, etc.)\n\nNote:\n\n* Compiling and deploying the Unity project takes a while (5-10 min). Machines with better performance are desirable.\n  Code wrapped in `#if WINDOWS_UWP` and `#if ENABLE_WINMD_SUPPORT` does not compile when the editor loads the scripts (\n  necessary, or it does not compile in the editor). But when actually building the app, those code compiles, so only\n  until then you may get more errors. To save time, double-check the code (like missing semicolons) before compiling.\n* Build + deploy with the device portal in Unity Build Settings is typically the fastest way. In Unity 2019.4.22f1, the\n  textbox of the device portal password behaves weirdly, especially after refreshing the scripts. If it shows the\n  error message of \"not able to connect to the device\", try to re-type the password. The generated VS project can\n  also be opened and built, but VS debugging and Unity debugger do not work (see Known Issues below).\n\n### HoloLens Research Mode Plugin\n\nThe HoloLens Research Mode API is only available in C++. To use the API in Unity/C#, a Windows Runtime (WinRT)\nComponent ([HL2UnityPlugin](HL2UnityPlugin)) is created. The plugin is compiled into a DLL as a plugin in Unity.\n\nA compiled plugin is already in the Unity project (dll and winmd\nin [UnitySample/Assets/Plugins/WSA/ARM64](UnitySample/Assets/Plugins/WSA/ARM64)).\nIf the plugin is modified, it needs to be recompiled and copied to the Plugin directory.\nSee [HL2UnityPlugin/README.md](HL2UnityPlugin/README.md) for instructions.\n\nBased on the original project, we extend the plugin with another module HL2Interactions, which acquires the user\ninteraction data (hand tracking, head tracking, eye gaze, etc.). We also extend the HL2ResearchMode module to acquire\nthe AHAT extrinsic matrix and depth look-up table (LUT).\n\nNote:\n\n* In the original project, `_XM_NO_INTRINSICS_` was used for DirectXMath. It is changed to `_XM_ARM_NEON_INTRINSICS_`.\n\n## The Server App Running on PC\n\nThe repo contains Git submodules. To check out the submodules:\n\n```shell\ngit submodule update --init\n```\n\nBuild requirements:\n\n* CMake\n* Boost (1.80.0 or so)\n* Open3D\n    * For macOS and Windows, compile from\n      source: [Build from source — Open3D 0.16.0 documentation](http://www.open3d.org/docs/release/compilation.html)\n* OpenGL \u0026 libIGL\n\nThe current branch is tested on macOS Apple Silicon with Clang. For Windows with MSVC, some more modifications might be\nneeded. For example:\n\n* `target_compile_options(DepthToPCD PRIVATE \"/MT$\u003c$\u003cCONFIG:Release\u003e:\u003e\")` to correctly link the Open3D library.\n* Directory to Boost and Open3D may need to be specified in order to be found.\n* [CPP/DirectXMath/Inc/sal.h](CPP/DirectXMath/Inc/sal.h) and [unsal.h](CPP/DirectXMath/Inc/sal.h) (as well\n  as `#include \"unsal.h\"` at the end of [DirectXMath.h](CPP/DirectXMath/Inc/DirectXMath.h)) needs to be removed\n  for Windows. According to [Microsoft/DirectXMath](https://github.com/microsoft/DirectXMath), sal.h is needed for Linux\n  and macOS. However, for Ubuntu, macros defined inside pollutes the standard library. Therefore, we add an unsal.h to\n  undef them. For Windows, such changes should be undone.\n\nSee the other branches for reference.\n\nTo build the program:\n\n```shell\nmkdir build\ncd build\ncmake -DCMAKE_BUILD_TYPE=Release ..\nmake DepthToPCD\n```\n\nNinja also works.\n\nOptionally to specify a data directory to store scanning results other than the default \"../data\",\nadd `-DDATA_FOLDER=...` to CMake command line argument.\n\n## System Usage\n\n=\u003e [Demo video](https://polybox.ethz.ch/index.php/s/QPejCiHCGQGyyPG)\n\n* Build and deploy the Unity App to HoloLens.\n* Start the app (HL2ResearchMode with a Unity icon) once. It will request for permissions. Allow all and **restart**\n  the app.\n* The menu can be dragged and dropped.\n* Use \"Toggle Point Cloud\" to switch between local per-frame point cloud and reconstruction results. Notice that\n  per-frame\n  point cloud is not equal to what being seen at scanning. For example, an object being too close or too far may have no\n  points received (see Known Issue below). The local point cloud is used to evaluate the environment noise (e.g.\n  [\"aliased depth\"](https://github.com/microsoft/HoloLens2ForCV/issues/12) of the AHAT sensor).\n* Compile and start the server. The server must be in the same network as the client (type-C connection works as an\n  ethernet, or in the same WiFi) and has **one of the fixed IPs** in the `hostIPAddresses` of TCPClient of the Unity\n  app, which is set in the Unity Editor. The current accepted IPs (former ones take higher priority) are:\n    * 169.254.41.190\n    * 192.168.1.108\n    * 192.168.1.107\n    * 192.168.155.61\n* Click \"Connect to the Server\". The LED should turn green if succeeds.\n* Hold the object with **both hands** in front of the HoloLens. Adjust the distance to the HoloLens as needed.\n* Look at the Start button to start scanning (try not to turn your head too much).\n* Wait for a while. Once the scanning starts, the reconstructed point cloud overlaps on the Object. Due to the algorithm\n  being used for registration, the scanning should have a good starting point. If that is not the case, restart the\n  scanning or even restart the server. If the server restarts, the client needs to disconnect and reconnect.\n* If you see hands jumping in the HoloLens, that means HoloLens has problem tracking your hands and the frame gets\n  dropped. Do not proceed and adjust your hand gestures until they become stable. Not both hands need to hold on to the\n  object, but both must be visible to the HoloLens.\n* If you see the point cloud turn orange, it means the registration fails. Do not proceed and realign the object with\n  the point cloud until it becomes white again.\n* Once all faces are scanned. Stop scanning using the eye-tracking button. A folder named as the current timestamp is\n  created in the Data Folder and the final point cloud is stored.\n\n## Known Issues\n\n* Depth filtering based on hand position does not work when the object is getting too close or too far from the\n  HoloLens, although the depth clipping range seems right.\n* While doing the scanning, very fast movement of the object causes the registration to lose tracking. An orange point\n  cloud is sent to the HoloLens, but then the TCP connection seems corrupted and cannot be automatically recovered with\n  the state machine in UnitySample/Assets/Scipts/TCPClient.cs. This seems not to happen when the point cloud normally\n  loses tracking.\n* The Unity application sometimes crashes. The issue is unknown yet. When debugging in VS, exceptions are thrown in GC\n  code but the root cause has not yet been located.\n* Deployment build of Unity + VS Unity debugger does not work. Viewing from the log, the debugger seems not to start on\n  HoloLens. The issue is unknown yet.\n* Eye tracking data streaming from HL2UnityPlugin/HL2Interactions to TCPDataSource is not tested yet.\n* class FileSystemSource is out of maintenance.\n* To limit the bandwidth usage of streaming raw data to the server, only every `rawDataDownSampleFactor`\n  frames ([ResearchModeVideoStream.cs](UnitySample/Assets/Scipts/ResearchModeVideoStream.cs)) is sent. Currently, the\n  processing bandwidth is \u003c 10 frame per seconds, and exceeded frames are dropped at the server anyway. But if a higher\n  processing frame rate is achieved, this number should be decreased accordingly.\n* Apple Silicon Mac seems to have problems scheduling the Open3D algorithms. Sometimes the CPU usage is far less than\n  100% (not even a single core) and the processing frame rate is extremely low. The cause is unknown yet. Running a VM\n  at the same time seems to solve the problem (reaching 500% CPU usage) but the solution is not confirmed yet. A\n  possible reason is the out-of-date OpenMP library being used when compiling Open3D (newer version is not supported).\n\n## License\n\nThe code is released under the MIT license.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliuzikai%2Fholoscanner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fliuzikai%2Fholoscanner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliuzikai%2Fholoscanner/lists"}