{"id":16984693,"url":"https://github.com/rikba/hypersim_multiview","last_synced_at":"2025-07-25T17:10:34.307Z","repository":{"id":49564409,"uuid":"376756299","full_name":"rikba/hypersim_multiview","owner":"rikba","description":"Implements and analyses frame-to-frame pixel reprojection for the Hypersim Evermotion data set. ","archived":false,"fork":false,"pushed_at":"2021-06-14T13:08:22.000Z","size":4504,"stargazers_count":4,"open_issues_count":3,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-15T02:33:39.203Z","etag":null,"topics":["apple","deep","description","detection","extrinsics","geometry","hypersim","intrinsics","key","learning","matching","multi","point","reprojection","self-supervised","view"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rikba.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-14T08:35:41.000Z","updated_at":"2024-08-13T06:49:22.000Z","dependencies_parsed_at":"2022-09-04T04:52:10.276Z","dependency_job_id":null,"html_url":"https://github.com/rikba/hypersim_multiview","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rikba%2Fhypersim_multiview","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rikba%2Fhypersim_multiview/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rikba%2Fhypersim_multiview/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rikba%2Fhypersim_multiview/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rikba","download_url":"https://codeload.github.com/rikba/hypersim_multiview/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226895183,"owners_count":17699347,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple","deep","description","detection","extrinsics","geometry","hypersim","intrinsics","key","learning","matching","multi","point","reprojection","self-supervised","view"],"created_at":"2024-10-14T02:33:39.841Z","updated_at":"2024-11-28T09:42:51.457Z","avatar_url":"https://github.com/rikba.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hypersim Multi View Geometry\nThis repository implements frame transformation methods to interface and use the [Apple Hypersim Evermotion data set](https://github.com/apple/ml-hypersim) with multi view geometry applications.\nFor example this package can be used to label pixel correspondences for self-supervised key point detection and description as in [KP2D](https://github.com/TRI-ML/KP2D)\n\n## Frame-to-frame pixel projection\nPixel correspondences between two frames of the same scene can be found through the pixel world position in the source frame and the target camera pose and camera intrinsic calibration.\n\n`warp()`:\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11293852/121869932-d4474000-cd02-11eb-9ec2-d773ee1cbae2.png\" alt=\"Frame-to-frame pixel projection, source\" height=\"240\"/\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11293852/121869938-d6110380-cd02-11eb-8644-2ef423608e06.png\" alt=\"Frame-to-frame pixel projection, target\" height=\"240\"/\u003e\n\u003c/p\u003e\n\n## Occlusion detection\nPixel that are visible in the source frame but occluded in the target frame can be detected automatically.\n\u003cimg src=\"https://user-images.githubusercontent.com/11293852/121869958-db6e4e00-cd02-11eb-8ebc-b83d669fc641.png\" alt=\"Occlusion detection\" height=\"240\"/\u003e\n\n## Reflectance detection\nReflecting and transparent surface can be masked using the diffuse reflectance information of the Hypersim dataset.\n\u003cimg src=\"https://user-images.githubusercontent.com/11293852/121869955-da3d2100-cd02-11eb-9915-7f047708ecb9.png\" alt=\"Reflectance detection\" height=\"240\"/\u003e\n\n\n# Installation\nInstall the hypersim_multiview toolbox with\n```\nconda develop $HOME/hypersim_multiview\n```\nor\n```\npip3 install -e $HOME/hypersim_multiview\n```\n\n# Minimum required dataset\nThe following files are required to compute the projection, the occlusion detection, and the reflectance detection:\n```\n|_detail\n|--*\n|images\n|--scene_cam_CC_final_preview\n|----frame.FFFF.color.jpg\n|----frame.FFFF.diffuse_reflectance.jpg\n|--scene_cam_CC_geometry.hdf5\n|----frame.FFFF.position.hdf5\n```\n\nThe minimum dataset is still 500 Gb, thus we have to download it through the official source.\nUse [99991's alternative downloader](https://github.com/apple/ml-hypersim/tree/b125e8fa4f55539cbb2237ddb052504bf7d377bc/contrib/99991) to download the reduced dataset:\n```\npython3 ./download.py -d /diretory/to/download/to --contains _detail --silent\npython3 ./download.py -d /diretory/to/download/to --contains .color.jpg --contains final_preview --silent\npython3 ./download.py -d /diretory/to/download/to --contains .diffuse_reflectance.jpg --silent\npython3 ./download.py -d /diretory/to/download/to --contains .position.hdf5 --silent\n```\n\nThe download script will first list all files that it is skipping.\nThis may take 15 minutes.\nThen it starts downloading.\n\n# Acknowledgement\nThis work was created within the [3D Vision class](https://www.cvg.ethz.ch/teaching/3dvision/) student project \u003cem\u003eViewpoint Adaptation in a Synthetic Environment\u003c/em\u003e with Shengqu Cai (@HexagonPrime), Menelaos Kanakis (@menelaoskanakis) and Mihai Dusmanu (@mihaidusmanu) at ETH Zurich.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frikba%2Fhypersim_multiview","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frikba%2Fhypersim_multiview","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frikba%2Fhypersim_multiview/lists"}