Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/rikba/hypersim_multiview
Implements and analyses frame-to-frame pixel reprojection for the Hypersim Evermotion data set.
https://github.com/rikba/hypersim_multiview
apple deep description detection extrinsics geometry hypersim intrinsics key learning matching multi point reprojection self-supervised view
Last synced: about 1 month ago
JSON representation
Implements and analyses frame-to-frame pixel reprojection for the Hypersim Evermotion data set.
- Host: GitHub
- URL: https://github.com/rikba/hypersim_multiview
- Owner: rikba
- License: mit
- Created: 2021-06-14T08:35:41.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2021-06-14T13:08:22.000Z (over 3 years ago)
- Last Synced: 2023-03-02T06:07:47.807Z (over 1 year ago)
- Topics: apple, deep, description, detection, extrinsics, geometry, hypersim, intrinsics, key, learning, matching, multi, point, reprojection, self-supervised, view
- Language: Python
- Homepage:
- Size: 4.3 MB
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Hypersim Multi View Geometry
This repository implements frame transformation methods to interface and use the [Apple Hypersim Evermotion data set](https://github.com/apple/ml-hypersim) with multi view geometry applications.
For example this package can be used to label pixel correspondences for self-supervised key point detection and description as in [KP2D](https://github.com/TRI-ML/KP2D)
## Frame-to-frame pixel projection
Pixel correspondences between two frames of the same scene can be found through the pixel world position in the source frame and the target camera pose and camera intrinsic calibration.
`warp()`:
## Occlusion detection
Pixel that are visible in the source frame but occluded in the target frame can be detected automatically.
## Reflectance detection
Reflecting and transparent surface can be masked using the diffuse reflectance information of the Hypersim dataset.
# Installation
Install the hypersim_multiview toolbox with
```
conda develop $HOME/hypersim_multiview
```
or
```
pip3 install -e $HOME/hypersim_multiview
```
# Minimum required dataset
The following files are required to compute the projection, the occlusion detection, and the reflectance detection:
```
|_detail
|--*
|images
|--scene_cam_CC_final_preview
|----frame.FFFF.color.jpg
|----frame.FFFF.diffuse_reflectance.jpg
|--scene_cam_CC_geometry.hdf5
|----frame.FFFF.position.hdf5
```
The minimum dataset is still 500 Gb, thus we have to download it through the official source.
Use [99991's alternative downloader](https://github.com/apple/ml-hypersim/tree/b125e8fa4f55539cbb2237ddb052504bf7d377bc/contrib/99991) to download the reduced dataset:
```
python3 ./download.py -d /diretory/to/download/to --contains _detail --silent
python3 ./download.py -d /diretory/to/download/to --contains .color.jpg --contains final_preview --silent
python3 ./download.py -d /diretory/to/download/to --contains .diffuse_reflectance.jpg --silent
python3 ./download.py -d /diretory/to/download/to --contains .position.hdf5 --silent
```
The download script will first list all files that it is skipping.
This may take 15 minutes.
Then it starts downloading.
# Acknowledgement
This work was created within the [3D Vision class](https://www.cvg.ethz.ch/teaching/3dvision/) student project Viewpoint Adaptation in a Synthetic Environment with Shengqu Cai (@HexagonPrime), Menelaos Kanakis (@menelaoskanakis) and Mihai Dusmanu (@mihaidusmanu) at ETH Zurich.