{"id":21946690,"url":"https://github.com/baegwangbin/surface_normal_uncertainty","last_synced_at":"2025-04-07T10:29:32.146Z","repository":{"id":41184334,"uuid":"397774768","full_name":"baegwangbin/surface_normal_uncertainty","owner":"baegwangbin","description":"[ICCV 2021 Oral] Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation","archived":false,"fork":false,"pushed_at":"2022-08-18T15:06:49.000Z","size":4790,"stargazers_count":220,"open_issues_count":5,"forks_count":22,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-08-22T11:04:43.145Z","etag":null,"topics":["3d-reconstruction","computer-vision","deep-learning","iccv2021","surface-normal","surface-normals","surface-normals-estimation","uncertainty","uncertainty-estimation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/baegwangbin.png","metadata":{"files":{"readme":"ReadMe.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-19T00:57:15.000Z","updated_at":"2024-08-12T14:18:41.000Z","dependencies_parsed_at":"2022-07-10T02:46:05.970Z","dependency_job_id":null,"html_url":"https://github.com/baegwangbin/surface_normal_uncertainty","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baegwangbin%2Fsurface_normal_uncertainty","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baegwangbin%2Fsurface_normal_uncertainty/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baegwangbin%2Fsurface_normal_uncertainty/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baegwangbin%2Fsurface_normal_uncertainty/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/baegwangbin","download_url":"https://codeload.github.com/baegwangbin/surface_normal_uncertainty/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247635316,"owners_count":20970717,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","computer-vision","deep-learning","iccv2021","surface-normal","surface-normals","surface-normals-estimation","uncertainty","uncertainty-estimation"],"created_at":"2024-11-29T04:34:27.594Z","updated_at":"2025-04-07T10:29:32.125Z","avatar_url":"https://github.com/baegwangbin.png","language":"Python","readme":"# Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/estimating-and-exploiting-the-aleatoric/surface-normals-estimation-on-nyu-depth-v2-1)](https://paperswithcode.com/sota/surface-normals-estimation-on-nyu-depth-v2-1?p=estimating-and-exploiting-the-aleatoric)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/estimating-and-exploiting-the-aleatoric/surface-normals-estimation-on-scannetv2)](https://paperswithcode.com/sota/surface-normals-estimation-on-scannetv2?p=estimating-and-exploiting-the-aleatoric)\n\nOfficial implementation of the paper\n\n\u003e **Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation** \\\n\u003e ICCV 2021 [oral] \\\n\u003e [Gwangbin Bae](https://baegwangbin.com), [Ignas Budvytis](https://mi.eng.cam.ac.uk/~ib255/), and [Roberto Cipolla](https://mi.eng.cam.ac.uk/~cipolla/) \\\n\u003e [[arXiv]](https://arxiv.org/abs/2109.09881) [[youtube]](https://youtu.be/mTy85tJ2oAQ)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=70% src=\"https://github.com/baegwangbin/surface_normal_uncertainty/blob/main/figs/readme_scannet.png?raw=true\"\u003e\n\u003c/p\u003e\n\nThe proposed method estimates the per-pixel surface normal probability distribution, from which the expected angular error can be inferred to quantify the aleatoric uncertainty. \nWe also introduce a novel decoder framework where pixel-wise MLPs are trained on a subset of pixels selected based on the uncertainty. \nSuch uncertainty-guided sampling prevents the bias in training towards large planar surfaces, thereby improving the level of the detail in the prediction.\n\n## Getting Started\n\nWe recommend using a virtual environment.\n```\npython3.6 -m venv --system-site-packages ./venv\nsource ./venv/bin/activate\n```\n\nInstall the necessary dependencies by\n```\npython3.6 -m pip install -r requirements.txt\n```\n\nDownload the pre-trained model weights and sample images.\n\n```python\npython download.py \u0026\u0026 cd examples \u0026\u0026 unzip examples.zip \u0026\u0026 cd ..\n```\n[25 Apr 2022] ***`download.py` does not work anymore. Please download the models and example images directly from [this link](https://drive.google.com/drive/folders/1Ku25Am69h_HrbtcCptXn4aetjo7sB33F?usp=sharing), and unzip them under `./checkpoints/` and `./examples/`.***\n\nRunning the above will download \n* `./checkpoints/nyu.pt` (model trained on NYUv2)\n* `./checkpoints/scannet.pt` (model trained on ScanNet)\n* `./examples/*.png` (sample images)\n\n## Run Demo\n\nTo test on your own images, please add them under `./examples/`. The images should be in `.png` or `.jpg`.\n\nTest using the network trained on [NYUv2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html). We used the ground truth and data split provided by [GeoNet](https://github.com/xjqi/GeoNet).\n\u003ePlease note that the ground truth for NYUv2 is only defined for the center crop of image. The prediction is therefore not accurate outside the center. When testing on your own images, we recommend using the network trained on ScanNet. \n\n```python\npython test.py --pretrained nyu --architecture GN\n```\n\nTest using the network trained on [ScanNet](http://www.scan-net.org/). We used the ground truth and data split provided by [FrameNet](https://github.com/hjwdzh/FrameNet).\n\n```python\npython test.py --pretrained scannet --architecture BN\n```\n\nRunning the above will save the predicted surface normal and uncertainty under `./examples/results/`. If successful, you will obtain images like below.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=100% src=\"https://github.com/baegwangbin/surface_normal_uncertainty/blob/main/figs/readme_generalize.png?raw=true\"\u003e\n\u003c/p\u003e\n\nThe predictions in the figure above are obtained by the network trained only on ScanNet. The network generalizes well to objects unseen during training (e.g., humans, cars, animals). The last row shows interesting examples where the input image only contains edges.\n\n## Training\n\n### Step 1. Download dataset\n\n* **NYUv2 (official)**: The official train/test split contains 795/654 images. The dataset can be downloaded from [this link](https://drive.google.com/drive/folders/1Ku25Am69h_HrbtcCptXn4aetjo7sB33F?usp=sharing). Unzip the file `nyu_dataset.zip` under `./datasets`, so that `./datasets/nyu/train` and `./datasets/nyu/test/` exist.\n\n* **NYUv2 (big)**: Please visit [GeoNet](https://github.com/xjqi/GeoNet) to download a larger training set consisting of 30907 images. This is the training set used to train our model.\n\n* **ScanNet:** Please visit [FrameNet](https://github.com/hjwdzh/framenet/tree/master/src) to download ScanNet with ground truth surface normals.\n\n### Step 2. Train\n\n* If you wish to train on NYUv2 official split, simply run \n```python\npython train.py\n```\n* If you wish to train your own model, modify the file `./models/baseline.py` and add `--use_baseline` flag. The default loss function `UG_NLL_ours` assumes uncertainty-guided sampling, so this should be changed to something else (e.g. try `--loss_fn NLL_ours`).\n\n\n## Citation\n\nIf you find our work useful in your research please consider citing our paper:\n\n```\n@InProceedings{Bae2021,\n    title   = {Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation}\n    author  = {Gwangbin Bae and Ignas Budvytis and Roberto Cipolla},\n    booktitle = {International Conference on Computer Vision (ICCV)},\n    year = {2021}                         \n}\n```\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaegwangbin%2Fsurface_normal_uncertainty","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbaegwangbin%2Fsurface_normal_uncertainty","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaegwangbin%2Fsurface_normal_uncertainty/lists"}