{"id":19517454,"url":"https://github.com/cvg/geocalib","last_synced_at":"2025-05-15T19:01:46.994Z","repository":{"id":256498241,"uuid":"852711648","full_name":"cvg/GeoCalib","owner":"cvg","description":"GeoCalib: Learning Single-image Calibration with Geometric Optimization (ECCV 2024)","archived":false,"fork":false,"pushed_at":"2025-05-08T09:37:58.000Z","size":24118,"stargazers_count":581,"open_issues_count":18,"forks_count":37,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-05-08T10:34:59.637Z","etag":null,"topics":["camera-calibration","deep-learning","optimization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cvg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-05T09:35:30.000Z","updated_at":"2025-05-08T09:38:01.000Z","dependencies_parsed_at":"2024-09-11T05:16:36.468Z","dependency_job_id":"84a24691-880a-4147-8f48-55e867cb847f","html_url":"https://github.com/cvg/GeoCalib","commit_stats":{"total_commits":11,"total_committers":4,"mean_commits":2.75,"dds":0.4545454545454546,"last_synced_commit":"c192c6abd1bbb9efe2541907bf7956627e20abe2"},"previous_names":["cvg/geocalib"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvg%2FGeoCalib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvg%2FGeoCalib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvg%2FGeoCalib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvg%2FGeoCalib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cvg","download_url":"https://codeload.github.com/cvg/GeoCalib/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254404337,"owners_count":22065639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["camera-calibration","deep-learning","optimization"],"created_at":"2024-11-11T00:03:43.853Z","updated_at":"2025-05-15T19:01:45.341Z","avatar_url":"https://github.com/cvg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ch1 align=\"center\"\u003e\u003cins\u003eGeoCalib\u003c/ins\u003e 📸\u003cbr\u003eSingle-image Calibration with Geometric Optimization\u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://www.linkedin.com/in/alexander-veicht/\"\u003eAlexander Veicht\u003c/a\u003e\n    ·\n    \u003ca href=\"https://psarlin.com/\"\u003ePaul-Edouard\u0026nbsp;Sarlin\u003c/a\u003e\n    ·\n    \u003ca href=\"https://www.linkedin.com/in/philipplindenberger/\"\u003ePhilipp Lindenberger\u003c/a\u003e\n    ·\n    \u003ca href=\"https://www.microsoft.com/en-us/research/people/mapoll/\"\u003eMarc\u0026nbsp;Pollefeys\u003c/a\u003e\n  \u003c/p\u003e\n  \u003ch2 align=\"center\"\u003e\n    \u003cp\u003eECCV 2024\u003c/p\u003e\n    \u003ca href=\"https://arxiv.org/pdf/2409.06704\" align=\"center\"\u003ePaper\u003c/a\u003e | \n    \u003ca href=\"https://veichta-geocalib.hf.space\" align=\"center\"\u003eDemo 🤗\u003c/a\u003e | \n    \u003ca href=\"https://colab.research.google.com/drive/1oMzgPGppAPAIQxe-s7SRd_q8r7dVfnqo#scrollTo=etdzQZQzoo-K\" align=\"center\"\u003eColab\u003c/a\u003e |\n    \u003ca href=\"https://youtu.be/uOmTwvKreM4\" align=\"center\"\u003eVideo\u003c/a\u003e\n  \u003c/h2\u003e\n  \n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"\"\u003e\u003cimg src=\"assets/teaser.gif\" alt=\"example\" width=80%\u003e\u003c/a\u003e \u003c!--TODO: update link--\u003e\n    \u003cbr\u003e\n    \u003cem\u003e\n      GeoCalib accurately estimates the camera intrinsics and gravity direction from a single image \n      \u003cbr\u003e\n      by combining geometric optimization with deep learning.\n    \u003c/em\u003e\n\u003c/p\u003e\n\n##\n\nGeoCalib is an algorithm for single-image calibration: it estimates the camera intrinsics and gravity direction from a single image only. By combining geometric optimization with deep learning, GeoCalib provides a more flexible and accurate calibration compared to previous approaches. This repository hosts the [inference](#setup-and-demo), [evaluation](#evaluation), and [training](#training) code for GeoCalib and instructions to download our training set [OpenPano](#openpano-dataset).\n\n\n## Setup and demo \n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1oMzgPGppAPAIQxe-s7SRd_q8r7dVfnqo#scrollTo=etdzQZQzoo-K)\n[![Hugging Face](https://img.shields.io/badge/Gradio-Demo-blue)](https://veichta-geocalib.hf.space)\n\nWe provide a small inference package [`geocalib`](geocalib) that requires only minimal dependencies and Python \u003e= 3.9. First clone the repository and install the dependencies:\n\n```bash\ngit clone https://github.com/cvg/GeoCalib.git \u0026\u0026 cd GeoCalib\npython -m pip install -e .\n# OR\npython -m pip install -e \"git+https://github.com/cvg/GeoCalib#egg=geocalib\"\n```\n\nHere is a minimal usage example:\n\n```python\nfrom geocalib import GeoCalib\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = GeoCalib().to(device)\n\n# load image as tensor in range [0, 1] with shape [C, H, W]\nimage = model.load_image(\"path/to/image.jpg\").to(device)\nresult = model.calibrate(image)\n\nprint(\"camera:\", result[\"camera\"])\nprint(\"gravity:\", result[\"gravity\"])\n```\n\nCheck out our [demo notebook](demo.ipynb) for a full working example.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e[Interactive demo for your webcam - click to expand]\u003c/b\u003e\u003c/summary\u003e\nRun the following command:\n  \n```bash\npython -m geocalib.interactive_demo --camera_id 0\n```\n\nThe demo will open a window showing the camera feed and the calibration results. If `--camera_id` is not provided, the demo will ask for the IP address of a [droidcam](https://droidcam.app) camera.\n\nControls:\n\n\u003eToggle the different features using the following keys:\n\u003e\n\u003e- ```h```: Show the estimated horizon line\n\u003e- ```u```: Show the estimated up-vectors\n\u003e- ```l```: Show the estimated latitude heatmap\n\u003e- ```c```: Show the confidence heatmap for the up-vectors and latitudes\n\u003e- ```d```: Show undistorted image, will overwrite the other features\n\u003e- ```g```: Shows a virtual grid of points\n\u003e- ```b```: Shows a virtual box object\n\u003e\n\u003eChange the camera model using the following keys:\n\u003e\n\u003e- ```1```: Pinhole -\u003e Simple and fast\n\u003e- ```2```: Simple Radial -\u003e For small distortions\n\u003e- ```3```: Simple Divisional -\u003e For large distortions\n\u003e\n\u003ePress ```q``` to quit the demo.\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e[Load GeoCalib with torch hub - click to expand]\u003c/b\u003e\u003c/summary\u003e\n\n```python\nmodel = torch.hub.load(\"cvg/GeoCalib\", \"GeoCalib\", trust_repo=True)\n```\n\n\u003c/details\u003e\n\n### Camera models\nGeoCalib currently supports the following camera models via the `camera_model` parameter:\n1. `pinhole` (default) models only the focal lengths `fx` and `fy` but no lens distortion.\n2. `simple_radial` models weak distortions with a single polynomial distortion parameter `k1`.\n3. `radial` models stronger distortions with polynomial distortion parameters `k1`, `k2` (thanks to [RuibinMa #25](https://github.com/cvg/GeoCalib/pull/25)).\n4. `simple_divisional` models strong fisheye distortions with a single distortion parameter `k1`, as proposed by Fitzgibbon in [_Simultaneous linear estimation of multiple view geometry and lens distortion_](https://www.robots.ox.ac.uk/~vgg/publications/2001/Fitzgibbon01b/fitzgibbon01b.pdf) (CVPR 2001).\n\nThe default model is optimized for pinhole images. To handle lens distortion, use the following:\n```python\nmodel = GeoCalib(weights=\"distorted\")  # default is \"pinhole\"\nresult = model.calibrate(image, camera_model=\"simple_radial\")  # or pinhole, simple_divisional\n```\nThe principal point is assumed to be at the center of the image and is not optimized. Additional models can be implemented by extending the [`Camera`](geocalib/camera.py) object.\n\n### Partial calibration\nWhen either the intrinsics or the gravity are already known, they can be provided as follows:\n\n```python\n# known intrinsics:\nresult = model.calibrate(image, priors={\"focal\": focal_length_tensor})\n\n# known gravity:\nresult = model.calibrate(image, priors={\"gravity\": gravity_direction_tensor})\n```\n\n### Multi-image calibration\nTo calibrate multiple images captured by the same camera, pass a list of images to GeoCalib:\n```python\n# batch is a list of tensors, each with shape [C, H, W]\nresult = model.calibrate(batch, shared_intrinsics=True)\n```\n\n## Evaluation\n\nThe full evaluation and training code is provided in the single-image calibration library [`siclib`](siclib), which can be installed as:\n```bash\npython -m pip install -e siclib\n```\n\nRunning the evaluation commands will write the results to `outputs/results/`.\n\n### LaMAR\n\nRunning the evaluation commands will download the dataset to ```data/lamar2k``` which will take around 400 MB of disk space.\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate GeoCalib]\u003c/summary\u003e\n\nTo evaluate GeoCalib trained on the OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.lamar2k --conf geocalib-pinhole --tag geocalib --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate DeepCalib]\u003c/summary\u003e\n\nTo evaluate DeepCalib trained on the OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.lamar2k --conf deepcalib --tag deepcalib --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate Perspective Fields]\u003c/summary\u003e\n\nTo evaluate Perspective Fields, first setup the files following the instructions in the [ParamNet-siclib](https://github.com/veichta/ParamNet-siclib) repository. Then run:\n\n```bash\npython -m siclib.eval.lamar2k --conf perspective-cities data.preprocessing.resize_backend=\"PIL\" --overwrite\n```\nTo evaluate the model trained on our OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.lamar2k --conf perspective-openpano --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate UVP]\u003c/summary\u003e\n\nTo evaluate UVP, install the [VP-Estimation-with-Prior-Gravity](https://github.com/cvg/VP-Estimation-with-Prior-Gravity) under ```third_party/VP-Estimation-with-Prior-Gravity```. Then run:\n\n```bash\npython -m siclib.eval.lamar2k --conf uvp --tag uvp --overwrite data.preprocessing.edge_divisible_by=null\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate your own model]\u003c/summary\u003e\n\nIf you have trained your own model, you can evaluate it by running:\n\n```bash\npython -m siclib.eval.lamar2k --checkpoint \u003cexperiment name\u003e --tag \u003ceval name\u003e --overwrite\n```\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e[Results]\u003c/summary\u003e\n\nHere are the results for the Area Under the Curve (AUC) for the roll, pitch and field of view (FoV) errors at 1/5/10 degrees for the different methods:\n\n| Approach            | Roll               | Pitch              | FoV                |\n| ------------------- | ------------------ | ------------------ | ------------------ |\n| DeepCalib           | 44.1 / 73.9 / 84.8 | 10.8 / 28.3 / 49.8 | 00.7 / 13.0 / 24.0 |\n| ParamNet            | 38.7 / 69.4 / 82.8 | 19.0 / 44.7 / 65.7 | 01.8 / 06.2 / 13.2 |\n| ParamNet (OpenPano) | 51.7 / 77.0 / 86.0 | 27.0 / 52.7 / 70.2 | 02.8 / 06.8 / 14.3 |\n| UVP                 | 72.7 / 81.8 / 85.7 | 42.3 / 59.9 / 69.4 | 15.6 / 30.6 / 43.5 |\n| GeoCalib            | 86.4 / 92.5 / 95.0 | 55.0 / 76.9 / 86.2 | 19.1 / 41.5 / 60.0 |\n\u003c/details\u003e\n\n### MegaDepth\n\nRunning the evaluation commands will download the dataset to ```data/megadepth2k``` or ```data/memegadepth2k-radial``` which will take around 2.1 GB and 1.47 GB of disk space respectively.\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate GeoCalib]\u003c/summary\u003e\n\nTo evaluate GeoCalib trained on the OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.megadepth2k --conf geocalib-pinhole --tag geocalib --overwrite\n```\n\nTo run the eval on the radial distorted images, run:\n\n```bash\npython -m siclib.eval.megadepth2k_radial --conf geocalib-pinhole --tag geocalib --overwrite model.camera_model=simple_radial\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate DeepCalib]\u003c/summary\u003e\n\nTo evaluate DeepCalib trained on the OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.megadepth2k --conf deepcalib --tag deepcalib --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate Perspective Fields]\u003c/summary\u003e\n\nTo evaluate Perspective Fields, first setup the files following the instructions in the [ParamNet-siclib](https://github.com/veichta/ParamNet-siclib) repository. Then run:\n\n```bash\npython -m siclib.eval.megadepth2k --conf perspective-cities data.preprocessing.resize_backend=\"PIL\" --overwrite\n```\n\nTo evaluate the model trained on our OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.megadepth2k --conf perspective-openpano --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate UVP]\u003c/summary\u003e\n\nTo evaluate UVP, install the [VP-Estimation-with-Prior-Gravity](https://github.com/cvg/VP-Estimation-with-Prior-Gravity) under ```third_party/VP-Estimation-with-Prior-Gravity```. Then run:\n\n```bash\npython -m siclib.eval.megadepth2k --conf uvp --tag uvp --overwrite data.preprocessing.edge_divisible_by=null\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate your own model]\u003c/summary\u003e\n\nIf you have trained your own model, you can evaluate it by running:\n\n```bash\npython -m siclib.eval.megadepth2k --checkpoint \u003cexperiment name\u003e --tag \u003ceval name\u003e --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Results]\u003c/summary\u003e\n\nHere are the results for the Area Under the Curve (AUC) for the roll, pitch and field of view (FoV) errors at 1/5/10 degrees for the different methods:\n\n| Approach            | Roll               | Pitch              | FoV                |\n| ------------------- | ------------------ | ------------------ | ------------------ |\n| DeepCalib           | 34.6 / 65.4 / 79.4 | 11.9 / 27.8 / 44.8 | 5.6 / 12.1 / 22.9  |\n| ParamNet            | 37.0 / 66.4 / 80.8 | 15.8 / 37.3 / 57.1 | 5.3 / 12.8 / 24.0  |\n| ParamNet (OpenPano) | 43.4 / 70.7 / 82.2 | 15.4 / 34.5 / 53.3 | 3.2 / 10.1 / 21.3  |\n| UVP                 | 69.2 / 81.6 / 86.9 | 21.6 / 36.2 / 47.4 | 8.2 / 18.7 / 29.8  |\n| GeoCalib            | 82.6 / 90.6 / 94.0 | 32.4 / 53.3 / 67.5 | 13.6 / 31.7 / 48.2 |\n\u003c/details\u003e\n\n### TartanAir\n\nRunning the evaluation commands will download the dataset to ```data/tartanair``` which will take around 1.85 GB of disk space.\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate GeoCalib]\u003c/summary\u003e\n\nTo evaluate GeoCalib trained on the OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.tartanair --conf geocalib-pinhole --tag geocalib --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate DeepCalib]\u003c/summary\u003e\n\nTo evaluate DeepCalib trained on the OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.tartanair --conf deepcalib --tag deepcalib --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate Perspective Fields]\u003c/summary\u003e\n\nTo evaluate Perspective Fields, first setup the files following the instructions in the [ParamNet-siclib](https://github.com/veichta/ParamNet-siclib) repository. Then run:\n\n```bash\npython -m siclib.eval.tartanair --conf perspective-cities data.preprocessing.resize_backend=\"PIL\" --overwrite\n```\n\nTo evaluate the model trained on our OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.tartanair --conf perspective-openpano --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate UVP]\u003c/summary\u003e\n\nTo evaluate UVP, install the [VP-Estimation-with-Prior-Gravity](https://github.com/cvg/VP-Estimation-with-Prior-Gravity) under ```third_party/VP-Estimation-with-Prior-Gravity```. Then run:\n\n```bash\npython -m siclib.eval.tartanair --conf uvp --tag uvp --overwrite data.preprocessing.edge_divisible_by=null\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate your own model]\u003c/summary\u003e\n\nIf you have trained your own model, you can evaluate it by running:\n\n```bash\npython -m siclib.eval.tartanair --checkpoint \u003cexperiment name\u003e --tag \u003ceval name\u003e --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Results]\u003c/summary\u003e\n\nHere are the results for the Area Under the Curve (AUC) for the roll, pitch and field of view (FoV) errors at 1/5/10 degrees for the different methods:\n\n| Approach            | Roll               | Pitch              | FoV                |\n| ------------------- | ------------------ | ------------------ | ------------------ |\n| DeepCalib           | 24.7 / 55.4 / 71.5 | 16.3 / 38.8 / 58.5 | 01.5 / 08.8 / 27.2 |\n| ParamNet            | 23.3 / 51.4 / 71.0 | 19.9 / 43.8 / 62.9 | 08.5 / 22.5 / 40.8 |\n| ParamNet (OpenPano) | 34.5 / 59.2 / 73.9 | 19.4 / 42.0 / 60.3 | 06.0 / 16.8 / 31.6 |\n| UVP                 | 52.1 / 64.8 / 71.9 | 36.2 / 48.8 / 58.6 | 15.8 / 25.8 / 35.7 |\n| GeoCalib            | 71.3 / 83.8 / 89.8 | 38.2 / 62.9 / 76.6 | 14.1 / 30.4 / 47.6 |\n\u003c/details\u003e\n\n### Stanford2D3D\n\nBefore downloading and running the evaluation, you will need to agree to the [terms of use](https://docs.google.com/forms/d/e/1FAIpQLScFR0U8WEUtb7tgjOhhnl31OrkEs73-Y8bQwPeXgebqVKNMpQ/viewform?c=0\u0026w=1) for the Stanford2D3D dataset.\nRunning the evaluation commands will download the dataset to ```data/stanford2d3d``` which will take around 885 MB of disk space.\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate GeoCalib]\u003c/summary\u003e\n\nTo evaluate GeoCalib trained on the OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.stanford2d3d --conf geocalib-pinhole --tag geocalib --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate DeepCalib]\u003c/summary\u003e\n\nTo evaluate DeepCalib trained on the OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.stanford2d3d --conf deepcalib --tag deepcalib --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate Perspective Fields]\u003c/summary\u003e\n\nTo evaluate Perspective Fields, first setup the files following the instructions in the [ParamNet-siclib](https://github.com/veichta/ParamNet-siclib) repository. Then run:\n\n```bash\npython -m siclib.eval.stanford2d3d --conf perspective-cities data.preprocessing.resize_backend=\"PIL\" --overwrite\n```\n\nTo evaluate the model trained on our OpenPano dataset, run:\n\n```bash\npython -m siclib.eval.stanford2d3d --conf perspective-openpano --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate UVP]\u003c/summary\u003e\n\nTo evaluate UVP, install the [VP-Estimation-with-Prior-Gravity](https://github.com/cvg/VP-Estimation-with-Prior-Gravity) under ```third_party/VP-Estimation-with-Prior-Gravity```. Then run:\n\n```bash\npython -m siclib.eval.stanford2d3d --conf uvp --tag uvp --overwrite data.preprocessing.edge_divisible_by=null\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Evaluate your own model]\u003c/summary\u003e\n\nIf you have trained your own model, you can evaluate it by running:\n\n```bash\npython -m siclib.eval.stanford2d3d --checkpoint \u003cexperiment name\u003e --tag \u003ceval name\u003e --overwrite\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Results]\u003c/summary\u003e\n\nHere are the results for the Area Under the Curve (AUC) for the roll, pitch and field of view (FoV) errors at 1/5/10 degrees for the different methods:\n\n| Approach            | Roll               | Pitch              | FoV                |\n| ------------------- | ------------------ | ------------------ | ------------------ |\n| DeepCalib           | 33.8 / 63.9 / 79.2 | 21.6 / 46.9 / 65.7 | 08.1 / 20.6 / 37.6 |\n| ParamNet            | 20.6 / 48.5 / 68.1 | 20.9 / 44.2 / 61.5 | 07.4 / 18.0 / 33.2 |\n| ParamNet (OpenPano) | 44.6 / 73.9 / 84.8 | 29.2 / 56.7 / 73.1 | 05.8 / 14.3 / 27.8 |\n| UVP                 | 65.3 / 74.6 / 79.1 | 51.2 / 63.0 / 69.2 | 22.2 / 39.5 / 51.3 |\n| GeoCalib            | 83.1 / 91.8 / 94.8 | 52.3 / 74.8 / 84.6 | 17.4 / 40.0 / 59.4 |\n\n\u003c/details\u003e\n\n### Evaluation options\n\nIf you want to provide priors during the evaluation, you can add one or multiple of the following flags:\n\n```bash\npython -m siclib.eval.\u003cbenchmark\u003e --conf \u003cconfig\u003e \\\n    --tag \u003ctag\u003e \\\n    data.use_prior_focal=true \\\n    data.use_prior_gravity=true \\\n    data.use_prior_k1=true\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e[Visual inspection]\u003c/summary\u003e\n\nTo visually inspect the results of the evaluation, you can run the following command:\n\n```bash\npython -m siclib.eval.inspect \u003cbenchmark\u003e \u003cone or multiple tags\u003e\n\n```\nFor example, to inspect the results of the evaluation of the GeoCalib model on the LaMAR dataset, you can run:\n```bash\npython -m siclib.eval.inspect lamar2k geocalib\n```\n\u003c/details\u003e\n\n## OpenPano Dataset\n\nThe OpenPano dataset is a new dataset for single-image calibration which contains about 2.8k panoramas from various sources, namely [HDRMAPS](https://hdrmaps.com/hdris/), [PolyHaven](https://polyhaven.com/hdris), and the [Laval Photometric Indoor HDR dataset](http://hdrdb.com/indoor-hdr-photometric/). While this dataset is smaller than previous ones, it is publicly available and it provides a better balance between indoor and outdoor scenes.\n\n\u003cdetails\u003e\n\u003csummary\u003e[Downloading and preparing the dataset]\u003c/summary\u003e\n\nIn order to assemble the training set, first download the Laval dataset following the instructions on [the corresponding project page](http://hdrdb.com/indoor-hdr-photometric/) and place the panoramas in ```data/indoorDatasetCalibrated```. Then, tonemap the HDR images using the following command:\n\n```bash\npython -m siclib.datasets.utils.tonemapping --hdr_dir data/indoorDatasetCalibrated --out_dir data/laval-tonemap\n```\n\nWe provide a script to download the PolyHaven and HDRMAPS panos. The script will create folders ```data/openpano/panoramas/{split}``` containing the panoramas specified by the ```{split}_panos.txt``` files. To run the script, execute the following commands:\n\n```bash\npython -m siclib.datasets.utils.download_openpano --name openpano --laval_dir data/laval-tonemap\n```\nAlternatively, you can download the PolyHaven and HDRMAPS panos from [here](https://cvg-data.inf.ethz.ch/GeoCalib_ECCV2024/).\n\n\nAfter downloading the panoramas, you can create the training set by running the following command:\n\n```bash\npython -m siclib.datasets.create_dataset_from_pano --config-name openpano\n```\n\nThe dataset creation can be sped up by using multiple workers and a GPU. To do so, add the following arguments to the command:\n\n```bash\npython -m siclib.datasets.create_dataset_from_pano --config-name openpano n_workers=10 device=cuda\n```\n\nThis will create the training set in ```data/openpano/openpano``` with about 37k images for training, 2.1k for validation, and 2.1k for testing.\n\n\u003cdetails\u003e\n\u003csummary\u003e[Distorted OpenPano]\u003c/summary\u003e\n\nTo create the OpenPano dataset with radial distortion, run the following command:\n\n```bash\npython -m siclib.datasets.create_dataset_from_pano --config-name openpano_radial\n```\n\n\u003c/details\u003e\n\n\u003c/details\u003e\n\n## Training\n\nAs for the evaluation, the training code is provided in the single-image calibration library [`siclib`](siclib), which can be installed by:\n\n```bash\npython -m pip install -e siclib\n```\n\nOnce the [OpenPano Dataset](#openpano-dataset) has been downloaded and prepared, we can train GeoCalib with it:\n\nFirst download the pre-trained weights for the [MSCAN-B](https://cloud.tsinghua.edu.cn/d/c15b25a6745946618462/) backbone:\n\n```bash\nmkdir weights\nwget \"https://cloud.tsinghua.edu.cn/d/c15b25a6745946618462/files/?p=%2Fmscan_b.pth\u0026dl=1\" -O weights/mscan_b.pth\n```\n\nThen, start the training with the following command:\n\n```bash\npython -m siclib.train geocalib-pinhole-openpano --conf geocalib --distributed\n```\n\nFeel free to use any other experiment name. By default, the checkpoints will be written to ```outputs/training/```. The default batch size is 24 which requires 2x 4090 GPUs with 24GB of VRAM each. Configurations are managed by [Hydra](https://hydra.cc/) and can be overwritten from the command line.\nFor example, to train GeoCalib on a single GPU with a batch size of 5, run:\n\n```bash\npython -m siclib.train geocalib-pinhole-openpano \\\n    --conf geocalib \\\n    data.train_batch_size=5 # for 1x 2080 GPU\n```\n\nBe aware that this can impact the overall performance. You might need to adjust the learning rate and number of training steps accordingly.\n\nIf you want to log the training progress to [tensorboard](https://www.tensorflow.org/tensorboard) or [wandb](https://wandb.ai/), you can set the ```train.writer``` option:\n\n```bash\npython -m siclib.train geocalib-pinhole-openpano \\\n    --conf geocalib \\\n    --distributed \\\n    train.writer=tensorboard\n```\n\nThe model can then be evaluated using its experiment name:\n\n```bash\npython -m siclib.eval.\u003cbenchmark\u003e --checkpoint geocalib-pinhole-openpano \\\n    --tag geocalib-retrained\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e[Training DeepCalib]\u003c/summary\u003e\n\nTo train DeepCalib on the OpenPano dataset, run:\n\n```bash\npython -m siclib.train deepcalib-openpano --conf deepcalib --distributed\n```\n\nMake sure that you have generated the [OpenPano Dataset](#openpano-dataset) with radial distortion or add\nthe flag ```data=openpano``` to the command to train on the pinhole images.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[Training Perspective Fields]\u003c/summary\u003e\n\nComing soon!\n\n\u003c/details\u003e\n\n## BibTeX citation\n\nIf you use any ideas from the paper or code from this repo, please consider citing:\n\n```bibtex\n@inproceedings{veicht2024geocalib,\n  author    = {Alexander Veicht and\n               Paul-Edouard Sarlin and\n               Philipp Lindenberger and\n               Marc Pollefeys},\n  title     = {{GeoCalib: Single-image Calibration with Geometric Optimization}},\n  booktitle = {ECCV},\n  year      = {2024}\n}\n```\n\n## License\n\nThe code is provided under the [Apache-2.0 License](LICENSE) while the weights of the trained model are provided under the [Creative Commons Attribution 4.0 International Public License](https://creativecommons.org/licenses/by/4.0/legalcode). Thanks to the authors of the [Laval Indoor HDR dataset](http://hdrdb.com/indoor/#presentation) for allowing this.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvg%2Fgeocalib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcvg%2Fgeocalib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvg%2Fgeocalib/lists"}