{"id":18600778,"url":"https://github.com/autonomousvision/connecting_the_dots","last_synced_at":"2025-09-06T20:35:36.616Z","repository":{"id":99789996,"uuid":"191343413","full_name":"autonomousvision/connecting_the_dots","owner":"autonomousvision","description":"This repository contains the code for the paper \"Connecting the Dots: Learning Representations for Active Monocular Depth Estimation\" https://avg.is.tuebingen.mpg.de/publications/riegler2019cvpr","archived":false,"fork":false,"pushed_at":"2021-06-07T15:54:59.000Z","size":2571,"stargazers_count":59,"open_issues_count":7,"forks_count":16,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-11T04:51:36.932Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autonomousvision.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-06-11T09:55:17.000Z","updated_at":"2025-01-25T03:16:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"9bc2ec76-c314-4ae1-92e1-a32e5bb2db35","html_url":"https://github.com/autonomousvision/connecting_the_dots","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/autonomousvision/connecting_the_dots","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fconnecting_the_dots","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fconnecting_the_dots/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fconnecting_the_dots/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fconnecting_the_dots/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autonomousvision","download_url":"https://codeload.github.com/autonomousvision/connecting_the_dots/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fconnecting_the_dots/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273959814,"owners_count":25198284,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-06T02:00:13.247Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T02:05:24.151Z","updated_at":"2025-09-06T20:35:36.598Z","avatar_url":"https://github.com/autonomousvision.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Connecting the Dots: Learning Representations for Active Monocular Depth Estimation\n\n![example](img/img.png)\n\nThis repository contains the code for the paper\n\n**[Connecting the Dots: Learning Representations for Active Monocular Depth Estimation](http://www.cvlibs.net/publications/Riegler2019CVPR.pdf)**\n\u003cbr\u003e\n[Gernot Riegler](https://griegler.github.io/), [Yiyi Liao](https://yiyiliao.github.io/), [Simon Donne](https://avg.is.tuebingen.mpg.de/person/sdonne), [Vladlen Koltun](http://vladlen.info/), and [Andreas Geiger](http://www.cvlibs.net/)\n\u003cbr\u003e\n[CVPR 2019](http://cvpr2019.thecvf.com/)\n\n\u003e We propose a technique for depth estimation with a monocular structured-light camera, i.e., a calibrated stereo set-up with one camera and one laser projector. Instead of formulating the depth estimation via a correspondence search problem, we show that a simple convolutional architecture is sufficient for high-quality disparity estimates in this setting. As accurate ground-truth is hard to obtain, we train our model in a self-supervised fashion with a combination of photometric and geometric losses. Further, we demonstrate that the projected pattern of the structured light sensor can be reliably separated from the ambient information. This can then be used to improve depth boundaries in a weakly supervised fashion by modeling the joint statistics of image and depth edges. The model trained in this fashion compares favorably to the state-of-the-art on challenging synthetic and real-world datasets. In addition, we contribute a novel simulator, which allows to benchmark active depth prediction algorithms in controlled conditions.\n\n\nIf you find this code useful for your research, please cite\n\n```\n@inproceedings{Riegler2019Connecting,\n  title={Connecting the Dots: Learning Representations for Active Monocular Depth Estimation},\n  author={Riegler, Gernot and Liao, Yiyi and Donne, Simon and Koltun, Vladlen and Geiger, Andreas},\n  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},\n  year={2019}\n}\n```\n\n\n## Dependencies\n\nThe network training/evaluation code is based on `Pytorch`.\n```\nPyTorch\u003e=1.1\nCuda\u003e=10.0\n```\nUpdated on 07.06.2021: The code is now compatible with the latest Pytorch version (1.8).\n\nThe other python packages can be installed with `anaconda`:\n```\nconda install --file requirements.txt\n```\n\n### Structured Light Renderer\nTo train and evaluate our method in a controlled setting, we implemented an structured light renderer.\nIt can be used to render a virtual scene (arbitrary triangle mesh) with the structured light pattern projected from a customizable projector location.\nTo build it, first make sure the correct `CUDA_LIBRARY_PATH` is set in `config.json`.\nAfterwards, the renderer can be build by running `make` within the `renderer` directory.\n\n### PyTorch Extensions\nThe network training/evaluation code is based on `PyTorch`.\nWe implemented some custom layers that need to be built in the `torchext` directory.\nSimply change into this directory and run\n\n```\npython setup.py build_ext --inplace\n```\n\n### Baseline HyperDepth\nAs baseline we partially re-implemented the random forest based method [HyperDepth](http://openaccess.thecvf.com/content_cvpr_2016/papers/Fanello_HyperDepth_Learning_Depth_CVPR_2016_paper.pdf).\nThe code resided in the `hyperdepth` directory and is implemented in `C++11` with a Python wrapper written in `Cython`.\nTo build it change into the directory and run\n\n```\npython setup.py build_ext --inplace\n```\n\n## Running\n\n\n### Creating Synthetic Data\nTo create synthetic data and save it locally, download [ShapeNet V2](https://www.shapenet.org/) and correct `SHAPENET_ROOT` in `config.json`. Then the data can be generated and saved to `DATA_ROOT` in `config.json` by running\n```\n./create_syn_data.sh\n```\nIf you are only interested in evaluating our pre-trained model, [here (3.7G)](https://s3.eu-central-1.amazonaws.com/avg-projects/connecting_the_dots/val_data.zip) is a validation set that contains a small amount of images.\n\n### Training Network\n\nAs a first stage, it is recommended to train the disparity decoder and edge decoder without the geometric loss. To train the network on synthetic data for the first stage run\n```\npython train_val.py\n```\n\nAfter the model is pretrained without the geometric loss, the full model can be trained from the initialized weights by running\n```\npython train_val.py --loss phge\n```\n\n\n### Evaluating Network\nTo evaluate a specific checkpoint, e.g. the 50th epoch, one can run \n```\npython train_val.py --cmd retest --epoch 50\n```\n\n### Evaluating a Pre-trained Model\nWe provide a model pre-trained using the photometric loss. Once you have prepared the synthetic dataset and changed `DATA_ROOT` in `config.json`, the pre-trained model can be evaluated on the validation set by running:\n```\nmkdir -p output\nmkdir -p output/exp_syn\nwget -O output/exp_syn/net_0099.params https://s3.eu-central-1.amazonaws.com/avg-projects/connecting_the_dots/net_0099.params\npython train_val.py --cmd retest --epoch 99\n```\nYou can also download our validation set from [here (3.7G)](https://s3.eu-central-1.amazonaws.com/avg-projects/connecting_the_dots/val_data.zip).\n\n## Acknowledgement \nThis work was supported by the Intel Network on Intelligent Systems.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fconnecting_the_dots","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautonomousvision%2Fconnecting_the_dots","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fconnecting_the_dots/lists"}