{"id":13794539,"url":"https://github.com/alexklwong/void-dataset","last_synced_at":"2025-05-12T21:31:36.408Z","repository":{"id":74099751,"uuid":"249894291","full_name":"alexklwong/void-dataset","owner":"alexklwong","description":"Visual Odometry with Inertial and Depth (VOID) dataset","archived":false,"fork":false,"pushed_at":"2024-07-31T21:58:15.000Z","size":4045,"stargazers_count":116,"open_issues_count":2,"forks_count":9,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-08-03T23:03:42.789Z","etag":null,"topics":["3d-reconstruction","3d-vision","computer-vision","dataset","depth-completion","icra","machine-learning","ral","sparse-to-dense"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexklwong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-25T05:26:36.000Z","updated_at":"2024-07-31T21:58:19.000Z","dependencies_parsed_at":"2024-01-07T06:20:11.299Z","dependency_job_id":"eafb7f2b-060a-4d47-a5c4-a57296ef18ae","html_url":"https://github.com/alexklwong/void-dataset","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexklwong%2Fvoid-dataset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexklwong%2Fvoid-dataset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexklwong%2Fvoid-dataset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexklwong%2Fvoid-dataset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexklwong","download_url":"https://codeload.github.com/alexklwong/void-dataset/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225157000,"owners_count":17429698,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","3d-vision","computer-vision","dataset","depth-completion","icra","machine-learning","ral","sparse-to-dense"],"created_at":"2024-08-03T23:00:42.902Z","updated_at":"2024-11-18T09:30:33.809Z","avatar_url":"https://github.com/alexklwong.png","language":"Shell","funding_links":[],"categories":["About Sparse-to-Dense Depth Completion \u003ca name=\"about-sparse-to-dense-depth-completion\"\u003e\u003c/a\u003e"],"sub_categories":[],"readme":"# VOID Dataset\nVisual Odometry with Inertial and Depth (VOID) dataset\n\nfrom *Unsupervised Depth Completion from Visual Inertial Odometry* (in RA-L January 2020 \u0026 ICRA 2020)\n\n[[arxiv]](https://arxiv.org/pdf/1905.08616.pdf) [[poster]]()\n\nAuthors: [Alex Wong](http://web.cs.ucla.edu/~alexw/), [Xiaohan Fei](https://feixh.github.io/), Stephanie Tsuei\n\nIf you use this dataset, please cite our paper:\n```\n@article{wong2020unsupervised,\n title={Unsupervised Depth Completion From Visual Inertial Odometry},\n  author={Wong, Alex and Fei, Xiaohan and Tsuei, Stephanie and Soatto, Stefano},\n  journal={IEEE Robotics and Automation Letters},\n  volume={5},\n  number={2},\n  pages={1899--1906},\n  year={2020},\n  publisher={IEEE}\n}\n```\n\nTo follow the VOID sparse-to-dense-depth completion benchmark, please visit: [Awesome State of Depth Completion](https://github.com/alexklwong/awesome-state-of-depth-completion)\n\n## About the dataset\nWe propose the VOID dataset for real world use case of depth completion by bootstrapping sparse reconstruction in *metric*}* space from a VIO system ([XIVO][xivo_github]).\n\nThe dataset was collected using the Intel [RealSense D435i camera](https://realsense.intel.com/depth-camera), which was configured to produce synchronized accelerometer and gyroscope measurements at 400 Hz, along with synchronized VGA-size (640 x 480) RGB and depth streams at 30 Hz. The depth frames are acquired using active stereo and is aligned to the RGB frame using the sensor factory calibration. All the measurements are timestamped.\n\nThe dataset contains 56 sequences in total, both indoor and outdoor with challenging motion. Typical scenes include classrooms, offices, stairwells, laboratories, and gardens. Of the 56 sequences, 48 sequences (approximately 47K frames) are designated for training and 8 sequences for testing, from which we sampled 800 frames to construct the testing set.\nEach sequence constains sparse depth maps at three density levels, 1500, 500 and 150 points, corresponding to 0.5%, 0.15% and 0.05% of VGA size.\n\n| Staircase | Classroom |\n| :--------: | :-------: |\n| \u003cimg src=\"figures/teaser_stairs.png\" width=\"450\"\u003e | \u003cimg src=\"figures/teaser_classroom.png\" width=\"450\"\u003e |\n\n\u003cp align=\"center\"\u003e\n\u003cimg align=\"center\" src=\"figures/void_samples.png\" width=\"800\"\u003e\n\u003c/p\u003e\n\n## Dataset structure\nFor the release version of the dataset:\n```\nvoid_release\n|---- \u003cdensity\u003e\n      |---- data\n            |---- \u003csequence\u003e\n                  |---- image\n                        |---- \u003ctimestamp\u003e.png\n                        |---- ...\n                  |---- sparse_depth\n                        |---- \u003ctimestamp\u003e.png\n                        |---- ...\n                  |---- validity_map\n                        |---- \u003ctimestamp\u003e.png\n                        |---- ...\n                  |---- ground_truth\n                        |---- \u003ctimestamp\u003e.png\n                        |---- ...\n                  |---- absolute_pose\n                        |---- \u003ctimestamp\u003e.txt\n                        |---- ...\n                  |----K.txt\n            |---- ...\n      |---- train_image.txt\n      |---- train_sparse_depth.txt\n      |---- train_validity_map.txt\n      |---- train_ground_truth.txt\n      |---- train_absolute_pose.txt\n      |---- train_intrinsics.txt\n      |---- test_image.txt\n      |---- test_sparse_depth.txt\n      |---- test_validity_map.txt\n      |---- test_ground_truth.txt\n      |---- test_absolute_pose.txt\n      |---- test_intrinsics.txt\n|---- ...\n```\nDensities include 150, 500, and 1500 points, corresponding to the directories void_150, void_500, void_1500, respectively. Text files prefixed with train and test contains the paths for the training and testing sets.\n\nFor the raw dataset (rosbags):\n```\nvoid_raw\n|---- \u003csequence\u003e\n      |---- dataset\n      |---- dataset_500\n      |---- dataset_1500\n      |---- raw.bag\n|---- ...\n```\nFiles prefixed with dataset are the output of [XIVO][xivo_github]. The dataset file without the density suffix (``dataset'') denotes the dataset file for 150 points.\n\n## Setting up your virtual environment\nWe will create a virtual environment with the necessary dependencies\n```\nvirtualenv -p /usr/bin/python3 void-py3env\nsource void-py3env/bin/activate\npip install numpy opencv-python Pillow matplotlib gdown\n```\n\n## Installing ROS (Kinetic)\nThis is only necessary for processing the raw dataset (rosbag). You may skip this step if you plan on only using the release version.\n\nTo install ROS Kinetic:\n```\nsudo sh -c 'echo \"deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main\" \u003e /etc/apt/sources.list.d/ros-latest.list'\nsudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654\nsudo apt-get update\nsudo apt-get install ros-kinetic-desktop-full\n```\n\nTo install build packages\n```\nsudo apt install python-rosinstall python-rosinstall-generator python-wstool build-essential\n```\n\nTo enable your ROS environment:\n```\nsource /opt/ros/kinetic/setup.bash\n```\n\n\n## Downloading VOID\n**We have migrated from Google Drive to OneDrive**\n\nThe bash scripts below are now obsolete!\n```\nbash/setup_dataset_void.sh\nbash/setup_dataset_void_raw.sh\n```\n\nTo download the release version of the dataset through your browser you may visit:\n```\nhttps://yaleedu-my.sharepoint.com/:u:/g/personal/alex_wong_yale_edu/Ebwvk0Ji8HhNinmAcKI5vSkBEjJTIWlA8PXwKNQX_FvB7g\n```\n\nAssuming you are in the root of the repository, you can download the dataset via commandline using wget.\nTo construct the same dataset structure as the setup script above:\n```\n      \nwget -O void_release.zip 'https://yaleedu-my.sharepoint.com/:u:/g/personal/alex_wong_yale_edu/Ebwvk0Ji8HhNinmAcKI5vSkBEjJTIWlA8PXwKNQX_FvB7g?e=0Zqe7g\u0026download=1'\nunzip void_release.zip\n```\n\nIf you encounter `error: invalid zip file with overlapped components (possible zip bomb)`. Please do the following\n```\nexport UNZIP_DISABLE_ZIPBOMB_DETECTION=TRUE\n```\nand run the above again.\n\nTo download the raw VOID dataset (rosbag) through your browser, you may visit:\n```\nhttps://yaleedu-my.sharepoint.com/:u:/g/personal/alex_wong_yale_edu/EcdGqVyQ8DVBqnCX8lUqSDcBRVi0SpV2XwMdYiYbeCzOaA\n```\n\nAssuming you are in the root of the repository, you can download the dataset via commandline using wget.\nTo construct the same dataset structure as the setup script above:\n```\nwget -O void_raw.zip 'https://yaleedu-my.sharepoint.com/:u:/g/personal/alex_wong_yale_edu/EcdGqVyQ8DVBqnCX8lUqSDcBRVi0SpV2XwMdYiYbeCzOaA?e=QjCthr\u0026download=1'\nunzip void_raw.zip\n```\n\n## Loading calibration\nCalibration are stored as JSON and text (formatted as JSON) files within the `calibration` folder.\n\n```\nvoid_dataset\n|---- \u003ccalibration\u003e\n      |---- calibration.json\n      |---- calibration.txt\n```\n\nTo read calibration as a map or dictionary:\n```\nimport os, json\n\ncalibration_path = os.path.join('calibration', 'calibration.json')\n\nwith open(calibration_path, 'r') as json_file:\n    calibration = json.load(json_file)\n```\n\nNote: we use a radtan (plumb bob) distortion model.\n\nThe following are the definitions for the calibration parameter names:\n```\nf_x, f_y : focal length\nc_x, c_y : principal point\nk_0, k_1, k_2 : radial distortion coefficients\np_x, p_y : tangential distortion coefficients\nb_a, b_g : bias for accelerometer and gyroscope\nc_ar, c_as, c_gs, c_gru, c_grl : IMU axis alignment parameters\nn_a, n_g : noise for accelerometer and gyroscope\nt_camera_to_body : translation vector for camera to imu alignment\nw_camera_to_body : rotation (Rodrigues') parameters for camera to IMU alignment\n```\n\n## Loading and storing data\nTo load depth and validity map filepaths:\n```\nimport data_utils\n\ntrain_sparse_depth_filepath = 'data/void_1500/train_image.txt'\ntrain_validity_map_filepath = 'data/void_1500/train_image.txt'\ntrain_sparse_depth_paths = data_utils.load_paths(train_sparse_depth_filepath)\ntrain_validity_paths = data_utils.load_paths(train_sparse_depth_filepath)\n```\n\nTo load depth and validity maps:\n```\nsparse_depth = data_utils.load_depth(train_sparse_depth_paths[0])\nvalidity_map = data_utils.load_validity_map(train_validity_map_paths[0])\n```\n\nTo store depth and validity maps:\n```\nsparse_depth_outpath = 'sparse_depth.png'\nvalidity_map_outpath = 'validity_map.png'\ndata_utils.save_depth(sparse_depth)\ndata_utils.save_validity_map(validity_map)\n```\n\nTo read intrinsics or pose (both are store as numpy text files):\n```\nimport numpy as np\nK = np.loadtxt('K.txt')\n```\n\n## Related projects\nYou may also find the following projects useful:\n\n- [KBNet][kbnet_github]: *Unsupervised Depth Completion with Calibrated Backprojection Layers*. A fast (15 ms/frame) and accurate unsupervised sparse-to-dense depth completion method that introduces a calibrated backprojection layer that improves generalization across sensor platforms. This work is published as an *oral* paper in the International Conference on Computer Vision (ICCV) 2021.\n- [ScaffNet][scaffnet_github]: *Learning Topology from Synthetic Data for Unsupervised Depth Completion*. An unsupervised sparse-to-dense depth completion method that first learns a map from sparse geometry to an initial dense topology from synthetic data (where ground truth comes for free) and amends the initial estimation by validating against the image. This work is published in the Robotics and Automation Letters (RA-L) 2021 and the International Conference on Robotics and Automation (ICRA) 2021.\n- [AdaFrame][adaframe_github]: *Learning Topology from Synthetic Data for Unsupervised Depth Completion*. An adaptive framework for learning unsupervised sparse-to-dense depth completion that balances data fidelity and regularization objectives based on model performance on the data. This work is published in the Robotics and Automation Letters (RA-L) 2021 and the International Conference on Robotics and Automation (ICRA) 2021.\n- [VOICED][voiced_github]: *Unsupervised Depth Completion from Visual Inertial Odometry*. An unsupervised sparse-to-dense depth completion method, developed by the authors. The paper introduces Scaffolding for depth completion and a light-weight network to refine it. This work is published in the Robotics and Automation Letters (RA-L) 2020 and the International Conference on Robotics and Automation (ICRA) 2020.\n- [XIVO][xivo_github]: The Visual-Inertial Odometry system developed at UCLA Vision Lab. This work is built on top of XIVO. The VOID dataset used by this work also leverages XIVO to obtain sparse points and camera poses.\n- [GeoSup][geosup_github]: *Geo-Supervised Visual Depth Prediction*. A single image depth prediction method developed by the authors, published in the Robotics and Automation Letters (RA-L) 2019 and the International Conference on Robotics and Automation (ICRA) 2019. This work was awarded **Best Paper in Robot Vision** at ICRA 2019.\n- [AdaReg][adareg_github]: *Bilateral Cyclic Constraint and Adaptive Regularization for Unsupervised Monocular Depth Prediction.* A single image depth prediction method that introduces adaptive regularization. This work was published in the proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) 2019.\n\nWe also have works in adversarial attacks on depth estimation methods and medical image segmentation:\n- [Stereopagnosia][stereopagnosia_github]: *Stereopagnosia: Fooling Stereo Networks with Adversarial Perturbations.* Adversarial perturbations for stereo depth estimation, published in the Proceedings of AAAI Conference on Artificial Intelligence (AAAI) 2021.\n- [Targeted Attacks for Monodepth][targeted_attacks_monodepth_github]: *Targeted Adversarial Perturbations for Monocular Depth Prediction.* Targeted adversarial perturbations attacks for monocular depth estimation, published in the proceedings of Neural Information Processing Systems (NeurIPS) 2020.\n- [SPiN][spin_github] : *Small Lesion Segmentation in Brain MRIs with Subpixel Embedding.* Subpixel architecture for segmenting ischemic stroke brain lesions in MRI images, published in the Proceedings of Medical Image Computing and Computer Assisted Intervention (MICCAI) Brain Lesion Workshop 2021 as an **oral paper**.\n\n[kitti_dataset]: http://www.cvlibs.net/datasets/kitti/\n[nyu_v2_dataset]: https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html\n[void_github]: https://github.com/alexklwong/void-dataset\n[voiced_github]: https://github.com/alexklwong/unsupervised-depth-completion-visual-inertial-odometry\n[scaffnet_github]: https://github.com/alexklwong/learning-topology-synthetic-data\n[adaframe_github]: https://github.com/alexklwong/adaframe-depth-completion\n[kbnet_github]: https://github.com/alexklwong/calibrated-backprojection-network\n[xivo_github]: https://github.com/ucla-vision/xivo\n[geosup_github]: https://github.com/feixh/GeoSup\n[adareg_github]: https://github.com/alexklwong/adareg-monodispnet\n[stereopagnosia_github]: https://github.com/alexklwong/stereopagnosia\n[targeted_attacks_monodepth_github]: https://github.com/alexklwong/targeted-adversarial-perturbations-monocular-depth\n[spin_github]: https://github.com/alexklwong/subpixel-embedding-segmentation\n\n## License and disclaimer\n\nThis software is property of the UC Regents, and is provided free of charge for research purposes only. It comes with no warranties, expressed or implied, according to these [terms and conditions](license). For commercial use, please contact [UCLA TDG](https://tdg.ucla.edu).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexklwong%2Fvoid-dataset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexklwong%2Fvoid-dataset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexklwong%2Fvoid-dataset/lists"}