{"id":17222780,"url":"https://github.com/cheind/rgbd-correction","last_synced_at":"2025-06-19T08:34:23.650Z","repository":{"id":45323171,"uuid":"103556371","full_name":"cheind/rgbd-correction","owner":"cheind","description":"Code and data accompanying our work on spatio-thermal depth correction of RGB-D sensors based on Gaussian Process Regression in real-time.","archived":false,"fork":false,"pushed_at":"2022-07-13T15:40:43.000Z","size":2443,"stargazers_count":17,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-14T00:16:52.367Z","etag":null,"topics":["calibration","depth-maps","gaussian-process-regression","gaussian-processes","rgbd","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cheind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-14T16:32:27.000Z","updated_at":"2025-04-03T14:49:19.000Z","dependencies_parsed_at":"2022-09-13T06:02:50.935Z","dependency_job_id":null,"html_url":"https://github.com/cheind/rgbd-correction","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cheind/rgbd-correction","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheind%2Frgbd-correction","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheind%2Frgbd-correction/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheind%2Frgbd-correction/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheind%2Frgbd-correction/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cheind","download_url":"https://codeload.github.com/cheind/rgbd-correction/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheind%2Frgbd-correction/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260716041,"owners_count":23051358,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["calibration","depth-maps","gaussian-process-regression","gaussian-processes","rgbd","tensorflow"],"created_at":"2024-10-15T04:06:20.330Z","updated_at":"2025-06-19T08:34:18.637Z","avatar_url":"https://github.com/cheind.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dense RGB-D depth correction in the spatio-thermal domain\n\nThis repository contains code and data accompanying our work on spatio-thermal depth correction of RGB-D sensors based on Gaussian Processes in real-time.\n\n![](etc/correction_t17_p1200.png)\n\n## Documentation\n\nWe provide presentation slides of the ICMV 2017 conference event\n\n - [PDF Slides 16:9](etc/rgbd-correction-slides-169.pdf)\n - [PDF Slides 4:3](etc/rgbd-correction-slides-43.pdf)\n\nOur presentation was awarded the prize for ***best oral presentation of the conference***.\n\n## Capture setup\n\nOur capture setup consists of a RGB-D sensor looking towards a known planar object. The sensor is coupled with an electronic linear axis to adjust distance. We captured data at distances [40cm, 90cm, 10cm steps] in the temperate range of [25°C, 35°C, 1°C steps]. At each temperature/distance tuple we grabbed 50 images from both RGB and IR (aligned with RGB) sensors. We then created an artificial depth map for all RGB images utilizing the known calibration target in sight.\n\n## Dataset\n\nWe provide two versions of our dataset\n- [rgbd-correction-mini.zip](https://zenodo.org/record/6827435/files/rgbd-correction-mini.zip?download=1) ~80 MBytes | Contains mostly pre-processed mean depth images as described in our paper.\n- [rgbd-correction-raw.zip](https://zenodo.org/record/6827435/files/rgbd-correction-raw.zip?download=1) ~6 GBytes | Contains the entire raw capture data, plus artificial depth maps.\n\nDepending on your needs, you might choose one over the other. As a rule of thumb use `rgbd-correction-mini.zip` if you like to reproduce our results or play around with our code basis. Otherwise go with `rgbd-correction-raw.zip`.\n\n*Note* As of 2022 the data has been re-packaged and uploaded to [zenodo](https://zenodo.org/record/6827435#.Ys7SkXZBwuV). See [issue #1](https://github.com/cheind/rgbd-correction/issues/1) for details.\n\n### The `rgbd-correction-mini` version\n\nThe `.zip` has the following file structure\n```\nroot\n│   ReadMe.md                       Notes on the dataset\n│   intrinsics.txt                  Intrinsic camera parameters\n│   distortions.txt                 Lens distortion parameters\n|   preprocessed_depth.npz          Pre-processed data in numpy format\n```\n\nThe following snippet shows how to load the extracted data.\n\n```python\nimport numpy as np\nimport sys\n\n# Read npz file\ndata = np.load(sys.argv[1])    \n\n# All temps and positions enumerated\ntemps = data['temps']\nposes = data['poses']    \n\nprint('Temperatures {}'.format(temps))\nprint('Positions {}'.format(poses))\n\n# Access pre-processed mean depth maps\nall_depths_ir = data['depth_ir'][()]\nall_depths_rgb = data['depth_rgb'][()]\n\n# Index an individual depth map by position-temperature tuple\ndepth_ir = all_depths_ir[(poses[0], temps[0])]\nprint(depth_ir.shape)\n```\n\n### The `rgbd-correction-raw` version\n\nThe `.zip` has the following file structure\n```\nroot\n│   ReadMe.md                       Notes on the dataset\n│   intrinsics.txt                  Intrinsic camera parameters\n│   distortions.txt                 Lens distortion parameters\n|   index.csv                       CSV index of all files\n|   000000_t10_p0700_color.png      RGB image at t=10°C, pos=700\n|   000000_t10_p0700_depth.png      IR depth image in mm\n|   000000_t10_p0700_sdepth.exr     Artificial RGB float32 depth map in mm\n|   000000_t10_p0700_sdepth.exr     Artificial RGB ushort16 depth map in mm\n|   000000_t10_p0700_sdepth.txt     4x4 pose of calibration object w.r.t RGB\n|   ...\n|   007799_t35_p1200_sdepth.txt\n```\n\nThe following snippet makes use of index.csv to enumerate the data quickly.\n\n```python\nimport numpy as np\nimport pandas as pd\nimport sys\n\n# Read index.csv from CLI arguments\ndf = pd.DataFrame.from_csv(sys.argv[1], sep=' ')\n\n# Loop over all IR depth maps grouped by temperature\n# This will give 6x50 filenames per group\ngroups = df[df.Type == 'depth.png'].groupby('Temp')\n\nfor t, group in groups:\n    print('Processing temperature {}'.format(t))\n    for n in group['Name']:\n        print('  Found {}'.format(n))\n```\n\nThen use your favorite libraries to process the images.\n\n## Code\n\nThe code contains the necessary tools to train and predict pixel-wise dense depth corrections. The CPU part contains two separate implementations, one based on `sklearn` and a minimal standalone regressor that depends on `numpy`. The GPU implementation is based on [TensorFlow](https://www.tensorflow.org/).\n\n### Preprocessing data\n\nIf you are working with `rgbd-correction-raw.zip` you will first need to preprocess\nthe data by \n\n```\npython -m sensor_correction.apps.preprocess_depth --crop 20 --unitscale 0.001 \u003cpath-to-index.csv\u003e\n\nProcessing temperature 10\n  Processing position 700\n  Processing position 800\n  Processing position 900\n  Processing position 1000\n  Processing position 1100\n  Processing position 1200\nProcessing temperature 11\n  Processing position 700\n  Processing position 800\n  Processing position 900\n...\n```\n\nThis will result in a file named `input_depths.npz` that will be used next. In case you are working with `rgbd-correction-mini.zip` you can skip this step.\n\n### Fitting a Gaussian Process regressor\n\nTo train a GP regressor on preprocessed input data, type\n\n```\npython -m sensor_correction.apps.train input_depths.npz \u003cpath-to-intrinsics.txt\u003e\nRMSE 1.405188e-02\n\nOptimized length scale [  2.41625833  55.406589     0.36033164  99.50966992]\nOptimized signal std 0.5018155580645485\nOptimized noise std 0.0316227766016838\n```\n\nThe script automatically selects training data from all test data and optimizes the kernel hyper-parameters. This could take a while, be patient. Once completed a `gpr.pkl` should be generated.\n\n### Correcting depth maps\n\nTo correct depth maps on CPU use\n\n```\npython -m sensor_correction.apps.correct_depth gpr.pkl input_depths.npz  \u003cpath-to-intrinsics.txt\u003e\n```\n\nor for GPU accelerated computations, type\n\n```\npython -m sensor_correction.apps.correct_depth gpr.pkl input_depths.npz  \u003cpath-to-intrinsics.txt\u003e --gpu\n```\n\nDepending on your GPU model you should be able to see a significant speed-up compared to the CPU variant. Please note that the released GPU code does not contain the final optimized instruction set and thus might run a bit slower. \n\nOnce completed you should see a `corrected_depths.npz` file.\n\n### Plotting results\n\nTo plot correction results, use \n\n```\npython -m sensor_correction.apps.plot_corrected_depth input_depths.npz corrected_depths.npz\n```\n\nwhich should give results similar to\n\n![](etc/correction_t17_p1200.png)\n\n## Acknowledgements\nThis research is funded by the projects Lern4MRK (Austrian Ministry for Transport, Innovation and Technology), and AssistMe (FFG, 848653), as well as the European Union in cooperation with the State of Upper Austria within the project Investition in Wachstum und Beschäftigung (IWB).\n\nCode and dataset created by members of [PROFACTOR Gmbh](http://www.profactor.at).\n\n## Cite us\nIf you use our dataset / code in your research and would like to cite us, we suggest the following BibTeX format.\n\n```\n@InProceedings{,\n author    = {Heindl,Christoph and Poenitz, Thomas and Stuebl,Gernot and Pichler, Andreas and Scharinger, Josef},\n title     = {Spatio-thermal depth correction of {RGB-D} sensors based on {G}aussian {P}rocesses in real-time},\n booktitle = {The 10th International Conference on Machine Vision, to be published},\n year      = {2017},\n note      = {to be published},\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheind%2Frgbd-correction","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcheind%2Frgbd-correction","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheind%2Frgbd-correction/lists"}