{"id":19855187,"url":"https://github.com/leggedrobotics/raw_image_pipeline","last_synced_at":"2025-05-02T01:30:46.874Z","repository":{"id":165514026,"uuid":"599348036","full_name":"leggedrobotics/raw_image_pipeline","owner":"leggedrobotics","description":"Image processing pipeline for cameras that provide raw data","archived":false,"fork":false,"pushed_at":"2024-04-24T14:04:16.000Z","size":3439,"stargazers_count":43,"open_issues_count":0,"forks_count":11,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-25T13:58:55.358Z","etag":null,"topics":["debayer","gamma-correction","image-proc","image-processing","undistortion","vignetting","white-balance"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leggedrobotics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-09T00:19:40.000Z","updated_at":"2025-04-04T04:01:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"8b52265f-1011-4ff1-94a3-3faca14acda9","html_url":"https://github.com/leggedrobotics/raw_image_pipeline","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Fraw_image_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Fraw_image_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Fraw_image_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Fraw_image_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leggedrobotics","download_url":"https://codeload.github.com/leggedrobotics/raw_image_pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251969274,"owners_count":21673183,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["debayer","gamma-correction","image-proc","image-processing","undistortion","vignetting","white-balance"],"created_at":"2024-11-12T14:11:59.266Z","updated_at":"2025-05-02T01:30:46.868Z","avatar_url":"https://github.com/leggedrobotics.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RAW Image Pipeline\n\nImage processing utilities used for cameras that provide RAW data, such as the Alphasense Core unit.\n\n**Maintainers:** Matias Mattamala (matias@robots.ox.ac.uk)\n\n**Contributors:** Matias Mattamala, Timon Homberger, Marco Tranzatto, Samuel Zimmermann, Lorenz Wellhausen, Shehryar Khattak, Gabriel Waibel\n\n\u003cimg src=\"docs/raw_image_pipeline.png\" alt=\"raw_image_pipeline overview\" width=\"1000\"/\u003e\n\n\n## License\nThis source code is released under a [MIT License](LICENSE).\n\n`raw_image_pipeline_white_balance` relies on Shane Yuan's [AutoWhiteBalance package](https://github.com/yuanxy92/AutoWhiteBalance) licensed under [GNU](raw_image_pipeline_white_balance/LICENSE).\n\n`raw_image_pipeline_python` relies on Pascal Thomet's [cvnp](https://github.com/pthom/cvnp), licensed under [MIT](raw_image_pipeline_python/thirdparty/cvnp/LICENSE) as well.\n\n\n## Build\n\n[![Build Status](https://ci.leggedrobotics.com/buildStatus/icon?job=bitbucket_leggedrobotics/alphasense_rsl/master)](https://ci.leggedrobotics.com/job/bitbucket_leggedrobotics/job/alphasense_rsl/job/master/)\n\n## Overview\n### Packages\n1. **`raw_image_pipeline`**: ROS-independent implementation of the pipeline.\n2. **`raw_image_pipeline_python`**: Python bindings for `raw_image_pipeline`.\n3. **`raw_image_pipeline_ros`**: ROS interface to run the processing pipeline.\n4. **`raw_image_pipeline_white_balance`**: Additional white balance algorithm built upon Shane Yuan's [code](https://github.com/yuanxy92/AutoWhiteBalance), based on Barron's ([1](https://arxiv.org/abs/1507.00410), [2](https://arxiv.org/abs/1611.07596)).\n\n## Pipeline\n\nThe package implements diferent modules that are chained together to process the RAW images. Each can be disabled and the image will be processed by the subsequent modules.\n* **Debayer**: `auto`, `bayer_bggr8`, `bayer_gbrg8`, `bayer_grbg8`, `bayer_rggb8`\n* **Flip**: Flips the image 180 degrees\n* **White balance**: `simple`, `grey_world`, `learned` (from [OpenCV](https://docs.opencv.org/4.x/df/db9/namespacecv_1_1xphoto.html)), `ccc` (from `raw_image_pipeline_white_balance` package), `pca` (custom implementation)\n* **Color correction**: Simple color correction based on a mixing BGR 3x3 matrix.\n* **Gamma correction**: `default` (from OpenCV), `custom` (custom implementation)\n* **Vignetting correction**: Removes the darkening effect of the lens toward the edges of the image by applying a polynomial mask.\n* **Color enhancement**: Converts the image to HSV and applies a gain to the S (saturation) channel.\n* **Undistortion**: Corrects the image given the camera calibration file.\n\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\nDetailed modules description\n\u003c/summary\u003e\n\n### Debayer Module\nThis module demosaics a Bayer-encoded image into BGR values (following OpenCV's convention). It relies on OpenCV's methods for both CPU and GPU.\n\n**Parameters**\n- `debayer/enabled`: Enables the module. `True` by default\n- `debayer/encoding`: Encoding of the incoming image. `auto` uses the encoding field of the ROS message\n  - Values: `auto`, `bayer_bggr8`, `bayer_gbrg8`, `bayer_grbg8`, `bayer_rggb8`\n\n### Flip\nThis flips the image 180 degrees. Just that.\n\n**Parameters**\n- `debayer/enabled`: Enables the module. `False` by default\n\n### White Balance\nIt automatically corrects white balance using different available algorithms.\n\n**Parameters**\n- `white_balance/enabled`: Enables the module. `False` by default\n- `white_balance/method`: To select the method used for automatic white balance\n  - `simple`: from [OpenCV](https://docs.opencv.org/4.x/df/db9/namespacecv_1_1xphoto.html). Tends to saturate colors depending on the clipping percentile.\n  - `grey_world`: from [OpenCV](https://docs.opencv.org/4.x/df/db9/namespacecv_1_1xphoto.html)\n  - `learned`: from [OpenCV](https://docs.opencv.org/4.x/df/db9/namespacecv_1_1xphoto.html)\n  - `ccc`: from `raw_image_pipeline_white_balance` package\n  - `pca`: custom implementation\n- `white_balance/clipping_percentile`: Used in `simple` method\n  - Values: between 0 and 100\n- `white_balance/saturation_bright_thr`: Used in `grey_world`, `learned` and `ccc` methods\n  - Values: Between 1.0 and 0.0. E.g. 0.8 means that all the values above 0.8*255 (8 bit images) are discarded for white balance determination.\n- `white_balance/clipping_percentile`: Used in `simple` method\n  - Values: Between 1.0 and 0.0. E.g. 0.2 means that all the values below 0.2*255 (8 bit images) are discarded.\n- `white_balance/temporal_consistency`: Only for `ccc`. `False` by default. It uses a Kalman filter to smooth the illuminant estimate.\n\n### Color calibration\nIt applies a fixed affine transformation to each BGR pixel to mix and add a bias term to the color channels. It can be a replacement for the white balance module.\n\n- `color_calibration/enabled`: Enables the module. `False` by default\n- `color_calibration/calibration_file`: A YAML file with the color calibration matrix and bias. Example [file](raw_image_pipeline/config/alphasense_color_calib_example.yaml). This file can be obtained using a script in the `raw_image_pipeline_python` package: [`color_calibration.py`](raw_image_pipeline_python/scripts/color_calibration.py). To run it, we require a set of images capturing a calibration board ([example](raw_image_pipeline_python/config/color_calibration_checker.pdf)): a reference image `ref.png` ([example](raw_image_pipeline_python/config/color_ref.jpg)) and a collection of images from the camera to be calibrated. The usage is:\n\n```sh\ncolor_calibration.py [-h] -i INPUT -r REF [-o OUTPUT_PATH] [-p PREFIX] [--loss LOSS] [--compute_bias]\n\nPerforms color calibration between 2 images, using ArUco 4X4\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -i INPUT, --input INPUT\n                        Input image (to be calibrated), or folder with reference images\n  -r REF, --ref REF     Reference image to perform the calibration\n  -o OUTPUT_PATH, --output_path OUTPUT_PATH\n                        Output path to store the file. Default: current path\n  -p PREFIX, --prefix PREFIX\n                        Prefix for the calibration file. Default: none\n  --loss LOSS           Loss used in the optimization. Options: 'linear', 'soft_l1', 'huber', 'cauchy', 'arctan'\n  --compute_bias        If bias should be computed\n```\n\n\u003e :warning: This feature is experimental and it is not recommended for 'serious' applications.\n\n### Gamma correction\nIt applies a gamma correction to improve illumination.\n\n**Parameters**\n- `gamma_correction/enabled`: Enables the module. `False` by default\n- `gamma_correction/method`: To select the method used for automatic white balance\n  - `default`: correction from OpenCV (CUDA only)\n  - `custom`: a custom implementation based on a look-up table.\n-  `gamma_correction/k`: Gamma factor: \u003e1 is a forward gamma correction that makes the image darker; \u003c1 is an inverse correction that increases brightness.\n### Vignetting correction\nIt applies a polynomial illumination compensation to overcome the barrel effect of wide-angle lenses:\n`s * (r^2 * a2 + r^4 * a4)`, with `r` the distance to the image center.\n\n**Parameters**\n- `vignetting_correction/enabled`: Enables the module. `False` by default\n- `vignetting_correction/scale`: s value\n- `vignetting_correction/a2`: 2nd-order factor\n- `vignetting_correction/a4`: 4th-order factor\n\n\u003e :warning: This feature is experimental and it is not recommended for 'serious' applications.\n\n### Color enhancement\nIt increases the saturation of the colors by transforming the image to HSV and applying a linear gain.\n\n**Parameters**\n- `vignetting_correction/enabled`: Enables the module. `False` by default\n- `vignetting_correction/saturation_gain`: A factor to increase the saturation channel of the HSV channel.\n\n\n### Undistortion\nIt undistorts the image follow a pinhole model. It requires the intrinsic calibration from Kalibr.\n\n- `color_calibration/enabled`: Enables the module. `False` by default\n- `color_calibration/calibration_file`: Camera calibration from Kalibr, following the format of the [example file](raw_image_pipeline/config/alphasense_calib_1.6mp_example.yaml)\n\n\u003c/details\u003e\n\n\n## Requirements and compilation\n### Dependencies\n\n```sh\nsudo apt install libyaml-cpp-dev\n```\n\n```sh\ncd ~/git\ngit clone git@github.com:catkin/catkin_simple.git\ngit clone git@github.com:ethz-asl/glog_catkin.git\ngit clone git@github.com:leggedrobotics/pybind11_catkin.git\ncd ~/catkin_ws/src\nln -s ../../git/catkin_simple .\nln -s ../../git/glog_catkin .\nln -s ../../git/pybind11_catkin .\n```\n\nIf you need CUDA support, you need to build OpenCV with CUDA. Check the [instructions below](#cuda-support)\n\n### Build raw_image_pipeline_ros\n\nTo build the ROS package:\n```sh\ncatkin build raw_image_pipeline_ros\n```\n\nIf you also need the Python bindings, run:\n```sh\ncatkin build raw_image_pipeline_python\n```\n\n### CUDA support\nIf you are using a Jetson or another GPU-enabled computer and want to exploit the GPU, you need to compile OpenCV with CUDA support. Clone the [opencv_catkin](https://github.com/ori-drs/opencv_catkin) package, which setups **OpenCV 4.2 by default**.\n```sh\ncd ~/git\ngit clone git@github.com:ori-drs/opencv_catkin.git\ncd ~/catkin_ws/src\nln -s ../../git/opencv_catkin .\ncd ~/catkin_ws\n```\n\n\u003e :warning: Before compiling, you need to confirm the _compute capability_ of your NVidia GPU, which you can check in [this website](https://developer.nvidia.com/cuda-gpus) or the [CUDA wikipedia page](https://en.wikipedia.org/wiki/CUDA).\n\n#### Compilation on Jetson Xavier board (compute capability 7.2)\n```\ncatkin build opencv_catkin --cmake-args -DCUDA_ARCH_BIN=7.2\nsource devel/setup.bash\n```\n\n#### Compilation on Jetson Orin board (compute capability 8.7)\n```\ncatkin build opencv_catkin --cmake-args -DCUDA_ARCH_BIN=8.7\nsource devel/setup.bash\n```\n\n#### Compilation on other platforms (e.g. laptops, desktops)\nThere are some extra considerations if you plan to compile OpenCV with CUDA in your working laptop/desktop:\n1. **Compute capability may be different for your GPU:** Please check the aforementioned websites to set the flag correctly.\n2. **The opencv_catkin default flags are the minimum:** Graphical support libraries (such as GTK) are disabled, so you cannot use methods such as `cv::imshow`. If you want to enable it, you can check the flags in the CMakeLists of `opencv_catkin`\n3. **Default OpenCV version is 4.2:** The package installs by default OpenCV 4.2, which was the version compatible with ROS melodic. This can be changed by modyfing the CMakeLists of `opencv_catkin` as well.\n\nOpenCV's compilation will take a while - get a coffee in the meantime. When it's done, you can rebuild `raw_image_pipeline_ros`.\n\n\n#### Troubleshooting\n* If you get errors due to `glog`, remove `glog_catkin`, compile `opencv_catkin` using the system's glog, and _then_ build `raw_image_pipeline_ros` (which will compile `glog_catkin`)\n* If OpenCV fails due to CUDA errors, confirm that you compiled using the right compute capability for your GPU.\n* If you are using older versions of CUDA (10.x and before), they may require older GCC versions. For example, to use GCC 7 you can use:\n``` sh\ncatkin build opencv_catkin --cmake-args -DCUDA_ARCH_BIN=\u003cyour_compute_capability\u003e -DCMAKE_C_COMPILER=/usr/bin/gcc-7\n```\n\n## Run the node\nTo run, we use the same launch file as before:\n```sh\nroslaunch raw_image_pipeline_ros raw_image_pipeline_node.launch\n```\n\nThis launchfile was setup for Alphasense cameras. The parameters can be inspected in the [launch file itself](raw_image_pipeline_ros/launch/raw_image_pipeline_node.launch).\n\n## Alphasense-specific info\n### Setup\nPlease refer to [**Alphasense Setup**](docs/alphasense_setup.md) for the instructions to setup the host PC where the Alphasense will be connected.\nFor further information you can refer the [official manual](https://github.com/sevensense-robotics/alphasense_core_manual/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleggedrobotics%2Fraw_image_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleggedrobotics%2Fraw_image_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleggedrobotics%2Fraw_image_pipeline/lists"}