{"id":18792431,"url":"https://github.com/prbonn/bonnet","last_synced_at":"2025-04-06T10:12:24.639Z","repository":{"id":39739393,"uuid":"117097478","full_name":"PRBonn/bonnet","owner":"PRBonn","description":"Bonnet: An Open-Source Training and Deployment Framework for Semantic Segmentation in Robotics.","archived":false,"fork":false,"pushed_at":"2022-11-21T22:20:15.000Z","size":182,"stargazers_count":325,"open_issues_count":19,"forks_count":89,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-03-30T08:12:44.878Z","etag":null,"topics":["convolutional-neural-networks","robotics","ros","semantic-segmentation","tensorflow","tensorrt"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PRBonn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-11T12:29:14.000Z","updated_at":"2024-11-17T09:43:21.000Z","dependencies_parsed_at":"2022-08-28T06:40:19.894Z","dependency_job_id":null,"html_url":"https://github.com/PRBonn/bonnet","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fbonnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fbonnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fbonnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fbonnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PRBonn","download_url":"https://codeload.github.com/PRBonn/bonnet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247464222,"owners_count":20942970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convolutional-neural-networks","robotics","ros","semantic-segmentation","tensorflow","tensorrt"],"created_at":"2024-11-07T21:19:51.955Z","updated_at":"2025-04-06T10:12:24.623Z","avatar_url":"https://github.com/PRBonn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bonnet: An Open-Source Training and Deployment Framework for Semantic Segmentation in Robotics.\n\n[![Build Status](https://travis-ci.org/PRBonn/bonnet.svg?branch=master)](https://travis-ci.org/PRBonn/bonnet)\n\nBy [Andres Milioto](http://www.ipb.uni-bonn.de/people/andres-milioto/) @ University of Bonn.\n\n**__(for the new Pytorch version, go [here](https://github.com/PRBonn/bonnetal))__**\n\n![Image of cityscapes](https://image.ibb.co/i5tEQR/CITY.png)\nCityscapes Urban Scene understanding.\n- [Standalone Video Predictor - 512x256px](https://youtu.be/RXK_eYO_i08)\n- [Standalone Video Predictor - 768x384px (inception model)](https://youtu.be/3qD-GR87usw)\n- [Standalone Video Predictor - 1024x512px](https://youtu.be/tfeFHCq6YJs)\n\n![Image of Persons](https://image.ibb.co/nJRa3c/github_bruno.png)\nPerson Segmentation\n- [Uptown funk (inception model)](https://photos.app.goo.gl/Ls3MkCo2Cwhc6pVT2)\n- [Can't stop the feeling (inception model)](https://photos.app.goo.gl/Mi4N4zLrTyyDRqv48)\n\n\n![Image of cwc](https://image.ibb.co/fcKXC6/CWC.png)\nCrop vs. Weed Semantic Segmentation.\n- [ROS node prediction - Video](https://youtu.be/-XgxiC04hUI)\n\n\n## Description\n\nThis code provides a framework to easily add architectures and datasets, in order to \ntrain and deploy CNNs for a robot. It contains a full training pipeline in python\nusing Tensorflow and OpenCV, and it also some C++ apps to deploy a frozen\nprotobuf in ROS and standalone. The C++ library is made in a way which allows to\nadd other backends (such as TensorRT and MvNCS), but only Tensorflow and TensorRT\nare implemented for now. For now, we will keep it this way because we are mostly\ninterested in deployment for the Jetson and Drive platforms, but if you have a specific\nneed, we accept pull requests!\n\nThe networks included is based of of many other architectures\n(see below), but not exactly a copy of any of them. As seen in the videos, they run very fast in\nboth GPU and CPU, and they are designed with performance in mind, at the cost of\na slight accuracy loss. Feel free to use it as a model to implement your own\narchitecture.\n\nAll scripts have been tested on the following configurations:\n- x86 Ubuntu 16.04 with an NVIDIA GeForce 940MX GPU (nvidia-384, CUDA9, CUDNN7, TF 1.7, TensorRT3)\n- x86 Ubuntu 16.04 with an NVIDIA GTX1080Ti GPU (nvidia-375, CUDA9, CUDNN7, TF 1.7, TensorRT3)\n- x86 Ubuntu 16.04 and 14.04 with no GPU (TF 1.7, running on CPU in NHWC mode, no TensorRT support)\n- Jetson TX2 (full Jetpack 3.2)\n\nWe also provide a Dockerfile to make it easy to run without worrying about the dependencies, which is based on the official nvidia/cuda image containing cuda9 and cudnn7. In order to build and run this image with support for X11 (to display the results), you can run this in the repo root directory ([nvidia-docker](https://github.com/NVIDIA/nvidia-docker) should be used instead of vainilla docker):\n\n```sh\n  $ docker pull tano297/bonnet:cuda9-cudnn7-tf17-trt304\n  $ nvidia-docker build -t bonnet .\n  $ nvidia-docker run -ti --rm -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v $HOME/.Xauthority:/home/developer/.Xauthority -v /home/$USER/data:/shared --net=host --pid=host --ipc=host bonnet /bin/bash\n```\n\n_-v /home/$USER/data:/share_ can be replaced to point to wherever you store the data and trained models, in order to include the data inside the container for inference/training.\n\n#### Deployment\n\n- _/deploy_cpp_ contains C++ code for deployment on robot of the full pipeline,\nwhich takes an image as input and produces the pixel-wise predictions\nas output, and the color masks (which depend on the problem). It includes both\nstandalone operation which is meant as an example of usage and build, and a ROS\nnode which takes a topic with an image and outputs 2 topics with the labeled mask\nand the colored labeled mask.\n\n- Readme [here](deploy_cpp/README.md)\n\n#### Training\n\n- _/train_py_ contains Python code to easily build CNN Graphs in Tensorflow,\ntrain, and generate the trained models used for deployment. This way the\ninterface with Tensorflow can use the more complete Python API and we can easily\nwork with files to augment datasets and so on. It also contains some apps for using\nmodels, which includes the ability to save and use a frozen protobuf, and to use\nthe network using TensorRT, which reduces the time for inference when using NVIDIA\nGPUs.\n\n- Readme [here](train_py/README.md)\n\n#### Pre-trained models\n\nThese are some models trained on some sample datasets that you can use with the trainer and deployer,\nbut if you want to take time to write the parsers for another dataset (yaml file with classes and colors + python script to\nput the data into the standard dataset format) feel free to create a pull request.\n\nIf you don't have GPUs and the task is interesting for robots to exploit, I will\ngladly train it whenever I have some free GPU time in our servers.\n\n- Cityscapes:\n  - 512x256 [Link](http://www.ipb.uni-bonn.de/html/projects/bonnet/pretrained-models/v0.2/city_512.tar.gz)\n  - 768x384 [Link](http://www.ipb.uni-bonn.de/html/projects/bonnet/pretrained-models/v0.2/city_768_inception.tar.gz) (inception-like model)\n  - 768x384 [Link](http://www.ipb.uni-bonn.de/html/projects/bonnet/pretrained-models/v0.2/city_768_mobilenets.tar.gz) (mobilenets-like model)\n  - 1024x512 [Link](http://www.ipb.uni-bonn.de/html/projects/bonnet/pretrained-models/v0.2/city_1024.tar.gz)\n- Synthia:\n  - 512x384 [Link](http://www.ipb.uni-bonn.de/html/projects/bonnet/pretrained-models/v0.2/synthia_512.tar.gz)\n  - 960x720 [Link](http://www.ipb.uni-bonn.de/html/projects/bonnet/pretrained-models/v0.2/synthia_960.tar.gz)\n\n- Persons (+coco people):\n  - 512x512 [Link](http://www.ipb.uni-bonn.de/html/projects/bonnet/pretrained-models/v0.2/persons_512.tar.gz)\n\n- Crop-Weed (CWC):\n  - 512x384 [Link](http://www.ipb.uni-bonn.de/html/projects/bonnet/pretrained-models/v0.2/cwc_512.tar.gz)\n\n## License\n\n#### This software\n\nBonnet is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nBonnet is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\n#### Pretrained models\n\nThe pretrained models with a specific dataset keep the copyright of such dataset.\n\n- Cityscapes: [Link](https://www.cityscapes-dataset.com)\n- Synthia: [Link](http://synthia-dataset.net)\n- Persons: [Link](https://supervise.ly)\n- Coco: [Link](http://cocodataset.org/#home)\n- Crop-Weed (CWC): [Link](http://www.ipb.uni-bonn.de/data/sugarbeets2016/)\n\n## Citation\n\nIf you use our framework for any academic work, please cite its [paper](http://www.ipb.uni-bonn.de/wp-content/papercite-data/pdf/milioto2019icra.pdf).\n\n```\n@InProceedings{milioto2019icra,\nauthor = {A. Milioto and C. Stachniss},\ntitle = {{Bonnet: An Open-Source Training and Deployment Framework for Semantic Segmentation in Robotics using CNNs}},\nbooktitle = {Proc. of the IEEE Intl. Conf. on Robotics \\\u0026 Automation (ICRA)},\nyear = 2019,\ncodeurl = {https://github.com/Photogrammetry-Robotics-Bonn/bonnet},\nvideourl = {https://www.youtube.com/watch?v=tfeFHCq6YJs},\n}\n```\n\nOur networks are strongly based on the following architectures, so if you\nuse them for any academic work, please give a look at their papers and cite them\nif you think proper:\n\n- SegNet: [Link](https://arxiv.org/abs/1511.00561)\n- E-Net: [Link](https://arxiv.org/abs/1606.02147)\n- ERFNet: [Link](http://www.robesafe.uah.es/personal/eduardo.romera/pdfs/Romera17tits.pdf)\n- PSPNet [Link](https://arxiv.org/abs/1612.01105)\n\n## Other useful GitHub's:\n- [OpenAI Checkpointed Gradients](https://github.com/openai/gradient-checkpointing). Useful\nimplementation of checkpointed gradients to be able to fit big models in GPU memory without sacrificing\nruntime.\n- [Queueing tool](https://github.com/alexanderrichard/queueing-tool): Very nice\nqueueing tool to share GPU, CPU and Memory resources in a multi-GPU environment.\n- [Tensorflow_cc](https://github.com/FloopCZ/tensorflow_cc): Very useful repo\nto compile Tensorflow either as a shared or static library using CMake, in order\nto be able to compile our C++ apps against it.\n\n## Contributors\n\nMilioto, Andres\n- [University of Bonn](http://www.ipb.uni-bonn.de/people/andres-milioto/)\n- [Linkedin](https://www.linkedin.com/in/amilioto/)\n- [ResearchGate](https://www.researchgate.net/profile/Andres_Milioto)\n- [Google Scholar](https://scholar.google.de/citations?user=LzsKE7IAAAAJ\u0026hl=en)\n\nSpecial thanks to [Philipp Lottes](http://www.ipb.uni-bonn.de/people/philipp-lottes/)\nfor all the work shared during the last year, and to [Olga Vysotka](http://www.ipb.uni-bonn.de/people/olga-vysotska/) and\n[Susanne Wenzel](http://www.ipb.uni-bonn.de/people/susanne-wenzel/) for beta testing the \nframework :)\n\n## Acknowledgements\n\nThis work has partly been supported by the German Research Foundation under Germany's Excellence Strategy, EXC-2070 - 390732324 (PhenoRob). We also thank NVIDIA Corporation for providing a Quadro P6000 GPU partially used to develop this framework.\n\n## TODOs\n\n- Merge [Crop-weed CNN with background knowledge](https://arxiv.org/pdf/1709.06764.pdf) into this repo.\n- Make multi-camera ROS node that exploits batching to make inference faster than sequentially.\n- Movidius Neural Stick C++ backends (plus others as they become available).\n- Inference node to show the classes selectively (e.g. with some qt visual GUI)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2Fbonnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprbonn%2Fbonnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2Fbonnet/lists"}