{"id":20857841,"url":"https://github.com/zju3dv/pvo","last_synced_at":"2025-03-17T14:17:42.827Z","repository":{"id":42969582,"uuid":"495375617","full_name":"zju3dv/PVO","owner":"zju3dv","description":"[CVPR 2023] PVO: Panoptic Visual Odometry","archived":false,"fork":false,"pushed_at":"2023-07-14T06:45:31.000Z","size":53510,"stargazers_count":216,"open_issues_count":14,"forks_count":16,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-03-03T23:49:34.727Z","etag":null,"topics":["panoptic-segmentation","slam","visualodometry"],"latest_commit_sha":null,"homepage":"https://zju3dv.github.io/pvo/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zju3dv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-23T11:10:45.000Z","updated_at":"2025-02-05T17:26:51.000Z","dependencies_parsed_at":"2024-11-18T04:53:35.425Z","dependency_job_id":null,"html_url":"https://github.com/zju3dv/PVO","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FPVO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FPVO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FPVO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FPVO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zju3dv","download_url":"https://codeload.github.com/zju3dv/PVO/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244047646,"owners_count":20389206,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["panoptic-segmentation","slam","visualodometry"],"created_at":"2024-11-18T04:43:25.670Z","updated_at":"2025-03-17T14:17:42.801Z","avatar_url":"https://github.com/zju3dv.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PVO: Panoptic Visual Odometry\n### [Project Page](https://zju3dv.github.io/pvo/) | [Paper](https://arxiv.org/abs/2207.01610)\n\u003cbr/\u003e\n\n\u003e PVO: Panoptic Visual Odometry  \n\n\u003e [[Weicai Ye](https://ywcmaike.github.io/), [Xinyue Lan](https://github.com/siyisan)]\u003csup\u003eCo-Authors\u003c/sup\u003e, [Shuo Chen](https://github.com/Eric3778), [Yuhang Ming](https://github.com/YuhangMing), [Xinyuan Yu](https://github.com/RickyYXY), [Hujun Bao](http://www.cad.zju.edu.cn/home/bao/), [Zhaopeng Cui](https://zhpcui.github.io/), [Guofeng Zhang](http://www.cad.zju.edu.cn/home/gfzhang)\n\n\u003e CVPR 2023\n\n![demo_vid](assets/pvo_teaser.gif)\n\n## Test on vkitti 15-deg-left datasets.\n0) prepare.\nfollow [prepare.md](prepare.md)\n```\nconda activate droidenv\n```\n\n1) generate inital panoptic segmentation.\n```\nsh tools/initial_segmentation.sh  \n```\n\n2) vps-\u003evo，vo Module generate pose, flow and depth.\n```\nsh tools/test_vo_scene.sh  \n```\n\n3) vo-\u003evps, vps Module use flow and depth from vo Module and generate final video panoptic segmentation results and vpq.\n```\nsh tools/test_vps.sh  \n```\n\n## Metrics on Virtual_KITTI2\n|Scene|RMSE|vpq_all/vpq_thing/vpq_stuff|\n|-----|----|---------------------------|\n|Scene01|0.371|40.39/26.43/44.57|\n|Scene02|0.058|68.84/88.83/62.18|\n|Scene06|0.113|66.38/79.99/62.97|\n|Scene18|0.951|68.35/83.86/63.92|\n|Scene20|3.503|35.11/16.83/40.59|\n\nYou can get the results in the paper by iterating multiple times.\n\n## Train on vkitti 15-deg-left datasets.\n1)  To train VPS_Module, you can refer to [Detectron2](https://detectron2.readthedocs.io/en/latest/tutorials/getting_started.html) for more training details.\nHere for example, you can train  vkitti 15-deg-left on 4 GPUs, and training results are saved on `output/vps_training/`. You can modify the config-file according to the hardware conditions.\n```\npython3 -W ignore VPS_Module/tools/train_net.py \\\n--config-file VPS_Module/configs/COCO-PanopticSegmentation/panoptic_fpn_R_50_3x_vkitti_511.yaml --num-gpu 4 \\\nMODEL.WEIGHTS checkpoints/panFPN.pth \\\nOUTPUT_DIR output/vps_training/\n```\n\n2) To train VO_Module, you can refer to [DROID-SLAM](https://github.com/princeton-vl/DROID-SLAM) for more training details.\nHere for example, you can train vkitti on 4 GPUs.\n```\npython VO_Module/train.py --gpus=4 --lr=0.00025\n```\n\n## Visualization\nYou can refer to [DROID-SLAM](https://github.com/princeton-vl/DROID-SLAM) for visualization.\nAll demos can be run on a GPU with 11G of memory. While running, press the \"s\" key to increase the filtering threshold (= more points) and \"a\" to decrease the filtering threshold (= fewer points).\n```\npython VO_Module/evaluation_scripts/test_vo.py --datapath=datasets/Virtual_KITTI2/Scene01 --segm_filter True \n```\n\n## Citation\n\nIf you find this code useful for your research, please use the following BibTeX entry.\n\n```bibtex\n\n@inproceedings{Ye2023PVO,\n  title={{PVO: Panoptic visual odometry}},\n  author={Ye, Weicai and Lan, Xinyue and Chen, Shuo and Ming, Yuhang and Yu, Xingyuan and Bao, Hujun and Cui, Zhaopeng and Zhang, Guofeng},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={9579--9589},\n  year={2023}\n}\n\n```\n\n## Acknowledgement\n\nSome code snippets are borrowed from [DROID-SLAM](https://github.com/princeton-vl/DROID-SLAM) and [Detectron2](https://github.com/facebookresearch/detectron2). Thanks for these great projects.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzju3dv%2Fpvo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzju3dv%2Fpvo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzju3dv%2Fpvo/lists"}