{"id":13704794,"url":"https://github.com/detectRecog/PointTrack","last_synced_at":"2025-05-05T12:32:18.351Z","repository":{"id":37626683,"uuid":"245849780","full_name":"detectRecog/PointTrack","owner":"detectRecog","description":"PointTrack (ECCV2020 ORAL): Segment as Points for Efficient Online Multi-Object Tracking and Segmentation","archived":false,"fork":false,"pushed_at":"2023-10-03T21:43:04.000Z","size":4377,"stargazers_count":261,"open_issues_count":19,"forks_count":45,"subscribers_count":17,"default_branch":"master","last_synced_at":"2024-08-03T22:14:03.640Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/detectRecog.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-03-08T16:36:51.000Z","updated_at":"2024-07-12T07:54:28.000Z","dependencies_parsed_at":"2023-01-20T22:16:18.905Z","dependency_job_id":"3b109acb-12f1-4289-8767-9f41c4616b02","html_url":"https://github.com/detectRecog/PointTrack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/detectRecog%2FPointTrack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/detectRecog%2FPointTrack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/detectRecog%2FPointTrack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/detectRecog%2FPointTrack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/detectRecog","download_url":"https://codeload.github.com/detectRecog/PointTrack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224448679,"owners_count":17313096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T22:00:17.154Z","updated_at":"2024-11-13T12:30:56.861Z","avatar_url":"https://github.com/detectRecog.png","language":"Python","funding_links":[],"categories":["算法论文"],"sub_categories":["**2020**"],"readme":"## PointTrack(ECCV2020 ORAL): Segment as Points for Efficient Online Multi-Object Tracking and Segmentation\n\n## APOLLO MOTS (v1) is currently available. [APOLLO MOTS (code: vqp8)](https://pan.baidu.com/s/1kCK6tgoYEk-mnogMa94ZMQ).\n\nThis codebase implements **PointTrack** (and its extention **PointTrackV2**), a highly effective framework for multi-object tracking and segmentation (MOTS) described in: \n\n```\n@inproceedings{xu2020Segment,\n  title={Segment as Points for Efficient Online Multi-Object Tracking and Segmentation},\n  author={Xu, Zhenbo and Zhang, Wei and Tan, Xiao and Yang, Wei and Huang, Huan and Wen, Shilei and Ding, Errui and Huang, Liusheng},\n  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},\n  year={2020}\n}\n@article{xu2021segment,\n  title={Segment as Points for Efficient and Effective Online Multi-Object Tracking and Segmentation},\n  author={Xu, Zhenbo and Yang, Wei and Zhang, Wei and Tan, Xiao and Huang, Huan and Huang, Liusheng},\n  journal={IEEE Transactions on Pattern Analysis \\\u0026 Machine Intelligence},\n  number={01},\n  pages={1--1},\n  year={2021},\n  publisher={IEEE Computer Society}\n}\n```\n\nPointTrackV2 is accepted to TPAMI. The code is available in https://github.com/detectRecog/CCP.\n\n**PointTrack presents a new learning strategy for pixel-wise feature learning on the 2D image plane, which has proven to be effective for instance association.**\n\nOur network architecture adopts [SpatialEmbedding](https://github.com/davyneven/SpatialEmbeddings) as the segmentation sub-network. \nThe current ranking of PointTrack is available in [KITTI leader-board](http://www.cvlibs.net/datasets/kitti/eval_mots.php). Until now (07/03/2020), PointTrack++ still ranks first for both cars and pedestrians.\nThe detailed task description of MOTS is avaliable in [MOTS challenge](https://www.vision.rwth-aachen.de/page/mots).  \n\n\n## Getting started\n\nThis codebase showcases the proposed framework named PointTrack for MOTS using the KITTI MOTS dataset. \n\n### Prerequisites\nDependencies: \n- Pytorch 1.3.1 (and others), please set up an virtual env and run:\n```\n$ pip install -r requirements.txt\n```\n- Python 3.6 (or higher)\n- [KITTI Images](http://www.cvlibs.net/download.php?file=data_tracking_image_2.zip) + [Annotations](https://www.vision.rwth-aachen.de/media/resource_files/instances.zip)\n\nNote that the scripts for evaluation is included in this repo. After images and instances (annotations) are downloaded, put them under **kittiRoot** and change the path in **repoRoot**/config.py accordingly. \nThe structure under **kittiRoot** should looks like:\n\n```\nkittiRoot\n│   images -\u003e training/image_02/ \n│   instances\n│   │    0000\n│   │    0001\n│   │    ...\n│   training\n│   │   image_02\n│   │   │    0000\n│   │   │    0001\n│   │   │    ...  \n│   testing\n│   │   image_02\n│   │   │    0000\n│   │   │    0001\n│   │   │    ... \n```\n\n## Pretrained weights\nWe provide fine-tuned models on KITTI MOTS (for cars):\n- SpatialEmbedding for cars.\n- PointTrack for cars.\nYou can download them via [Baidu Disk](https://pan.baidu.com/s/1Mk9JWNcM1W08EAjhyq0yLA) or [Google Drive](https://drive.google.com/open?id=14Hn4ZztfjGUYEjVd-9FRNB5a-CtBkPXc).\n- testset segs can be found in the folder 'testset_segs'.\n\n\n## Testing\n\nYou can download pretrained models from the above links. Save these weight file under **repoRoot**/pointTrack_weights.\n\n1.To generate the instance segmentation results:\n\n```\n$ python -u test_mots_se.py car_test_se_to_save\n```\nThe segmentation result will be saved according to the config file **repoRoot**/config_mots/car_test_se_to_save.py.\n\n2.To test PointTrack on the instance segmentation results:\n```\n$ python -u test_tracking.py car_test_tracking_val\n```\n\nThe pretrained model gets 85.51 sMOTSA for cars on the validation set. \n\n\n## Training of PointTrack\nThe training procedure of instance association is as follows.\n\n1.To generate the segmentation result on the validation set as the instruction of the first step in Testing.\n\n2.To generate the instance DB from videos:\n```\n$ python -u datasets/MOTSInstanceMaskPool.py\n```\n\n3.Change the line which loads weights to the default path as follows:\n```\ncheckpoint_path='./pointTrack_weights/PointTrack.pthCar' --\u003e checkpoint_path='./car_finetune_tracking/checkpoint.pth'\n```\n\n4.Afterwards start training:\n```\n$ python -u train_tracker_with_val.py car_finetune_tracking\n```\nThe best tracker on the validation set will be saved under the folder specified in **repoRoot**/config_mots/car_finetune_tracking.py.\n\n\n## Training of SpatialEmbedding\n\nNote that the training of SpatialEmbedding needs KITTI object detection left color images as well as the KINS annotations.\nPlease download images from [KITTI dataset](http://www.cvlibs.net/download.php?file=data_object_image_2.zip), and unzip the zip file under **kittiRoot**.\nPlease download two KINS annotation json files from [instances_train.json,instances_val.json](https://github.com/qqlu/Amodal-Instance-Segmentation-through-KINS-Dataset), and put files under **kittiRoot**.\n\nFor the training of SpatialEmbedding, we follow the original training setting of [SpatialEmbedding](https://github.com/davyneven/SpatialEmbeddings). \nDifferent foreground weights are adopted for different classes (200 for cars and 50 for pedestrians). In this following, we take cars for example to explain training procedures. \n\n0.As there are many in-valid frames in MOTS that contain no cars, we only select these valid frames for training SpatialEmbedding.\n ```\n$ python -u datasets/MOTSImageSelect.py\n ```\n\n1.To parse KINS annotations, run:\n```\n$ python -u datasets/ParseKINSInstance.py\n```\nAfter this step, KINS annotations are saved under **kittiRoot**/training/KINS/ and **kittiRoot**/testing/KINS/.\n\n2.To generate these crops do the following:\n```\n$ python -u utils/generate_crops.py\n```\nAfter this step, crops are saved under **kittiRoot**/crop_KINS. (roughly 92909 crops)\n\n3.Afterwards start training on crops: \n```\n$ python -u train_SE.py car_finetune_SE_crop\n```\n\n4.Afterwards start finetuning on KITTI MOTS with BN fixed:\n```\n$ python -u train_SE.py car_finetune_SE_mots\n```\n\n\n## Cite us\nWe borrow some code from [SpatialEmebdding](https://github.com/davyneven/SpatialEmbeddings).\n```\n@inproceedings{xu2020Segment,\n  title={Segment as Points for Efficient Online Multi-Object Tracking and Segmentation},\n  author={Xu, Zhenbo and Zhang, Wei and Tan, Xiao and Yang, Wei and Huang, Huan and Wen, Shilei and Ding, Errui and Huang, Liusheng},\n  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},\n  year={2020}\n}\n@inproceedings{xupointtrack++,\n  title={PointTrack++ for Effective Online Multi-Object Tracking and Segmentation},\n  author={Xu, Zhenbo and Zhang, Wei and Tan, Xiao and Yang, Wei and Su, Xiangbo and Yuan, Yuchen and Zhang, Hongwu and Wen, Shilei and Ding, Errui and Huang, Liusheng},\n  booktitle={CVPR Workshops},\n  year={2020}\n}\n```\n\n## Q\u0026A\n1. Where is the APOLLO MOTS?\n- As suggested by ECCV reviewers, APOLLO MOTS should be revised before publish.\n\n2. Why is the training of **Temporal Seed Consistency** not included in this repo?\n- In PointTrack++, we find that the improvement of the temporal seed consistency is minor. Therefore, in this repo, we provide this simplified version for easy usage.\n\n3. The schedule of PointTrack++.\n- We are working on cleaning the code of PointTrack++. It will be publicly available soon.\n\n## Contact\nIf you find problems in the code, please open an issue.\n\nFor general questions, please contact the corresponding author Wei Yang (qubit@ustc.edu.cn).\n\n\n## License\n\nThis software is released under a creative commons license which allows for personal and research use only. For a commercial license please contact the authors. You can view a license summary [here](http://creativecommons.org/licenses/by-nc/4.0/).\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FdetectRecog%2FPointTrack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FdetectRecog%2FPointTrack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FdetectRecog%2FPointTrack/lists"}