{"id":15028246,"url":"https://github.com/ifzhang/fairmot","last_synced_at":"2025-05-14T16:13:56.524Z","repository":{"id":37456494,"uuid":"252939699","full_name":"ifzhang/FairMOT","owner":"ifzhang","description":"[IJCV-2021] FairMOT: On the Fairness of Detection and Re-Identification in Multi-Object Tracking","archived":false,"fork":false,"pushed_at":"2023-09-19T07:48:24.000Z","size":64887,"stargazers_count":4101,"open_issues_count":357,"forks_count":931,"subscribers_count":83,"default_branch":"master","last_synced_at":"2025-04-12T01:55:35.888Z","etag":null,"topics":["joint-detection-and-tracking","multi-object-tracking","one-shot-tracker","real-time"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ifzhang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-04-04T07:48:48.000Z","updated_at":"2025-04-11T06:27:28.000Z","dependencies_parsed_at":"2022-07-09T19:16:22.263Z","dependency_job_id":"f45ffeca-05d8-4145-8f06-2c79f192eaa1","html_url":"https://github.com/ifzhang/FairMOT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ifzhang%2FFairMOT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ifzhang%2FFairMOT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ifzhang%2FFairMOT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ifzhang%2FFairMOT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ifzhang","download_url":"https://codeload.github.com/ifzhang/FairMOT/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248505867,"owners_count":21115354,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["joint-detection-and-tracking","multi-object-tracking","one-shot-tracker","real-time"],"created_at":"2024-09-24T20:07:53.538Z","updated_at":"2025-04-12T01:55:44.610Z","avatar_url":"https://github.com/ifzhang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FairMOT\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/a-simple-baseline-for-multi-object-tracking/multi-object-tracking-on-2dmot15-1)](https://paperswithcode.com/sota/multi-object-tracking-on-2dmot15-1?p=a-simple-baseline-for-multi-object-tracking)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/a-simple-baseline-for-multi-object-tracking/multi-object-tracking-on-mot16)](https://paperswithcode.com/sota/multi-object-tracking-on-mot16?p=a-simple-baseline-for-multi-object-tracking)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/a-simple-baseline-for-multi-object-tracking/multi-object-tracking-on-mot17)](https://paperswithcode.com/sota/multi-object-tracking-on-mot17?p=a-simple-baseline-for-multi-object-tracking)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/a-simple-baseline-for-multi-object-tracking/multi-object-tracking-on-mot20-1)](https://paperswithcode.com/sota/multi-object-tracking-on-mot20-1?p=a-simple-baseline-for-multi-object-tracking)\n\nA simple baseline for one-shot multi-object tracking:\n![](assets/pipeline.png)\n\u003e [**FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking**](http://arxiv.org/abs/2004.01888),            \n\u003e Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, Wenyu Liu,        \n\u003e *IJCV2021 ([arXiv 2004.01888](http://arxiv.org/abs/2004.01888))*\n## Abstract\nThere has been remarkable progress on object detection and re-identification in recent years which are the core components for multi-object tracking. However, little attention has been focused on accomplishing the two tasks in a single network to improve the inference speed. The initial attempts along this path ended up with degraded results mainly because the re-identification branch is not appropriately learned. In this work, we study the essential reasons behind the failure, and accordingly present a simple baseline to addresses the problems. It remarkably outperforms the state-of-the-arts on the MOT challenge datasets at 30 FPS. We hope this baseline could inspire and help evaluate new ideas in this field.\n\n## News\n* (2021.08.03) Our paper is accepted by IJCV!\n* (2021.06.01) A [nice re-implementation](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.1/configs/mot) by Baidu [PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection)!\n* (2021.05.24) A light version of FairMOT using yolov5s backbone is released! \n* (2020.09.10) A new version of FairMOT is released! (73.7 MOTA on MOT17)\n\n## Main updates\n* We pretrain FairMOT on the CrowdHuman dataset using a weakly-supervised learning approach.\n* To detect bounding boxes outside the image, we use left, top, right and bottom (4 channel) to replace the WH head (2 channel). \n\n## Tracking performance\n### Results on MOT challenge test set\n| Dataset    |  MOTA | IDF1 | IDS | MT | ML | FPS |\n|--------------|-----------|--------|-------|----------|----------|--------|\n|2DMOT15  | 60.6 | 64.7 |  591 | 47.6% | 11.0% | 30.5 |\n|MOT16       | 74.9 | 72.8 | 1074 | 44.7% | 15.9% | 25.9 |\n|MOT17       | 73.7 | 72.3 | 3303 | 43.2% | 17.3% | 25.9 |\n|MOT20       | 61.8 | 67.3 | 5243 | 68.8% | 7.6% | 13.2 |\n\n All of the results are obtained on the [MOT challenge](https://motchallenge.net) evaluation server under the “private detector” protocol. We rank first among all the trackers on 2DMOT15, MOT16, MOT17 and  MOT20. The tracking speed of the entire system can reach up to **30 FPS**.\n\n### Video demos on MOT challenge test set\n\u003cimg src=\"assets/MOT15.gif\" width=\"400\"/\u003e   \u003cimg src=\"assets/MOT16.gif\" width=\"400\"/\u003e\n\u003cimg src=\"assets/MOT17.gif\" width=\"400\"/\u003e   \u003cimg src=\"assets/MOT20.gif\" width=\"400\"/\u003e\n\n\n## Installation\n* Clone this repo, and we'll call the directory that you cloned as ${FAIRMOT_ROOT}\n* Install dependencies. We use python 3.8 and pytorch \u003e= 1.7.0\n```\nconda create -n FairMOT\nconda activate FairMOT\nconda install pytorch==1.7.0 torchvision==0.8.0 cudatoolkit=10.2 -c pytorch\ncd ${FAIRMOT_ROOT}\npip install cython\npip install -r requirements.txt\n```\n* We use [DCNv2_pytorch_1.7](https://github.com/ifzhang/DCNv2/tree/pytorch_1.7) in our backbone network (pytorch_1.7 branch). Previous versions can be found in [DCNv2](https://github.com/CharlesShang/DCNv2).\n```\ngit clone -b pytorch_1.7 https://github.com/ifzhang/DCNv2.git\ncd DCNv2\n./make.sh\n```\n* In order to run the code for demos, you also need to install [ffmpeg](https://www.ffmpeg.org/).\n\n## Data preparation\n\n* **CrowdHuman**\nThe CrowdHuman dataset can be downloaded from their [official webpage](https://www.crowdhuman.org). After downloading, you should prepare the data in the following structure:\n```\ncrowdhuman\n   |——————images\n   |        └——————train\n   |        └——————val\n   └——————labels_with_ids\n   |         └——————train(empty)\n   |         └——————val(empty)\n   └------annotation_train.odgt\n   └------annotation_val.odgt\n```\nIf you want to pretrain on CrowdHuman (we train Re-ID on CrowdHuman), you can change the paths in src/gen_labels_crowd_id.py and run:\n```\ncd src\npython gen_labels_crowd_id.py\n```\nIf you want to add CrowdHuman to the MIX dataset (we do not train Re-ID on CrowdHuman), you can change the paths in src/gen_labels_crowd_det.py and run:\n```\ncd src\npython gen_labels_crowd_det.py\n```\n* **MIX**\nWe use the same training data as [JDE](https://github.com/Zhongdao/Towards-Realtime-MOT) in this part and we call it \"MIX\". Please refer to their [DATA ZOO](https://github.com/Zhongdao/Towards-Realtime-MOT/blob/master/DATASET_ZOO.md) to download and prepare all the training data including Caltech Pedestrian, CityPersons, CUHK-SYSU, PRW, ETHZ, MOT17 and MOT16. \n* **2DMOT15 and MOT20** \n[2DMOT15](https://motchallenge.net/data/2D_MOT_2015/) and [MOT20](https://motchallenge.net/data/MOT20/) can be downloaded from the official webpage of MOT challenge. After downloading, you should prepare the data in the following structure:\n```\nMOT15\n   |——————images\n   |        └——————train\n   |        └——————test\n   └——————labels_with_ids\n            └——————train(empty)\nMOT20\n   |——————images\n   |        └——————train\n   |        └——————test\n   └——————labels_with_ids\n            └——————train(empty)\n```\nThen, you can change the seq_root and label_root in src/gen_labels_15.py and src/gen_labels_20.py and run:\n```\ncd src\npython gen_labels_15.py\npython gen_labels_20.py\n```\nto generate the labels of 2DMOT15 and MOT20. The seqinfo.ini files of 2DMOT15 can be downloaded here [[Google]](https://drive.google.com/open?id=1kJYySZy7wyETH4fKMzgJrYUrTfxKlN1w), [[Baidu],code:8o0w](https://pan.baidu.com/s/1zb5tBW7-YTzWOXpd9IzS0g).\n\n## Pretrained models and baseline model\n* **Pretrained models**\n\nDLA-34 COCO pretrained model: [DLA-34 official](https://drive.google.com/file/d/1pl_-ael8wERdUREEnaIfqOV_VF2bEVRT/view).\nHRNetV2 ImageNet pretrained model: [HRNetV2-W18 official](https://1drv.ms/u/s!Aus8VCZ_C_33cMkPimlmClRvmpw), [HRNetV2-W32 official](https://1drv.ms/u/s!Aus8VCZ_C_33dYBMemi9xOUFR0w).\nAfter downloading, you should put the pretrained models in the following structure:\n```\n${FAIRMOT_ROOT}\n   └——————models\n           └——————ctdet_coco_dla_2x.pth\n           └——————hrnetv2_w32_imagenet_pretrained.pth\n           └——————hrnetv2_w18_imagenet_pretrained.pth\n```\n* **Baseline model**\n\nOur baseline FairMOT model (DLA-34 backbone) is pretrained on the CrowdHuman for 60 epochs with the self-supervised learning approach and then trained on the MIX dataset for 30 epochs. The models can be downloaded here: \ncrowdhuman_dla34.pth [[Google]](https://drive.google.com/file/d/1SFOhg_vos_xSYHLMTDGFVZBYjo8cr2fG/view?usp=sharing) [[Baidu, code:ggzx ]](https://pan.baidu.com/s/1JZMCVDyQnQCa5veO73YaMw) [[Onedrive]](https://microsoftapc-my.sharepoint.com/:u:/g/personal/v-yifzha_microsoft_com/EUsj0hkTNuhKkj9bo9kE7ZsBpmHvqDz6DylPQPhm94Y08w?e=3OF4XN).\nfairmot_dla34.pth [[Google]](https://drive.google.com/file/d/1iqRQjsG9BawIl8SlFomMg5iwkb6nqSpi/view?usp=sharing) [[Baidu, code:uouv]](https://pan.baidu.com/s/1H1Zp8wrTKDk20_DSPAeEkg) [[Onedrive]](https://microsoftapc-my.sharepoint.com/:u:/g/personal/v-yifzha_microsoft_com/EWHN_RQA08BDoEce_qFW-ogBNUsb0jnxG3pNS3DJ7I8NmQ?e=p0Pul1). (This is the model we get 73.7 MOTA on the MOT17 test set. )\nAfter downloading, you should put the baseline model in the following structure:\n```\n${FAIRMOT_ROOT}\n   └——————models\n           └——————fairmot_dla34.pth\n           └——————...\n```\n\n## Training\n* Download the training data\n* Change the dataset root directory 'root' in src/lib/cfg/data.json and 'data_dir' in src/lib/opts.py\n* Pretrain on CrowdHuman and train on MIX:\n```\nsh experiments/crowdhuman_dla34.sh\nsh experiments/mix_ft_ch_dla34.sh\n```\n* Only train on MIX:\n```\nsh experiments/mix_dla34.sh\n```\n* Only train on MOT17:\n```\nsh experiments/mot17_dla34.sh\n```\n* Finetune on 2DMOT15 using the baseline model:\n```\nsh experiments/mot15_ft_mix_dla34.sh\n```\n* Train on MOT20:\nThe data annotation of MOT20 is a little different from MOT17, the coordinates of the bounding boxes are all inside the image, so we need to uncomment line 313 to 316 in the dataset file src/lib/datasets/dataset/jde.py:\n```\n#np.clip(xy[:, 0], 0, width, out=xy[:, 0])\n#np.clip(xy[:, 2], 0, width, out=xy[:, 2])\n#np.clip(xy[:, 1], 0, height, out=xy[:, 1])\n#np.clip(xy[:, 3], 0, height, out=xy[:, 3])\n```\nThen, we can train on the mix dataset and finetune on MOT20:\n```\nsh experiments/crowdhuman_dla34.sh\nsh experiments/mix_ft_ch_dla34.sh\nsh experiments/mot20_ft_mix_dla34.sh\n```\nThe MOT20 model 'mot20_fairmot.pth' can be downloaded here: [[Google]](https://drive.google.com/file/d/1HVzDTrYSSZiVqExqG9rou3zZXX1-GGQn/view?usp=sharing) [[Baidu, code:jmce]](https://pan.baidu.com/s/1bpMtu972ZszsBx4TzIT_CA).\n* For ablation study, we use MIX and half of MOT17 as training data, you can use different backbones such as ResNet, ResNet-FPN, HRNet and DLA::\n```\nsh experiments/mix_mot17_half_dla34.sh\nsh experiments/mix_mot17_half_hrnet18.sh\nsh experiments/mix_mot17_half_res34.sh\nsh experiments/mix_mot17_half_res34fpn.sh\nsh experiments/mix_mot17_half_res50.sh\n```\nThe ablation study model 'mix_mot17_half_dla34.pth' can be downloaded here: [[Google]](https://drive.google.com/file/d/1dJDGSa6-FMq33XY-cOd_nYxuilv30YDM/view?usp=sharing) [[Onedrive]](https://microsoftapc-my.sharepoint.com/:u:/g/personal/v-yifzha_microsoft_com/ESh1SlUvZudKgUX4A8E3yksBhfRHIf2AsKaaPJ-v_5lVAw?e=NB6UHR) [[Baidu, code:iifa]](https://pan.baidu.com/s/1RQD8ik1labWuwd8jJ-0ukQ).\n* Performance on the test set of MOT17 when using different training data:\n\n| Training Data    |  MOTA | IDF1 | IDS     |\n|--------------|-----------|--------|-------|\n|MOT17  | 69.8 | 69.9 | 3996                |\n|MIX       | 72.9 | 73.2 | 3345             |\n|CrowdHuman + MIX     | 73.7 | 72.3 | 3303  |\n* We use CrowdHuman, MIX and MOT17 to train the light version of FairMOT using yolov5s as backbone:\n```\nsh experiments/all_yolov5s.sh\n```\nThe pretrained model of yolov5s on the COCO dataset can be downloaded here:  [[Google]](https://drive.google.com/file/d/1Ur3_pa9r3KRY-5qM2cdFhFJ5exghRJvh/view?usp=sharing) [[Baidu, code:wh9h]](https://pan.baidu.com/s/1JHjN_l1nkMnRHRF5TcHYXg).\n\nThe model of the light version 'fairmot_yolov5s' can be downloaded here:  [[Google]](https://drive.google.com/file/d/1MEvsRPyoAqYSCdKaS5Ofrl7ZfKbBZ1Jb/view?usp=sharing) [[Baidu, code:2y3a]](https://pan.baidu.com/s/1dyBEeiGpRfZhqae0c264rg).\n\n## Tracking\n* The default settings run tracking on the validation dataset from 2DMOT15. Using the baseline model, you can run:\n```\ncd src\npython track.py mot --load_model ../models/fairmot_dla34.pth --conf_thres 0.6\n```\nto see the tracking results (76.5 MOTA and 79.3 IDF1 using the baseline model). You can also set save_images=True in src/track.py to save the visualization results of each frame. \n* For ablation study, we evaluate on the other half of the training set of MOT17, you can run:\n```\ncd src\npython track_half.py mot --load_model ../exp/mot/mix_mot17_half_dla34.pth --conf_thres 0.4 --val_mot17 True\n```\nIf you use our pretrained model 'mix_mot17_half_dla34.pth', you can get 69.1 MOTA and 72.8 IDF1. \n* To get the txt results of the test set of MOT16 or MOT17, you can run:\n```\ncd src\npython track.py mot --test_mot17 True --load_model ../models/fairmot_dla34.pth --conf_thres 0.4\npython track.py mot --test_mot16 True --load_model ../models/fairmot_dla34.pth --conf_thres 0.4\n```\n* To run tracking using the light version of FairMOT (68.5 MOTA on the test of MOT17), you can run:\n```\ncd src\npython track.py mot --test_mot17 True --load_model ../models/fairmot_yolov5s.pth --conf_thres 0.4 --arch yolo --reid_dim 64\n```\nand send the txt files to the [MOT challenge](https://motchallenge.net) evaluation server to get the results. (You can get the SOTA results 73+ MOTA on MOT17 test set using the baseline model 'fairmot_dla34.pth'.)\n\n* To get the SOTA results of 2DMOT15 and MOT20, run the tracking code:\n```\ncd src\npython track.py mot --test_mot15 True --load_model your_mot15_model.pth --conf_thres 0.3\npython track.py mot --test_mot20 True --load_model your_mot20_model.pth --conf_thres 0.3\n```\nResults of the test set all need to be evaluated on the MOT challenge server. You can see the tracking results on the training set by setting --val_motxx True and run the tracking code. We set 'conf_thres' 0.4 for MOT16 and MOT17. We set 'conf_thres' 0.3 for 2DMOT15 and MOT20. \n\n## Demo\nYou can input a raw video and get the demo video by running src/demo.py and get the mp4 format of the demo video:\n```\ncd src\npython demo.py mot --load_model ../models/fairmot_dla34.pth --conf_thres 0.4\n```\nYou can change --input-video and --output-root to get the demos of your own videos.\n--conf_thres can be set from 0.3 to 0.7 depending on your own videos.\n\n## Train on custom dataset\nYou can train FairMOT on custom dataset by following several steps bellow:\n1. Generate one txt label file for one image. Each line of the txt label file represents one object. The format of the line is: \"class id x_center/img_width y_center/img_height w/img_width h/img_height\". You can modify src/gen_labels_16.py to generate label files for your custom dataset.\n2. Generate files containing image paths. The example files are in src/data/. Some similar code can be found in src/gen_labels_crowd.py\n3. Create a json file for your custom dataset in src/lib/cfg/. You need to specify the \"root\" and \"train\" keys in the json file. You can find some examples in src/lib/cfg/.\n4. Add --data_cfg '../src/lib/cfg/your_dataset.json' when training. \n\n## Acknowledgement\nA large part of the code is borrowed from [Zhongdao/Towards-Realtime-MOT](https://github.com/Zhongdao/Towards-Realtime-MOT) and [xingyizhou/CenterNet](https://github.com/xingyizhou/CenterNet). Thanks for their wonderful works.\n\n## Citation\n\n```\n@article{zhang2021fairmot,\n  title={Fairmot: On the fairness of detection and re-identification in multiple object tracking},\n  author={Zhang, Yifu and Wang, Chunyu and Wang, Xinggang and Zeng, Wenjun and Liu, Wenyu},\n  journal={International Journal of Computer Vision},\n  volume={129},\n  pages={3069--3087},\n  year={2021},\n  publisher={Springer}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fifzhang%2Ffairmot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fifzhang%2Ffairmot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fifzhang%2Ffairmot/lists"}