{"id":14971637,"url":"https://github.com/yeliudev/consnet","last_synced_at":"2025-09-30T12:30:26.947Z","repository":{"id":62564518,"uuid":"287545248","full_name":"yeliudev/ConsNet","owner":"yeliudev","description":"🚴‍♂️ ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection (MM 2020)","archived":false,"fork":false,"pushed_at":"2023-07-19T22:21:46.000Z","size":766,"stargazers_count":32,"open_issues_count":1,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-01-16T22:22:18.379Z","etag":null,"topics":["human-object-interaction","pytorch"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2008.06254","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yeliudev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-08-14T13:56:42.000Z","updated_at":"2024-12-12T07:49:54.000Z","dependencies_parsed_at":"2022-11-03T16:46:01.005Z","dependency_job_id":"5cdd0bb9-07b3-48dd-b8cc-301fefee797a","html_url":"https://github.com/yeliudev/ConsNet","commit_stats":{"total_commits":24,"total_committers":3,"mean_commits":8.0,"dds":"0.16666666666666663","last_synced_commit":"1dcfdf5b028e577c42bd44d48abdc82e63856020"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yeliudev%2FConsNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yeliudev%2FConsNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yeliudev%2FConsNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yeliudev%2FConsNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yeliudev","download_url":"https://codeload.github.com/yeliudev/ConsNet/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234732935,"owners_count":18878416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["human-object-interaction","pytorch"],"created_at":"2024-09-24T13:45:35.437Z","updated_at":"2025-09-30T12:30:21.506Z","avatar_url":"https://github.com/yeliudev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ConsNet\n\n[![DOI](https://badgen.net/badge/DOI/10.1145%2F3394171.3413600/blue?cache=300)](https://doi.org/10.1145/3394171.3413600)\n[![arXiv](https://badgen.net/badge/arXiv/2008.06254/red?cache=300)](https://arxiv.org/abs/2008.06254)\n[![PyPI](https://badgen.net/pypi/v/consnet?label=PyPI\u0026cache=300)](https://pypi.org/project/consnet)\n[![License](https://badgen.net/github/license/yeliudev/ConsNet?label=License\u0026color=cyan\u0026cache=300)](https://github.com/yeliudev/ConsNet/blob/main/LICENSE)\n\nThis repository maintains the official implementation of the paper **ConsNet: Learning Consistency Graph for Zero‐Shot Human‐Object Interaction Detection** by [Ye Liu](https://yeliu.dev/), [Junsong Yuan](https://cse.buffalo.edu/~jsyuan/) and [Chang Wen Chen](https://web.comp.polyu.edu.hk/chencw/), which has been accepted by [ACM Multimedia 2020](https://2020.acmmm.org/).\n\n\u003cp align=\"center\"\u003e\u003cimg width=\"850\" src=\"https://raw.githubusercontent.com/yeliudev/ConsNet/main/.github/model.svg\"\u003e\u003c/p\u003e\n\n## Installation\n\nThe ConsNet package could be installed directly from PyPI or manually from source for different uses. Please refer to the following environmental settings that we use.\n\n- CUDA 10.2 Update 2\n- CUDNN 8.0.5.39\n- Python 3.9.2\n- PyTorch 1.8.1\n- [MMDetection](https://github.com/open-mmlab/mmdetection) 2.11.0\n- [AllenNLP](https://github.com/allenai/allennlp) 2.2.0\n- [NNCore](https://github.com/yeliudev/nncore) 0.2.4\n\n### Install from PyPI\n\nYou may install ConsNet from PyPI and import it in your own project as a Python package. This library implements several useful functionalities including [Pair IoU](https://consnet.readthedocs.io/en/latest/consnet.api.bbox.html#consnet.api.bbox.pair_iou), [Pair NMS](https://consnet.readthedocs.io/en/latest/consnet.api.bbox.html#consnet.api.bbox.pair_nms) and [unified APIs for HICO-DET dataset](https://consnet.readthedocs.io/en/latest/consnet.api.data.html).\n\nSimply run the following command to install the latest version of ConsNet.\n\n```\npip install consnet\n```\n\nFor more details about `consnet.api`, please refer to our [documentation](https://consnet.readthedocs.io/).\n\n### Install from source\n\nBy installing ConsNet from source, you may access the full capabilities of this project, including pooling object features, constructing the consistency graph and benchmarking the ConsNet model.\n\n1. Clone the repository from GitHub.\n\n```\ngit clone https://github.com/yeliudev/ConsNet.git\ncd ConsNet\n```\n\n2. Install full dependencies and the package.\n\n```\npip install -e .[full]\n```\n\n## Getting Started\n\nWe pre-extract the visual features of all the humans and objects in the dataset and save them for training as well as testing. These features are also used to construct the consistency graph. Please refer to our paper for more details about feature extraction and data sampling.\n\n### Build dataset and construct the consistency graph\n\n1. Download the checkpoints of object detector and ELMo.\n\n```shell\nROOT='https://s3-us-west-2.amazonaws.com/allennlp/models/elmo'\nELMO='2x4096_512_2048cnn_2xhighway_5.5B'\n\n# Download object detector checkpoints\nwget https://huggingface.co/yeliudev/ConsNet/resolve/main/faster_rcnn_r50_fpn_3x_coco-26df6f6b.pth\nwget https://huggingface.co/yeliudev/ConsNet/resolve/main/faster_rcnn_r50_fpn_20e_hico_det-77b91312.pth\n\n# Download ELMo options and weights\nwget ${ROOT}/${ELMO}/elmo_${ELMO}_options.json\nwget ${ROOT}/${ELMO}/elmo_${ELMO}_weights.hdf5\n```\n\n2. Download [HICO-DET](http://www-personal.umich.edu/~ywchao/hico/) dataset and prepare the files in the following structure.\n\n```\nConsNet\n├── configs\n├── consnet\n├── tools\n├── checkpoints\n│   ├── faster_rcnn_r50_fpn_3x_coco-26df6f6b.pth\n│   ├── faster_rcnn_r50_fpn_20e_hico_det-77b91312.pth\n│   ├── elmo_2x4096_512_2048cnn_2xhighway_5.5B_options.json\n│   └── elmo_2x4096_512_2048cnn_2xhighway_5.5B_weights.hdf5\n├── data\n│   └── hico_20160224_det\n│       ├── anno_bbox.mat\n│       └── images\n│           ├── train2015\n│           └── test2015\n├── README.md\n├── setup.py\n└── ···\n```\n\n3. Convert the annotations to COCO format. The results will be saved in `data/hico_det/annotations`.\n\n```\npython tools/convert_anno.py\n```\n\n4. Build dataset and construct the consistency graph. The results will be saved in `data/hico_det`.\n\n```\npython tools/build_dataset.py --checkpoint \u003cpath-to-checkpoint\u003e\n```\n\n### Train a model\n\nRun the following command to train a model using specified configs.\n\n```\npython tools/launch.py --config \u003cpath-to-config\u003e\n```\n\n### Test a model and evaluate results\n\nRun the following command to test a model and evaluate results.\n\n```\npython tools/launch.py --config \u003cpath-to-config\u003e --checkpoint \u003cpath-to-checkpoint\u003e --eval\n```\n\n## Model Zoo\n\nWe provide multiple HICO-DET pre-trained models here. All the models are trained using a single NVIDIA Tesla V100-SXM2 GPU and are evaluated under the `default` metric of HICO-DET dataset.\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth rowspan=\"2\"\u003eDetector\u003c/th\u003e\n    \u003cth rowspan=\"2\"\u003eModel\u003c/th\u003e\n    \u003cth rowspan=\"2\"\u003eType\u003c/th\u003e\n    \u003cth colspan=\"5\"\u003ePerformance (mAP)\u003c/th\u003e\n    \u003cth rowspan=\"2\"\u003eDownload\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003cth\u003eFull\u003c/th\u003e\n    \u003cth\u003eRare\u003c/th\u003e\n    \u003cth\u003eNon-Rare\u003c/th\u003e\n    \u003cth\u003eSeen\u003c/th\u003e\n    \u003cth\u003eUnseen\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\" rowspan=\"5\"\u003e\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/faster_rcnn_r50_fpn_3x_coco-26df6f6b.pth\"\u003eCOCO\u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/yeliudev/ConsNet/blob/main/configs/consnet_uc_5e_hico_det.py\"\u003eConsNet\u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003eUC\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e19.78\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e14.43\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e21.37\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e20.69\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e16.13\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/consnet_uc_5e_hico_det-3a355824.pth\"\u003emodel\u003c/a\u003e |\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/consnet_uc_5e_hico_det.json\"\u003emetrics\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/yeliudev/ConsNet/blob/main/configs/consnet_uo_5e_hico_det.py\"\u003eConsNet\u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003eUO\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e20.71\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e16.81\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e21.87\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e20.99\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e19.27\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/consnet_uo_5e_hico_det-21652552.pth\"\u003emodel\u003c/a\u003e |\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/consnet_uo_5e_hico_det.json\"\u003emetrics\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/yeliudev/ConsNet/blob/main/configs/consnet_ua_5e_hico_det.py\"\u003eConsNet\u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003eUA\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e19.04\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e14.54\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e20.38\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e20.02\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e14.12\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/consnet_ua_5e_hico_det-492bab60.pth\"\u003emodel\u003c/a\u003e |\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/consnet_ua_5e_hico_det.json\"\u003emetrics\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/yeliudev/ConsNet/blob/main/configs/consnet_gt_5e_hico_det.py\"\u003eConsNet\u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003eGT\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e53.04\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e38.79\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e57.3\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e—\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e—\u003c/td\u003e\n    \u003ctd align=\"center\" rowspan=\"2\"\u003e\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/consnet_5e_hico_det-684a879d.pth\"\u003emodel\u003c/a\u003e |\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/consnet_5e_hico_det.json\"\u003emetrics\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/yeliudev/ConsNet/blob/main/configs/consnet_5e_hico_det.py\"\u003eConsNet\u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e—\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e22.15\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e17.55\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e23.52\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e—\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e—\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/faster_rcnn_r50_fpn_20e_hico_det-77b91312.pth\"\u003eHICO-DET\u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/yeliudev/ConsNet/blob/main/configs/consnet_5e_hico_det.py\"\u003eConsNet-F\u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e—\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e25.94\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e19.35\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e27.91\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e—\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e—\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/consnet_f_5e_hico_det-44c8412c.pth\"\u003emodel\u003c/a\u003e |\n      \u003ca href=\"https://huggingface.co/yeliudev/ConsNet/resolve/main/consnet_f_5e_hico_det.json\"\u003emetrics\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\nNote that: Type `UC`, `UO`, `UA` and `GT` represent unseen action-object combination, unseen object, unseen action and ground truth scenarios respectively.\n\n## Customization\n\nThanks to the modulized implementation based on [NNCore](https://github.com/yeliudev/nncore), this project is highly customizable with a number of replaceable modules. You may play with the hyperparameters in `configs` or construct your own HOI detection pipeline by replacing the dataset, detector, embedder, etc. Please check the [documentation](https://nncore.readthedocs.io/) of NNCore for more details about customizing the engine and modules.\n\n## Citation\n\nIf you find this project useful for your research, please kindly cite our paper.\n\n```bibtex\n@inproceedings{liu2020consnet,\n  title={ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection},\n  author={Liu, Ye and Yuan, Junsong and Chen, Chang Wen},\n  booktitle={Proceedings of The ACM International Conference on Multimedia (MM)},\n  pages={4235--4243},\n  year={2020}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyeliudev%2Fconsnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyeliudev%2Fconsnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyeliudev%2Fconsnet/lists"}