{"id":20852429,"url":"https://github.com/dirtyharrylyl/hake","last_synced_at":"2025-04-13T09:40:37.819Z","repository":{"id":79335554,"uuid":"192167748","full_name":"DirtyHarryLYL/HAKE","owner":"DirtyHarryLYL","description":"HAKE: Human Activity Knowledge Engine (CVPR'18/19/20, NeurIPS'20, TPAMI'21)","archived":false,"fork":false,"pushed_at":"2023-08-31T10:54:29.000Z","size":22813,"stargazers_count":224,"open_issues_count":0,"forks_count":14,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-03-27T01:11:10.388Z","etag":null,"topics":["action-recognition","activity-recognition","benchmark","dataset","human-object-interaction","knowledge-base","object-recognition"],"latest_commit_sha":null,"homepage":"http://hake-mvig.cn","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DirtyHarryLYL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-16T08:28:38.000Z","updated_at":"2025-03-23T07:34:10.000Z","dependencies_parsed_at":"2025-01-09T19:41:52.397Z","dependency_job_id":"45834817-15a2-409c-aa9b-d90ab9765a5b","html_url":"https://github.com/DirtyHarryLYL/HAKE","commit_stats":{"total_commits":132,"total_committers":2,"mean_commits":66.0,"dds":"0.11363636363636365","last_synced_commit":"d5c39a6913948e3551f173ec2638f393571fc3c7"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DirtyHarryLYL%2FHAKE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DirtyHarryLYL%2FHAKE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DirtyHarryLYL%2FHAKE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DirtyHarryLYL%2FHAKE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DirtyHarryLYL","download_url":"https://codeload.github.com/DirtyHarryLYL/HAKE/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248693139,"owners_count":21146738,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["action-recognition","activity-recognition","benchmark","dataset","human-object-interaction","knowledge-base","object-recognition"],"created_at":"2024-11-18T03:17:38.783Z","updated_at":"2025-04-13T09:40:37.791Z","avatar_url":"https://github.com/DirtyHarryLYL.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HAKE: Human Activity Knowledge Engine\n\n\u003cp align='center'\u003e\n    \u003cimg src=\"https://github.com/DirtyHarryLYL/HAKE-Action-Torch/blob/Activity2Vec/demo/hake_history.jpg\", height=\"300\"\u003e\n\u003c/p\u003e\n\nFor more details please refer to HAKE website http://hake-mvig.cn.\n\n# HAKE project:\n- **HAKE-Reasoning** (TPAMI): Neural-Symbolic reasoning engine. [HAKE-Reasoning](https://github.com/DirtyHarryLYL/HAKE-Action-Torch/tree/HAKE-Reason)\n- **HAKE-Image** (CVPR'18/20): Human body part state labels in images. [HAKE-HICO](https://github.com/DirtyHarryLYL/HAKE#hake-hico-for-image-level-hoi-recognition), [HAKE-HICO-DET](https://github.com/DirtyHarryLYL/HAKE#hake-hico-det-for-instance-level-hoi-detection), [HAKE-Large](https://github.com/DirtyHarryLYL/HAKE#hake-large-for-instance-level-action-understanding-pre-training), [Extra-40-verbs](https://github.com/DirtyHarryLYL/HAKE#extra-40-verb-categories).\n- **[HAKE-AVA](https://github.com/DirtyHarryLYL/HAKE-AVA)**: Human body part state labels in videos from AVA dataset.\n- **[CLIP-A2V](https://github.com/DirtyHarryLYL/HAKE-Action-Torch/tree/CLIP-Activity2Vec)**: CLIP-based part states \u0026 verb recognizer.\n- **[HAKE-A2V](https://github.com/DirtyHarryLYL/HAKE-Action-Torch/tree/Activity2Vec)** (CVPR'20): Activity2Vec, a general activity feature extractor based on HAKE data, converting a human (box) to a fixed-size vector, PaSta and action scores.\n- **[HAKE-Action-TF](https://github.com/DirtyHarryLYL/HAKE-Action), [HAKE-Action-Torch](https://github.com/DirtyHarryLYL/HAKE-Action-Torch)** (CVPR'19/20/22, NeurIPS'20, TPAMI'21/22, ECCV'22): SOTA action understanding methods and the corresponding HAKE-enhanced versions ([TIN](https://github.com/DirtyHarryLYL/Transferable-Interactiveness-Network), [IDN](https://github.com/DirtyHarryLYL/HAKE-Action-Torch/tree/IDN-(Integrating-Decomposing-Network)), [IF](https://github.com/Foruck/Interactiveness-Field), [mPD](https://github.com/Foruck/OC-Immunity), [PartMap](https://github.com/enlighten0707/Body-Part-Map-for-Interactiveness)).\n- **HAKE-3D** (CVPR'20): 3D human-object representation for action understanding ([DJ-RN](https://github.com/DirtyHarryLYL/DJ-RN)).\n- **HAKE-Object** (CVPR'20, TPAMI'21): object knowledge learner to advance action understanding ([SymNet](https://github.com/DirtyHarryLYL/SymNet)).\n- [**Halpe**](https://github.com/Fang-Haoshu/Halpe-FullBody): a joint project under [AlphaPose](https://github.com/MVIG-SJTU/AlphaPose) and [HAKE](http://hake-mvig.cn), full-body human keypoints (body, face, hand, 136 points) of 50,000 HOI images.\n- [**HOI Learning List**](https://github.com/DirtyHarryLYL/HOI-Learning-List): a list of recent HOI (Human-Object Interaction) papers, code, datasets and leaderboard on widely-used benchmarks. Hope it could help everyone interested in HOI.\n\n#### **News**: (2022.12.19) HAKE 2.0 is accepted by TPAMI!\n\n(2022.11.19) We release the interactive object bounding boxes \u0026 classes in the interactions within AVA dataset (2.1 \u0026 2.2)! [HAKE-AVA](https://github.com/DirtyHarryLYL/HAKE-AVA), [[Paper]](https://arxiv.org/abs/2211.07501). BTW, we also release a CLIP-based human body part states recognizer in [CLIP-Activity2Vec](https://github.com/DirtyHarryLYL/HAKE-Action-Torch/tree/CLIP-Activity2Vec)!\n\n(2022.07.29) Our new work PartMap is released! [Paper](https://github.com/enlighten0707/Body-Part-Map-for-Interactiveness/blob/main), [Code](https://github.com/DirtyHarryLYL/HAKE-Action-Torch)\n\n(2022.04.23) Two new works on HOI learning are releassed! [Interactiveness Field](https://arxiv.org/abs/2204.07718) (CVPR'22) and a new HOI metric [mPD](https://arxiv.org/abs/2202.09492) (AAAI'22).\n\n(2022.02.14) We release the human body part state labels based on AVA: [HAKE-AVA](https://github.com/DirtyHarryLYL/HAKE-AVA) and [HAKE 2.0 paper](https://arxiv.org/abs/2202.06851).\n\n(2021.10.06) Our extended version of [SymNet](https://github.com/DirtyHarryLYL/SymNet) is accepted by TPAMI! Paper and code are coming soon.\n\n(2021.2.7) Upgraded [HAKE-Activity2Vec](https://github.com/DirtyHarryLYL/HAKE-Action-Torch/tree/Activity2Vec) is released! Images/Videos --\u003e human box + ID + skeleton + part states + action + representation. [[Description]](https://drive.google.com/file/d/1iZ57hKjus2lKbv1MAB-TLFrChSoWGD5e/view?usp=sharing)\n\u003cp align='center'\u003e\n    \u003cimg src=\"https://github.com/DirtyHarryLYL/HAKE-Action-Torch/blob/Activity2Vec/demo/a2v-demo.gif\", height=\"400\"\u003e\n\u003c/p\u003e\n\n\u003c!-- ## Full demo: [[YouTube]](https://t.co/hXiAYPXEuL?amp=1), [[bilibili]](https://www.bilibili.com/video/BV1s54y1Y76s) --\u003e\n\n(2021.1.15) Our extended version of [TIN (Transferable Interactiveness Network)](https://arxiv.org/abs/2101.10292) is accepted by TPAMI!\n\n(2020.10.27) The code of [IDN](https://github.com/DirtyHarryLYL/HAKE-Action-Torch/tree/IDN-(Integrating-Decomposing-Network)) ([Paper](https://arxiv.org/abs/2010.16219)) in NeurIPS'20 is released!\n\n(2020.6.16) Our larger version [HAKE-Large](https://github.com/DirtyHarryLYL/HAKE#hake-large-for-instance-level-hoi-detection) (\u003e122K images, activity and part state labels) and [Extra-40-verbs](https://github.com/DirtyHarryLYL/HAKE#extra-40-verb-categories) (40 new actions) are released!\n\n**The image-level and instance-level part state annotations upon HICO and HICO-DET are available!**\n\n- Paper is here: [PaStaNet](https://arxiv.org/abs/2004.00945), [HAKE 2.0 paper](https://arxiv.org/abs/2202.06851).\n\n- Corresponding Code and model ([HAKE-Action](https://github.com/DirtyHarryLYL/HAKE-Action)): [Image-level](https://github.com/DirtyHarryLYL/HAKE-Action/tree/Image-level-HAKE-Action) and [Instance-level](https://github.com/DirtyHarryLYL/HAKE-Action/tree/Instance-level-HAKE-Action).\n\n#### Note that: \n- Image-level means that what Human-Object Interactions are included in an image, and the corrsponding task is the HOI recognition (image-level multi-label classification from [HICO](http://www-personal.umich.edu/~ywchao/hico/)). \n- Instance-level means that what HOIs are performed by a person, and the task is HOI detection (instance-level multi-label detection from [HICO-DET](http://www-personal.umich.edu/~ywchao/publications/chao_wacv2018.pdf)).\n\nIf you find HAKE useful, please cite our papers:\n\n    @article{li2023hake,\n     title={HAKE: A Knowledge Engine Foundation for Human Activity Understanding},\n    author={Li, Yong-Lu and Liu, Xinpeng and Wu, Xiaoqian and Li, Yizhuo and Qiu, Zuoyu and Xu, Liang and Xu, Yue and Fang, Hao-Shu and Lu, Cewu},\n    journal={TPAMI},\n    year={2023}\n    }\n    \n    @inproceedings{li2020pastanet,\n        title={PaStaNet: Toward Human Activity Knowledge Engine},\n        author={Li, Yong-Lu and Xu, Liang and Liu, Xinpeng and Huang, Xijie and Xu, Yue and Wang, Shiyi and Fang, Hao-Shu and Ma, Ze and Chen, Mingyang and Lu, Cewu},\n        booktitle={CVPR},\n        year={2020}\n    }\n    \n    @inproceedings{lu2018beyond,\n        title={Beyond holistic object recognition: Enriching image understanding with part states},\n        author={Lu, Cewu and Su, Hao and Li, Yonglu and Lu, Yongyi and Yi, Li and Tang, Chi-Keung and Guibas, Leonidas J},\n        booktitle={CVPR},\n        year={2018}\n    }\n\n## HAKE-HICO (For Image-level HOI Recognition)\n\nWe have released image-level part state annotations on [HICO](http://www-personal.umich.edu/~ywchao/hico/). \nHOI recognition task can be modeled as a multi-label classification problem with 600 HOI categories. Given a still image, the model should tell the involved HOI categories in this image.\n\nAll the 38,116 images in train set of HICO dataset are annotated with finer human body part states. \nFor better understanding of HOI recognition task, you could refer to these works: [HICO](http://www-personal.umich.edu/~ywchao/publications/chao_iccv2015.pdf), [Pair-wise](http://openaccess.thecvf.com/content_ECCV_2018/papers/Haoshu_Fang_Pairwise_Body-Part_Attention_ECCV_2018_paper.pdf), [HAKE](https://arxiv.org/abs/2004.00945.pdf).\n\n### Dataset\nThe labels are packaged in **Annotations/hico-image-level.tar.gz**, you can use:\n\n    cd Annotations\n    tar zxvf hico-image-level.tar.gz\n\nto unzip them and get hico-training-set-image-level.json for train set of HICO respectively. More details about the format are shown in [Dataset format](Annotations/README.md#image-level-partstate-for-hico).\n\nThe HICO dataset can be found here: [HICO](http://www-personal.umich.edu/~ywchao/hico/).\n\n### Code and Models\nThe corresponding code and models can be found [here](https://github.com/DirtyHarryLYL/HAKE-Action/tree/Image-level-HAKE-Action).\n\n### Results\nWe provide our current state-of-the-art result file on HICO.\n\n\n| Method | Few@1 | Few@5 | Few@10 | mAP | result |\n| --- | ------ | ------ | --- | --- | ---|\n| Pairwise-Part+HAKE-ALL | 25.40 | 32.48 | 33.71 | 47.09 | [hico\\_result\\_pairwise\\_hake\\_all.csv](https://drive.google.com/file/d/1nWi44-UaMB0cnkdGoIN8dGi3cwsRcnVb/view?usp=sharing) |\n\n### Evaluation\nAfter downloading above result file, you could use the following commands to evaluate: \n\n1. Download evaluation [code](https://drive.google.com/drive/folders/1mvXAtCe0Yc7JUQXCu3D_wpWt7r048lGc?usp=sharing) here (It is a modification of [this benchmark](https://github.com/ywchao/hico_benchmark))\n2. Copy the result file to #/data/test-result.csv, where # means the folder of the evaluation code\n3. run `matlab -nodesktop -nodisplay`\n4. run `eval_default_run`\n\n\n## HAKE-HICO-DET (For Instance-level HOI Detection)\n\nInstance-level part state annotations on [HICO-DET](http://www-personal.umich.edu/~ywchao/hico/) are also available.\n\n### Dataset\nThe labels are packaged in **Annotations/hico-det-instance-level.tar.gz**, you could use:\n\n    cd Annotations\n    tar zxvf hico-det-instance-level.tar.gz\n\nto unzip them and get hico-det-training-set-instance-level.json for train set of HICO-DET respectively. \nMore details about the format are shown in [Dataset format](Annotations/README.md).\n\nThe HICO-DET dataset can be found here: [HICO-DET](http://www-personal.umich.edu/~ywchao/hico/).\n\n### Code and Models\nThe corresponding code and models can be found [here](https://github.com/DirtyHarryLYL/HAKE-Action/tree/Instance-level-HAKE-Action).\n\n## HAKE-Large (For Instance-level Action Understanding Pre-training)\n\nInstance-level part state annotations on HAKE-Large are also available now!\n\n\n### Dataset\nThe labels are packaged in **Annotations/hake\\_large\\_annotation.tar.gz**, you could use:\n\n    cd Annotations\n    tar zxvf hake_large_annotation.tar.gz\n\nto unzip them and get hake\\_large\\_annotation.json for train set of HAKE-Large respectively. \nMore details about the format are shown in [Dataset format](Annotations/README.md).\n\n### Images\nYou could download the corresponding images following [this](https://github.com/DirtyHarryLYL/HAKE/tree/master/Images#download-images-for-hake).\n\n### Extra 40 verb categories\n\nWe also provided the image set and part-state labels of the extra 40 verb categories (includes both HOI and human-only actions). You can download them from [Google Drive](https://drive.google.com/open?id=14K_4FfjviJNDVLJdGM96W2ZLN55dDb2-). The verb\\_list, part-state\\_list is attached in the zip file. For these 40 verb categories, objects are also from coco80 categories but object bounding boxes and categories are optional (e.g. `dance` has no interactive objects).\n\n\n## HAKE-AVA (For Instance-level Action Detection from Videos)\nFine-grained Spatio-Temporal Activity Understanding based on AVA videos. HAKE-AVA contains the human body part states (PaSta) annotations upon AVA and covers all the labeled human instances. PaSta (Part State) describes the action states of 10 human body parts, i.e., head, arms, hands, hip, legs, and feet.\n\nFor details, please refer to this [repo](https://github.com/DirtyHarryLYL/HAKE-AVA).\n\n\n## TODOS\n- [x] Image-level label results on HICO\n- [x] Image-level code and models\n- [x] Instance-level label results on HICO-DET\n- [x] Instance-level code and models\n- [x] HAKE-Large data\n- [x] HAKE-A2V, pipeline, model\n- [x] HAKE-Action in PYTorch\n- [x] HAKE-AVA data\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdirtyharrylyl%2Fhake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdirtyharrylyl%2Fhake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdirtyharrylyl%2Fhake/lists"}