{"id":25902939,"url":"https://github.com/prasunroy/mcma","last_synced_at":"2025-09-09T18:33:15.886Z","repository":{"id":278630845,"uuid":"935701372","full_name":"prasunroy/mcma","owner":"prasunroy","description":":fire: Exploring Mutual Cross-Modal Attention for Context-Aware Human Affordance Generation (official code).","archived":false,"fork":false,"pushed_at":"2025-08-07T21:10:17.000Z","size":27,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-07T23:21:45.534Z","etag":null,"topics":["affordance-generation","cross-attention","human-pose","vae"],"latest_commit_sha":null,"homepage":"http://arxiv.org/abs/2502.13637","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prasunroy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-19T21:54:15.000Z","updated_at":"2025-08-07T21:10:21.000Z","dependencies_parsed_at":"2025-02-20T20:33:10.540Z","dependency_job_id":"78c2ebe1-6a88-4612-94e3-e7f147ac2250","html_url":"https://github.com/prasunroy/mcma","commit_stats":null,"previous_names":["prasunroy/mcma"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/prasunroy/mcma","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prasunroy%2Fmcma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prasunroy%2Fmcma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prasunroy%2Fmcma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prasunroy%2Fmcma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prasunroy","download_url":"https://codeload.github.com/prasunroy/mcma/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prasunroy%2Fmcma/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274340937,"owners_count":25267297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-09T02:00:10.223Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["affordance-generation","cross-attention","human-pose","vae"],"created_at":"2025-03-03T03:17:50.386Z","updated_at":"2025-09-09T18:33:15.870Z","avatar_url":"https://github.com/prasunroy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Official code for Exploring Mutual Cross-Modal Attention for Context-Aware Human Affordance Generation.\n\n*Accepted in the IEEE Transactions on Artificial Intelligence (TAI) 2025.*\n\n[![badge_torch](https://img.shields.io/badge/made_with-PyTorch_2.0-EE4C2C?style=flat-square\u0026logo=PyTorch)](https://pytorch.org/)\n[![badge_arxiv](https://img.shields.io/badge/arXiv-2502.13637-brightgreen?style=flat-square)](https://arxiv.org/abs/2502.13637)\n\n![teaser](https://github.com/user-attachments/assets/c0099350-db25-412b-ad7f-1fbe6d77a856)\n\n### :zap: Getting Started\n\u003e Note: This release is tested on Python 3.9.16.\n```bash\ngit clone https://github.com/prasunroy/mcma.git\ncd mcma\npython -m venv .venv\nsource .venv/bin/activate\npython -m pip install --upgrade pip\npip install -r requirements.txt\n```\n\n### :wrench: Preparing the [Binge Watching](https://www.cs.cmu.edu/~xiaolonw/affordance.html) Dataset\n* Download the [dataset](https://www.dropbox.com/s/2ny1h8nvzvfo7vr/affordance_data.tar.gz) and extract into `datasets/binge_watching` directory.\n* Download the [pose cluster centers](https://www.cs.cmu.edu/~xiaolonw/affordance/centers_30.txt) into `datasets/binge_watching` directory.\n```bash\nmkdir datasets\ncd datasets\nwget https://www.dropbox.com/s/2ny1h8nvzvfo7vr/affordance_data.tar.gz\nwget https://www.cs.cmu.edu/~xiaolonw/affordance/centers_30.txt\ntar -xzf affordance_data.tar.gz\nmv centers_30.txt affordance_data/\nrm -r affordance_data/pose_gt_testset/\nrm -r affordance_data/*.m\nrm affordance_data/pose_all_list.txt\nrm affordance_data/README.txt\nrm affordance_data.tar.gz\nmv affordance_data/ binge_watching/\ncd ..\n```\n* Generate the semantic maps.\n```bash\npython utils/compute_semantic_maps.py\n```\n\u003e Alternatively, download the [precomputed semantic maps](https://drive.google.com/file/d/1CtbtZCamIxQ2Cl8QYJ0AVKK-eIM-BZQ1/view) and extract into `datasets/binge_watching/semantic_maps_oneformer` directory.\n* Reduce the initial 150 semantic labels to 8 groups.\n```bash\npython utils/remap_semantic_labels.py\n```\n\n### :rocket: Training / Testing Models and Running Inference Pipeline\n* Configure and run `train.py` / `test.py` from individual subdirectories under the `models` directory.\n* Copy the best checkpoint of each model into `pipeline/checkpoints`. Then configure and run `test.py` from the `pipeline` directory.\n\n### :heart: Citation\n```\n@article{roy2025exploring,\n  title   = {Exploring Mutual Cross-Modal Attention for Context-Aware Human Affordance Generation},\n  author  = {Roy, Prasun and Bhattacharya, Saumik and Ghosh, Subhankar and Pal, Umapada and Blumenstein, Michael},\n  journal = {IEEE Transactions on Artificial Intelligence},\n  year    = {2025},\n  issn    = {2691-4581},\n  doi     = {https://doi.org/10.1109/TAI.2025.3581897}\n}\n```\n\n### :page_facing_up: License\n```\nCopyright 2025 by the authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```\n\n\u003cbr\u003e\n\n##### Made with :heart: and :pizza: on Earth.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprasunroy%2Fmcma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprasunroy%2Fmcma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprasunroy%2Fmcma/lists"}