{"id":17717416,"url":"https://github.com/wgcban/apt","last_synced_at":"2025-10-28T07:34:00.799Z","repository":{"id":226358934,"uuid":"644177755","full_name":"wgcban/apt","owner":"wgcban","description":"PyTorch Implementation of Attention Prompt Tuning: Parameter-Efficient Adaptation of Pre-Trained Models for Action Recognition","archived":false,"fork":false,"pushed_at":"2024-03-12T16:04:51.000Z","size":965,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-03-13T03:50:40.299Z","etag":null,"topics":["adapter-tuning","computer-vision","hmdb51","linear-probing","prompt-tuning","something-something","tuning-parameters","ucf-101","video-understanding"],"latest_commit_sha":null,"homepage":"https://www.wgcban.com/research#h.wgch37qlgaku","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wgcban.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-23T01:38:49.000Z","updated_at":"2024-03-13T01:56:10.000Z","dependencies_parsed_at":"2024-10-25T17:18:14.707Z","dependency_job_id":"1f9dcf91-dee8-4832-b0ea-ba36f1bab3e6","html_url":"https://github.com/wgcban/apt","commit_stats":null,"previous_names":["wgcban/apt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2Fapt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2Fapt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2Fapt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2Fapt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wgcban","download_url":"https://codeload.github.com/wgcban/apt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232802752,"owners_count":18578684,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adapter-tuning","computer-vision","hmdb51","linear-probing","prompt-tuning","something-something","tuning-parameters","ucf-101","video-understanding"],"created_at":"2024-10-25T14:20:40.035Z","updated_at":"2025-10-04T00:03:18.038Z","avatar_url":"https://github.com/wgcban.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# APT: Attention Prompt Tuning\n\u003e A Parameter-Efficient Adaptation of Pre-Trained Models for Action Recognition ...\u003cbr\u003e\n\n\u003e [Wele Gedara Chaminda Bandara](https://github.com/wgcban), [Vishal M Patel](https://engineering.jhu.edu/vpatel36/team/vishalpatel/)\u003cbr\u003eJohns Hopkins University\n\n\u003e Accepted at [FG'24](https://fg2024.ieee-biometrics.org)\n\n\u003e [Paper (on ArXiv)](https://arxiv.org/abs/2403.06978)\u003cbr\u003e\n\n## Overview of Proposed Method\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src='figs/apt-intro.jpg' width='600' alt\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n    \u003cem\u003eComparison of our Attention Prompt Tuning (APT) for videos action classification with other existing tuning methods:  linear probing, adapter tuning, visual prompt tuning (VPT), and full fine-tuning.\u003c/em\u003e\n\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src='figs/apt-method.jpg' width='300' alt\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n    \u003cem\u003eAttention Prompt Tuning (APT) injects learnable prompts directly into the MHA unlike VPT.\u003c/em\u003e\n\u003c/p\u003e\n\n## Getting Started\n\n### \u003cins\u003eStep 1\u003c/ins\u003e: Conda Environment\n\nSetup the virtual conda environment using the `environment.yml`:\n```\nconda env create -f environment.yml\n```\n\nThen activate the conda environment:\n```\nconda activate apt\n```\n\n### \u003cins\u003eStep 2\u003c/ins\u003e: Download the VideoMAE Pre-trained Models:\n\nWe use [VideoMAE](https://github.com/MCG-NJU/VideoMAE) pretrianed on [Kinetics-400](https://github.com/cvdfoundation/kinetics-dataset) dataset for our experiments.\n\nThe pre-trained models for ViT-Small and ViT-Base backbones can be downloaded from below links:\n\n|  Method  | Extra Data | Backbone | Epoch | \\#Frame |                          Pre-train                           |\n| :------: | :--------: | :------: | :---: | :-----: | :----------------------------------------------------------: |\n| VideoMAE |  ***no***  |  ViT-S   |  1600  | 16x5x3  | [checkpoint](https://drive.google.com/file/d/1nU-H1u3eJ-VuyCveU7v-WIOcAVxs5Hww/view?usp=sharing) |\n| VideoMAE |  ***no***  |  ViT-B   | 1600  | 16x5x3  | [checkpoint](https://drive.google.com/file/d/1tEhLyskjb755TJ65ptsrafUG2llSwQE1/view?usp=sharing) |\n\nIf you need other pre-trained models please refer [MODEL_ZOO.md](https://github.com/wgcban/apt/blob/main/MODEL_ZOO.md).\n\n### \u003cins\u003eStep 3\u003c/ins\u003e: Download the datasets\n\nWe conduct experiments on three action recognition datasets: 1) UCF101 2) HMDB51 3) Something-Something-V2. \n\nPlease refer [DATASETS.md](https://github.com/wgcban/apt/blob/main/DATASET.md) for access to those links and pre-processing steps.\n\n### \u003cins\u003eStep 4\u003c/ins\u003e: Attention Prompt Tuning\n\nWe provide example scripts to run the attention prompt tuning on UCF101, HMDB51, and SSv2 datasets in `scripts/` folder.\n\nInside `scripts/` you can find two folders which corresponds to APT finetuning with ViT-Small and ViT-Base architectures. \n\nTo fine-tune with APT you just need to execute `finetune.sh` file -- which will launch the job with distributed training by\n\n\nFor example, to fine-tune ViT-Base on SSv2 with APT, you may run:\n```\nsh scripts/ssv2/vit_base/finetune.sh\n```\n\nThe `finetune.sh` looks like this:\n\n```bash\n# APT on SSv2\nOUTPUT_DIR='experiments/APT/SSV2/ssv2_videomae_pretrain_base_patch16_224_frame_16x2_tube_mask_ratio_0.9_e2400/adam_mome9e-1_wd1e-5_lr5se-2_pl2_ps0_pe11_drop10'\nDATA_PATH='datasets/ss2/list_ssv2/'\nMODEL_PATH='experiments/pretrain/ssv2_videomae_pretrain_base_patch16_224_frame_16x2_tube_mask_ratio_0.9_e2400/checkpoint.pth'\n\nNCCL_P2P_DISABLE=1 OMP_NUM_THREADS=1 CUDA_VISIBLE_DEVICES=0,1,3,4,5,6,7,8 python -m torch.distributed.launch --nproc_per_node=8 \\\n    run_class_apt.py \\\n    --model vit_base_patch16_224 \\\n    --transfer_type prompt \\\n    --prompt_start 0 \\\n    --prompt_end 11 \\\n    --prompt_num_tokens 2 \\\n    --prompt_dropout 0.1 \\\n    --data_set SSV2 \\\n    --nb_classes 174 \\\n    --data_path ${DATA_PATH} \\\n    --finetune ${MODEL_PATH} \\\n    --log_dir ${OUTPUT_DIR} \\\n    --output_dir ${OUTPUT_DIR} \\\n    --batch_size 8 \\\n    --batch_size_val 8 \\\n    --num_sample 2 \\\n    --input_size 224 \\\n    --short_side_size 224 \\\n    --save_ckpt_freq 10 \\\n    --num_frames 16 \\\n    --opt adamw \\\n    --lr 0.05 \\\n    --weight_decay 0.00001 \\\n    --epochs 100 \\\n    --warmup_epochs 10 \\\n    --test_num_segment 2 \\\n    --test_num_crop 3 \\\n    --dist_eval \\\n    --pin_mem \\\n    --enable_deepspeed \\\n    --prompt_reparam \\\n    --is_aa \\\n    --aa rand-m4-n2-mstd0.2-inc1\n\n```\n\nHere,\n\n- `OUTPUT_DIR`: place where you wants to save the results (i.e., logs and checkpoints)\n- `DATA_PATH`: path to where the dataset is stored\n- `MODEL_PATH`: path to the downloaded videomae pre-trained model\n- specifiy thich gpus (gpu ids) you wants to use for finetuning in `CUDA_VISIBLE_DEVICES=`...\n- `nproc_per_node` is the number of gpus using for fine-tuning\n- `model` is the vit-base (vit_base_patch16_224) or vit-small (vit_small_patch16_224)\n- `transfer_type` specifies which finetuning method to use. 'random' means random initialization, 'end2end' means full end-to-end fine tuning, 'prompt' means APT (ours), 'linear' means linear probing\n- `prompt_start` refers to starting trasnformer block where you add attention prompts. 0 means you start adding learninable prompts from 1st transformer block in vit\n- `prompt_end` refers to ending trasformer block where you stop adding attention prompts. vit-base / vit-small has 12 transformer blocks. hence 11 here means you add prompts until last trasnformer block\n- `data_set` specifies the dataset\n- * all the other parameters are hyperparamters related to apt fine-tuning. \n\n\n## ✏️ Citation\n\nIf you think this project is helpful, please feel free to leave a star and cite our paper:\n\n```bibtex\n@misc{bandara2024attention,\n      title={Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling}, \n      author={Wele Gedara Chaminda Bandara and Vishal M. Patel},\n      year={2024},\n      eprint={2403.06978},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\n\n## ✏️ Disclaimer\n\nThis repocitory is built on top of VideoMAE: https://github.com/MCG-NJU/VideoMAE codebase and we approcite the authors of VideoMAE for making their codebase publically available.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwgcban%2Fapt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwgcban%2Fapt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwgcban%2Fapt/lists"}