{"id":20731458,"url":"https://github.com/opendrivelab/maskalign","last_synced_at":"2025-06-22T22:04:06.651Z","repository":{"id":107747374,"uuid":"566637231","full_name":"OpenDriveLab/maskalign","owner":"OpenDriveLab","description":"[CVPR 2023] Official repository for paper \"Stare at What You See: Masked Image Modeling without Reconstruction\"","archived":false,"fork":false,"pushed_at":"2023-12-06T13:12:44.000Z","size":629,"stargazers_count":70,"open_issues_count":0,"forks_count":6,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-06-14T14:24:02.604Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"github":["OpenDriveLab"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2022-11-16T04:53:00.000Z","updated_at":"2025-06-10T12:56:17.000Z","dependencies_parsed_at":"2023-03-13T14:33:25.740Z","dependency_job_id":null,"html_url":"https://github.com/OpenDriveLab/maskalign","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OpenDriveLab/maskalign","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2Fmaskalign","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2Fmaskalign/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2Fmaskalign/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2Fmaskalign/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/maskalign/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2Fmaskalign/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261374407,"owners_count":23148975,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T05:14:55.474Z","updated_at":"2025-06-22T22:04:01.617Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"Python","funding_links":["https://github.com/sponsors/OpenDriveLab"],"categories":[],"sub_categories":[],"readme":"# MaskAlign (CVPR 2023)\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"figs/framework.png\" alt=\"statistics\" width=\"80%\"/\u003e\n\u003c/p\u003e\n\n\nThis is the official PyTorch repository for CVPR 2023 paper [Stare at What You See: Masked Image Modeling without Reconstruction](https://arxiv.org/abs/2211.08887):\n```\n@article{xue2022stare,\n  title={Stare at What You See: Masked Image Modeling without Reconstruction},\n  author={Xue, Hongwei and Gao, Peng and Li, Hongyang and Qiao, Yu and Sun, Hao and Li, Houqiang and Luo, Jiebo},\n  journal={arXiv preprint arXiv:2211.08887},\n  year={2022}\n}\n```\n\n* This repo is a modification on the [MAE repo](https://github.com/facebookresearch/mae). Installation and preparation follow that repo.\n\n* The teacher models in this repo are called from [Huggingface](https://huggingface.co/). Please install transformers package by running: \u003cbr\u003e `pip install transformers`.\n\n## Pre-training\n\nTo pre-train ViT-base (recommended default) with **distributed training**, run the following on 8 GPUs:\n\n```\npython -m torch.distributed.launch --nproc_per_node=8 main_pretrain.py \\\n    --batch_size 128 \\\n    --model mae_vit_base_patch16 \\\n    --blr 1.5e-4 \\\n    --min_lr 1e-5 \\\n    --data_path ${IMAGENET_DIR} \\\n    --output_dir ${OUTPUT_DIR} \\\n    --target_norm whiten \\\n    --loss_type smoothl1 \\\n    --drop_path 0.1 \\\n    --head_type linear \\\n    --epochs 200 \\\n    --warmup_epochs 20 \\\n    --mask_type attention \\\n    --mask_ratio 0.7 \\\n    --loss_weights top5 \\\n    --fusion_type linear \\\n    --teacher_model openai/clip-vit-base-patch16\n```\n\n- Here the effective batch size is 128 (`batch_size` per gpu) * 8 (gpus) = 1024. If memory or # gpus is limited, use `--accum_iter` to maintain the effective batch size, which is `batch_size` (per gpu) * `nodes` * 8 (gpus) * `accum_iter`.\n- `blr` is the base learning rate. The actual `lr` is computed by the [linear scaling rule](https://arxiv.org/abs/1706.02677): `lr` = `blr` * effective batch size / 256.\n- This repo will automatically resume the checkpoints by keeping a \"latest checkpoint\".\n\nTo train ViT-Large, please set `--model mae_vit_large_patch16` and `--drop_path 0.2`. Currently, this repo supports three teacher models: `--teacher_model ${TEACHER}`, where `${TEACHER} in openai/clip-vit-base-patch16, openai/clip-vit-large-patch14 and facebook/dino-vitb16`.\n\n## Fine-tuning\n\nGet our pre-trained checkpoints from [here](ModelCard.md).\n\nTo fine-tune ViT-base (recommended default) with **distributed training**, run the following on 8 GPUs:\n```\npython -m torch.distributed.launch --nproc_per_node=8 main_finetune.py \\\n    --epochs 100 \\\n    --batch_size 128 \\\n    --model vit_base_patch16 \\\n    --blr 3e-4 \\\n    --layer_decay 0.55 \\\n    --weight_decay 0.05 \\\n    --drop_path 0.2 \\\n    --reprob 0.25 \\\n    --mixup 0.8 \\\n    --cutmix 1.0 \\\n    --dist_eval \\\n    --finetune ${PT_CHECKPOINT} \\\n    --data_path ${IMAGENET_DIR} \\\n    --output_dir ${OUTPUT_DIR}\n```\n\n- Here the effective batch size is 128 (`batch_size` per gpu) * 8 (gpus) = 1024.\n- `blr` is the base learning rate. The actual `lr` is computed by the [linear scaling rule](https://arxiv.org/abs/1706.02677): `lr` = `blr` * effective batch size / 256.\n\nTo fine-tune ViT-Large, please set `--model vit_large_patch16 --epochs 50 --drop_path 0.4 --layer_decay 0.75 --blr 3e-4`.\n\n\n## Linear Probing\n\nRun the following on 8 GPUs:\n```\npython -m torch.distributed.launch --nproc_per_node=8 main_linprobe.py \\\n    --epochs 90 \\\n    --batch_size 2048 \\\n    --model vit_base_patch16 \\\n    --blr 0.025 \\\n    --weight_decay 0.0 \\\n    --dist_eval \\\n    --finetune ${PT_CHECKPOINT} \\\n    --data_path ${IMAGENET_DIR} \\\n    --output_dir ${OUTPUT_DIR}\n```\n- Here the effective batch size is 2048 (`batch_size` per gpu) * 8 (gpus) = 16384.\n- `blr` is the base learning rate. The actual `lr` is computed by the [linear scaling rule](https://arxiv.org/abs/1706.02677): `lr` = `blr` * effective batch size / 256.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Fmaskalign","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopendrivelab%2Fmaskalign","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Fmaskalign/lists"}