{"id":23060194,"url":"https://github.com/opensparsellms/clip-moe","last_synced_at":"2025-08-15T08:31:38.738Z","repository":{"id":264673079,"uuid":"859778590","full_name":"OpenSparseLLMs/CLIP-MoE","owner":"OpenSparseLLMs","description":"CLIP-MoE: Mixture of Experts for CLIP","archived":false,"fork":false,"pushed_at":"2024-10-10T08:48:47.000Z","size":2460,"stargazers_count":17,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-25T17:48:44.581Z","etag":null,"topics":["clip","lvlm","mixture-of-experts","moe","openai-clip"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenSparseLLMs.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-19T09:06:49.000Z","updated_at":"2024-11-19T23:39:01.000Z","dependencies_parsed_at":"2024-11-25T17:48:47.417Z","dependency_job_id":"e53ea944-c18b-4306-8cb7-f99632ede544","html_url":"https://github.com/OpenSparseLLMs/CLIP-MoE","commit_stats":null,"previous_names":["opensparsellms/clip-moe"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenSparseLLMs%2FCLIP-MoE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenSparseLLMs%2FCLIP-MoE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenSparseLLMs%2FCLIP-MoE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenSparseLLMs%2FCLIP-MoE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenSparseLLMs","download_url":"https://codeload.github.com/OpenSparseLLMs/CLIP-MoE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229901617,"owners_count":18141740,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","lvlm","mixture-of-experts","moe","openai-clip"],"created_at":"2024-12-16T03:11:37.416Z","updated_at":"2024-12-16T03:11:37.918Z","avatar_url":"https://github.com/OpenSparseLLMs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling\nThis is the official implementation of Diversified Multiplet Upcycling proposed in [*CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling*](https://arxiv.org/abs/2409.19291)\n\n![Overview of Diversified Multiplet Upcycling](./images/method.png)\n\n## Set up\nTo get started, create and activate a Conda environment, \n```\nconda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=\u003cyour cuda version\u003e\npip install ftfy regex tqdm\npip install git+https://github.com/openai/CLIP.git\ncd CLIP-MoE\npip install -e .\n```\n\n\n## Evaluation\n\nHere we provide the weights of the CLIP-MoE 4 experts with top-2 activation obtained from both ShareGPT4V and Recap-DataComp. The model weights are available on [Hugging Face](https://huggingface.co/MajorDavidZhang/CLIP-MoE/tree/main). Please download the weights. After that you can modify and use the evaluation scripts in `CLIP-MoE/eval`.\n\n## Training\n\n### Datasets\nPlease prepare [ShareGPT4V](https://github.com/ShareGPT4Omni/ShareGPT4V) or [Recap-DataComp](https://github.com/UCSC-VLAA/Recap-DataComp-1B). Notice that our code currently only support regular image folder datasets. If you are using `img2dataset` to prepare the data, please set `--output_format files` instead of `--output_format webdataset`. For Recap-DataComp we only need a small subset (around 1-10M samples) to demonstrate the performance improvement.\n\n### Expert Extraction Using MCL\nMultistage Contrastive Learning contains multiple iterations of training-clustering. To obtain a final CLIP-MoE with N experts, we need to perform N-1 stages of MCL.\n#### Stage0\nFirst, save the base model weights (e.g., OpenAI CLIP ViT-L-14):\n```\nsrun --gres=gpu:8 python CLIP-MoE/train/train_mcl.py --epochs 0 --exp-name clip-mcl-s0\n```\nWhich will save the base model weights to `CLIP-MoE/train/checkpoints` as `clip-mcl-s0_weights.pt`\nSecond, get the image and text representations of the whole training dataset:\n```\nsrun --gres=gpu:8 python CLIP-MoE/train/inference.py --epochs 0 --exp-name clip-mcl-s0 --checkpoint-dir CLIP-MoE/train/checkpoints/clip-mcl-s0_weights.pt\n```\nWhich will generate the representations into `CLIP-MoE/train/save_mcl_tmp/`, splitting by the number of gpus used.\nThird, do clustering on the representations. Please set the parameters accordingly and run `CLIP-MoE/train/cluster.ipynb`. This will save the clustering results and pseudo_labels (the accumulated clustering results of all previous stages) for the next stage of MCL in `CLIP-MoE/train/save_mcl_tmp/clip-mcl_0_pseudo_labels.pt`\n#### Stage1\nFine-tuning the base CLIP model according to the mcl pseudo labels, and freeze all parameters except the mlp layers:\n```\nsrun --gres=gpu:8 python CLIP-MoE/train/train_mcl.py --epochs 1 --exp-name clip-mcl-s1 --MCL-label-path CLIP-MoE/train/save_mcl_tmp/clip-mcl_0_pseudo_labels.pt --lock-except-mlp\n```\nThen do the inference and clustering accordingly, and continue for N-1 stages.\n\n### Initialization of CLIP-MOE and Continuous Fine-tuning\nAfter expert extraction, we can use the obtained series of CLIP models with different mlp layers to build an MoE. First set the checkpoint list in line 26 of `CLIP-MoE/train/train_moe.py`. For example:\n```\ncheckpoints=['CLIP-MoE/train/checkpoints/clip-mcl-s0_weights.pt',\n    'CLIP-MoE/train/checkpoints/clip-mcl-s1_weights.pt',\n    'CLIP-MoE/train/checkpoints/clip-mcl-s2_weights.pt',\n    'CLIP-MoE/train/checkpoints/clip-mcl-s3_weights.pt']\n```\nThen, fine-tune the routers to get an optimized routing strategy. By defult we freeze all parameters except the routers. If you want to do large-scale continuous training, you can try different settings.\n```\nsrun --gres=gpu:8 python CLIP-MoE/train/train_moe.py --epochs 1 --top-k 2 --lock-except-gate\n```\nThis will optimize the routers with both CLIP loss and router balance loss, and finally save a CLIP-MoE model.\n\n\n## Acknowledgement\n- [Long-CLIP](https://github.com/beichenzbc/Long-CLIP) The codebase we built upon.\n- [MCL](https://github.com/MajorDavidZhang/MCL) Original implementation of Multistage Contrastive Learning.\n\n## Citation\nIf you find our work helpful for your research, please consider giving a citation:\n```\n@article{zhang2024clip,\n  title={CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling},\n  author={Zhang, Jihai and Qu, Xiaoye and Zhu, Tong and Cheng, Yu},\n  journal={arXiv preprint arXiv:2409.19291},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensparsellms%2Fclip-moe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopensparsellms%2Fclip-moe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensparsellms%2Fclip-moe/lists"}