{"id":17596226,"url":"https://github.com/lamm-mit/cephalo-idefics2-moe","last_synced_at":"2026-01-08T13:46:11.883Z","repository":{"id":243555718,"uuid":"812696847","full_name":"lamm-mit/Cephalo-Idefics2-MoE","owner":"lamm-mit","description":null,"archived":false,"fork":false,"pushed_at":"2024-06-09T19:20:58.000Z","size":51,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-06T10:31:22.518Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lamm-mit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-09T16:15:12.000Z","updated_at":"2024-06-10T19:49:35.000Z","dependencies_parsed_at":"2024-06-09T21:06:21.899Z","dependency_job_id":null,"html_url":"https://github.com/lamm-mit/Cephalo-Idefics2-MoE","commit_stats":null,"previous_names":["lamm-mit/cephalo-idefics2-moe"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamm-mit%2FCephalo-Idefics2-MoE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamm-mit%2FCephalo-Idefics2-MoE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamm-mit%2FCephalo-Idefics2-MoE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamm-mit%2FCephalo-Idefics2-MoE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lamm-mit","download_url":"https://codeload.github.com/lamm-mit/Cephalo-Idefics2-MoE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246423726,"owners_count":20774820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-22T08:24:27.937Z","updated_at":"2026-01-08T13:46:11.817Z","avatar_url":"https://github.com/lamm-mit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Model Summary\n\nCephalo is a series of multimodal materials science and engineering focused vision large language models (V-LLMs) designed to integrate visual and linguistic data for advanced understanding and interaction in human-AI or multi-agent AI frameworks. \n The model is developed to process diverse inputs, including images and text, facilitating a broad range of applications such as image captioning, visual question answering, and multimodal content generation. The architecture combines a vision encoder model and an autoregressive transformer to process complex natural language understanding. \n\n![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/kl5GWBP9WS0D4uwd1t3S7.png)\n\nThis version of Cephalo, lamm-mit/Cephalo-Idefics2-vision-3x8b-beta, is a Mixture-of-Expert model based on variants and fine-tuned versions of the Idefics-2 model. The basic model architecture is as follows:\n\n![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/b7BK8ZtDzTMsyFDi0wP3w.png)\n\nThis model leverages multiple expert networks to process different parts of the input, allowing for more efficient and specialized computations. For each token in the input sequence, a gating layer computes scores for all experts and selects the top-*k* experts based on these scores. We use a *softmax (..)* activation function to ensure that the weights across the chosen experts sum up to unity.  The output of the gating layer is a set of top-*k* values and their corresponding indices. The selected experts' outputs Y) are then computed and combined using a weighted sum, where the weights are given by the top-*k* values.  This sparse MoE mechanism allows our model to dynamically allocate computational resources, improving efficiency and performance for complex vision-language tasks. \n\nFor this sample model, the model has 20b parameters (three experts, 8b each, and 8b active parameters during inference). The instructions below include a detailed explanation about how other models can be constructed.\n\nModel weights and examples are provided at: [https://huggingface.co/lamm-mit/Cephalo-Phi-3-MoE-vision-128k-3x4b-beta](https://huggingface.co/lamm-mit/Cephalo-Phi-3-MoE-vision-128k-3x4b-beta)\n\n## Example Inference Notebook in Colab\n\n[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lamm-mit/Cephalo-Idefics2-MoE/blob/main/Cephalo-Idefics-2-MoE-3x8b.ipynb)\n\n## Download Idefics-2 MoE Model and Sample inference code\n\n```python\npip install transformers -U\n```\n\nInstall  FlashAttention-2\n```python\npip install flash-attn --no-build-isolation\n```\n\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoProcessor, AutoConfig  \n\ndef count_parameters(model):\n    total_params = sum(p.numel() for p in model.parameters())\n    trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n    #number of parameters in b\n    return total_params/1e9, trainable_params/1e9\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nmodel_name_moe = f\"lamm-mit/Cephalo-Idefics2-vision-3x8b-beta\"\nconfig = AutoConfig.from_pretrained(model_name_moe, trust_remote_code=True)\nprocessor = AutoProcessor.from_pretrained(model_name_moe, trust_remote_code=True) \nmoe_model = AutoModelForCausalLM.from_pretrained(\n    model_name_moe,config=config,\n    trust_remote_code=True,  torch_dtype=torch.bfloat16,   \n).to(device)\n\ncount_parameters(moe_model)\n```\n\nNow use the downloaded MoE model for inference:\n\n```python\nfrom transformers.image_utils import load_image\nDEVICE='cuda'\nimage = load_image(\"https://d2r55xnwy6nx47.cloudfront.net/uploads/2018/02/Ants_Lede1300.jpg\")\n\n# Create inputs\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image, and what is the relevance for materials design? Include a discussion of multi-agent AI.\"},\n        ]\n    },\n]\nprompt = processor.apply_chat_template(messages, add_generation_prompt=True)\n\n# Get inputs using the processor\ninputs = processor(text=prompt, images=[image], return_tensors=\"pt\")\ninputs = {k: v.to(DEVICE) for k, v in inputs.items()}\n\n# Generate\ngenerated_ids = moe_model.generate(**inputs, max_new_tokens=500)\ngenerated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)\n\nprint(generated_texts)\n```\nSample output:\n\n![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/5n6oRNHrfwHkBX0QertZp.png)\n\u003csmall\u003eImage by [Vaishakh Manohar](https://www.quantamagazine.org/the-simple-algorithm-that-ants-use-to-build-bridges-20180226/)\u003c/small\u003e\n\n\u003cpre style=\"white-space: pre-wrap;\"\u003e\nThe image shows a group of ants climbing over a vertical surface. The ants are using their legs and antennae to navigate the surface, demonstrating their ability to adapt to different environments and overcome obstacles. This behavior is relevant for materials design because it highlights the ants' ability to optimize their movements and interactions with their surroundings, which can inspire the development of advanced materials that mimic these natural adaptations.\n\nMulti-agent AI refers to the use of artificial intelligence algorithms to simulate and analyze the behavior of multiple agents, such as ants, in a system. This approach allows for the study of complex interactions and emergent properties that arise from the collective actions of individual agents. By understanding how ants navigate and interact with their environment, researchers can gain insights into the design of materials that exhibit similar properties, such as self-healing, adaptive behavior, and enhanced functionality. \n\nThe image of ants climbing over a vertical surface highlights their ability to adapt and optimize their movements, which can inspire the development of advanced materials that mimic these natural adaptations. Multi-agent AI provides a framework for analyzing and understanding the behavior of these agents, enabling the design of materials that exhibit similar properties.\n\u003c/pre\u003e\n\n## Make a Idefics-2-MoE model from scratch using several pre-trained models\n\nThis section includes detailed instructions to make your own Idefics-2-MoE model. First, download .py files that implement the Idefics-2 Mixture-of-Expert Vision model: \n\n```python\npip install huggingface_hub\n```\n\n```python\nfrom huggingface_hub import HfApi, hf_hub_download\nfrom tqdm.notebook import tqdm\nimport os\nimport shutil\n\n# Repository details\nrepo_id = \"lamm-mit/Cephalo-Idefics2-3x8b-beta\"\napi = HfApi()\n\n# List all files in the repository\nfiles_in_repo = api.list_repo_files(repo_id)\n\n# Filter for .py files\npy_files = [file for file in files_in_repo if file.endswith('.py')]\n\n# Directory to save the downloaded files\nsave_dir = \"./Idefics2_MoE/\"\nos.makedirs(save_dir, exist_ok=True)\n\n# Download each .py file\nfor file_name in tqdm(py_files):\n    file_path = hf_hub_download(repo_id=repo_id, filename=file_name)\n    new_path = os.path.join(save_dir, file_name)\n    shutil.move(file_path, new_path)\n    print(f\"Downloaded: {file_name}\")\n\nprint(\"Download completed.\")\n```\n\nSecond, we will download the models that will form the experts, as well as the base model. As a simple example, we use \n\n1) Materials-science fine-tuned model: lamm-mit/Cephalo-Idefics-2-vision-8b-beta (model_1)\n2) A chatty version: HuggingFaceM4/idefics2-8b-chatty (model_1) (model_2)\n3) A basic variant: HuggingFaceM4/idefics2-8b (model_3)\n\nOne of them (or another model) must be used as base model, from which the vision model, connector, self-attention, etc. are used. From the list of models provided as experts, the feed forward layers are used. Each model will become one expert.  \n\nTo transform an existing model into a Mixture of Experts (MoE) model, we first take the base model use a set of fine-tuned or otherwise trained models to create multiple expert models. Typically, each of the expert models specializes in different aspects of the input data, allowing for greater flexibility and efficiency in processing. To implement this, the original layers of the base model are replaced with modified layers that incorporate the gating and expert mechanisms. A custom configuration class is created to extend the base configuration, adding parameters specific to the MoE setup, such as the number of experts and the number of experts to select in each forward call ($k$). \n\nWithin the algorithm, the original MLP layers in the base model are replaced with a new MoE layer that combines the outputs of the selected experts.  This MoE layer uses the gate scores to select the relevant experts' outputs and combines them into a single output by computing a weighted sum. The modified layers are then integrated back into the model, creating a hybrid architecture that retains the original model's structure but with enhanced capabilities. \n\n```python\nfrom transformers import AutoProcessor, Idefics2ForConditionalGeneration , AutoTokenizer\nfrom transformers import BitsAndBytesConfig\nfrom Idefics2_MoE.moe_idefics2 import *\n\nDEVICE='cuda'\n\nmodel_id_1='lamm-mit/Cephalo-Idefics-2-vision-8b-beta'\n\nmodel_1 = Idefics2ForConditionalGeneration.from_pretrained( model_id_1,\n                                                            torch_dtype=torch.bfloat16, #if your GPU allows\n                                                            _attn_implementation=\"flash_attention_2\", #make sure Flash Attention 2 is installed\n                                                            trust_remote_code=True,\n                                                          ) \nprocessor = AutoProcessor.from_pretrained(\n    f\"{model_id_1}\",\n    do_image_splitting=True\n)\n\nconfig =  AutoConfig.from_pretrained(model_id_1, trust_remote_code=True)\n\nIDEFICS2_CHAT_TEMPLATE = \"{% for message in messages %}{{message['role'].capitalize()}}{% if message['content'][0]['type'] == 'image' %}{{':'}}{% else %}{{': '}}{% endif %}{% for line in message['content'] %}{% if line['type'] == 'text' %}{{line['text']}}{% elif line['type'] == 'image' %}{{ '\u003cimage\u003e' }}{% endif %}{% endfor %}\u003cend_of_utterance\u003e\\n{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}\"\nprocessor.chat_template = IDEFICS2_CHAT_TEMPLATE\n```\n\nNow, load the rest of the models:\n```python\nmodel_id_2='HuggingFaceM4/idefics2-8b-chatty'\n\nmodel_2 = Idefics2ForConditionalGeneration.from_pretrained( model_id_2,\n                                                            torch_dtype=torch.bfloat16, #if your GPU allows\n                                                            _attn_implementation=\"flash_attention_2\", #make sure Flash Attention 2 is installed\n                                                            trust_remote_code=True,\n                                                          ) \n\nmodel_id_3='HuggingFaceM4/idefics2-8b'\n\nmodel_3 = Idefics2ForConditionalGeneration.from_pretrained( model_id_3,\n                                                            torch_dtype=torch.bfloat16, #if your GPU allows\n                                                            _attn_implementation=\"flash_attention_2\", #make sure Flash Attention 2 is installed\n                                                            trust_remote_code=True,\n                                                          ) \n```\nPut on device:\n```python\nmodel_1.to(DEVICE)\nmodel_2.to(DEVICE)\nmodel_3.to(DEVICE)\n```\n\n### Construct MoE \n\nHere we show how a MoE is constructed from the set of expert models loaded earlier. We consider three models, model_1, model_2 and model_3. \n\nFirst, we designate the base model (here we use a deep copy of model_1) and the list of experts. We first create a config, then the moe_model. The config is based on the Idefics2 config from model_1, loaded above. \n\n```python\ndtype = torch.bfloat16  # Desired dtype for new layers\nbase_model = copy.deepcopy(model_1)  # Your base model\nexpert_models = [ model_1,  model_2,  model_3 ]  # List of expert models\n\nmoe_config = Idefics2ForCausalLMMoEConfig(config=config, k=1, num_expert_models=len (expert_models))\nmoe_model = Idefics2ForCausalLMMoE(moe_config, base_model, expert_models,  layer_dtype = dtype) \n\ncount_parameters(expert_models[0]),count_parameters(moe_model)\n```\nDelete models no longer needed:\n```python\ndel model_1\ndel model_2\ndel model_3 \n```\nPut MoE model on device:\n```python\nmoe_model.to(DEVICE)\n```\nTest if it works (untrained, may not produce desirable putput since gating layers have not been trained):\n```python\nfrom transformers.image_utils import load_image\n\nimage = load_image(\"https://d2r55xnwy6nx47.cloudfront.net/uploads/2018/02/Ants_Lede1300.jpg\")\n\n# Create inputs\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image, and what is the relevance for materials design? Include a discussion of multi-agent AI.\"},\n        ]\n    },\n]\nprompt = processor.apply_chat_template(messages, add_generation_prompt=True)\n\n# Get inputs using the processor\ninputs = processor(text=prompt, images=[image], return_tensors=\"pt\")\ninputs = {k: v.to(DEVICE) for k, v in inputs.items()}\n\n# Generate\ngenerated_ids = moe_model.generate(**inputs, max_new_tokens=500)\ngenerated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)\n\nprint(generated_texts)\n```\n\n### Now train MoE gating function\n\nWe train the gating layers by providing sample images/prompts for each of the three experts. Here is a simple example training set: \n\n```python\nimage_1 = Image.open(\"./Image_1.jpg\") \nimage_1a =Image.open(\"./Image_1b.jpg\") \n\nimage_2 = Image.open(\"./Image_2.jpg\") \nimage_2a =Image.open(\"./Image_2b.jpg\") \n\nimage_3 = Image.open(\"./Image_3.jpg\") \nimage_3a =Image.open(\"./Image_3b.jpg\") \n\nprompts_per_expert = [\n    [{\"text\": \"User:\u003cimage\u003eWhat is shown in this image. Explain the importance for materials design.\u003cend_of_utterance\u003eAssistant: The image shows\", \"image\": [image_1]}, \n     {\"text\": \"User:\u003cimage\u003eWhat is shown in this image. Explain the importance for materials design.\u003cend_of_utterance\u003eAssistant: The image shows\", \"image\": [image_1a]}, \n     ],\n\n    [{\"text\": \"User:\u003cimage\u003eWhat is shown in this image. \u003cend_of_utterance\u003eAssistant: The image shows a human.\", \"image\": [image_2]}, \n     {\"text\": \"User:\u003cimage\u003eWhat is shown in this image, and what does it mean in terms of human history? \u003cend_of_utterance\u003eAssistant: The image shows a historical image of human development.\", \"image\": [image_2a]}, \n     ],\n    \n     [{\"text\": \"User:\u003cimage\u003eWhat is shown in this image. Provide a brief answer. \u003cend_of_utterance\u003eAssistant: This is an apple, a fruit with good flavor.\", \"image\": [image_3]}, \n     {\"text\": \"User:\u003cimage\u003eWhat is shown in this image. Brief and concise answer. \u003cend_of_utterance\u003eAssistant: The image shows an apple.\", \"image\": [image_3a]}, \n     ],\n]\n\ngating_layer_params = moe_model.train_gating_layer_params_from_hidden_states(processor,\n                                              prompts_per_expert,\n                                              epochs=1000, loss_steps=100,  lr=5e-5, )\n\n# Set parameters for a specific layer \nmoe_model.set_gating_layer_params(gating_layer_params)\n```\n\n![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/mh4eFDuFsTBOYbjc38PYz.png)\n\nNow that the MoE model has been trained, we can try inference.  Inference after MoE gating layers are trained:\n\n```python\nfrom transformers.image_utils import load_image\n\nimage = load_image(\"https://d2r55xnwy6nx47.cloudfront.net/uploads/2018/02/Ants_Lede1300.jpg\")\n\n# Create inputs\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image, and what is the relevance for materials design? Include a discussion of multi-agent AI.\"},\n        ]\n    },\n]\nprompt = processor.apply_chat_template(messages, add_generation_prompt=True)\n\n# Get inputs using the processor\ninputs = processor(text=prompt, images=[image], return_tensors=\"pt\")\ninputs = {k: v.to(DEVICE) for k, v in inputs.items()}\n\n# Generate\ngenerated_ids = moe_model.generate(**inputs, max_new_tokens=500)\ngenerated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)\n\nprint(generated_texts[0])\n```\n\n### Push to hub and save locally\n\nWe can save the MoE model either in Hugging Face Hub or locally:\n\n```python\nrepo_id='...'\nmoe_name='Cephalo-Idefics2-3x8b-beta'\n\nprocessor.push_to_hub (f'{repo_id}/'+moe_name, )\nmoe_model.push_to_hub (f'{repo_id}/'+merged_name, )\n```\n\nSave locally:\n```python\nprocessor.save_pretrained(moe_name, )\nmoe_model.save_pretrained(moe_name,  )\n```\n\nLoading the model works as done above. Here included again for completeness:\n```python\nmodel_name_moe = f'{repo_id}/'+moe_name\nconfig = AutoConfig.from_pretrained(model_name_moe, trust_remote_code=True)\nprocessor = AutoProcessor.from_pretrained(model_name_moe, trust_remote_code=True) \nmoe_model = AutoModelForCausalLM.from_pretrained(\n    model_name_moe,config=config,\n    trust_remote_code=True,  torch_dtype=torch.bfloat16,   \n).to(device)\n\ncount_parameters(moe_model)\n```\n\n## Citation\n\nPlease cite as:\n\n```bibtex\n@article{Buehler_Cephalo_2024,\n  title={Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design},\n  author={Markus J. Buehler},\n  journal={arXiv preprint arXiv:2405.19076},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flamm-mit%2Fcephalo-idefics2-moe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flamm-mit%2Fcephalo-idefics2-moe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flamm-mit%2Fcephalo-idefics2-moe/lists"}