{"id":13563530,"url":"https://github.com/Aleph-Alpha/magma","last_synced_at":"2025-04-03T20:31:00.164Z","repository":{"id":40414168,"uuid":"438249975","full_name":"Aleph-Alpha/magma","owner":"Aleph-Alpha","description":"MAGMA - a GPT-style multimodal model that can understand any combination of images and language. NOTE: The freely available model from this repo is only a demo. For the latest multimodal and multilingual models from Aleph Alpha check out our website https://app.aleph-alpha.com","archived":false,"fork":false,"pushed_at":"2023-03-20T06:43:48.000Z","size":1892,"stargazers_count":489,"open_issues_count":7,"forks_count":57,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-31T04:04:43.355Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aleph-Alpha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-12-14T12:46:33.000Z","updated_at":"2025-03-17T12:53:01.000Z","dependencies_parsed_at":"2024-01-14T03:49:14.249Z","dependency_job_id":"b71ad139-8539-47d7-9a78-a99e27fe525e","html_url":"https://github.com/Aleph-Alpha/magma","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fmagma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fmagma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fmagma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fmagma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aleph-Alpha","download_url":"https://codeload.github.com/Aleph-Alpha/magma/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247074452,"owners_count":20879253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:20.303Z","updated_at":"2025-04-03T20:30:59.812Z","avatar_url":"https://github.com/Aleph-Alpha.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning\n\n## Authors\n\n### repo (alphabetical)\n\nConstantin (CoEich), Mayukh (Mayukhdeb), Sid (sdtblck)\n\n### paper\n\nConstantin Eichenberg, Sidney Black, Samuel Weinbach, [Aleph Alpha](https://aleph-alpha.com \"Independent AI R\u0026D\")\n\nLetitia Parcalabescu, Anette Frank, [Heidelberg University](https://www.cl.uni-heidelberg.de \"Computational Linguistics at Heidelberg University\")\n\n\n## Abstract\n\nLarge-scale pretraining is fast becoming the norm in Vision-Language (VL) modeling. However, prevailing VL approaches are limited by the requirement for labeled data and the use of complex multi-step pretraining objectives. We present MAGMA - a simple method for augmenting generative language models with additional modalities using adapter-based finetuning. Building on Frozen, we train a series of VL models that autoregressively generate text from arbitrary combinations of visual and textual input. The pretraining is entirely end-to-end using a single language modeling objective, simplifying optimization compared to previous approaches. Importantly, the language model weights remain unchanged during training, allowing for transfer of encyclopedic knowledge and in-context learning abilities from language pretraining. MAGMA outperforms Frozen on open-ended generative tasks, achieving state of the art results on the OKVQA benchmark and competitive results on a range of other popular VL benchmarks, while pretraining on 0.2% of the number of samples used to train SimVLM.\n\nPaper on arXiv: https://arxiv.org/abs/2112.05253\n\n## Examples (via Aleph Alpha playground)\n\n Photos |  Text \u0026 Technical\n --- | --- \n ![A man covering a woman's eyes to hide a present](examples/magma_present.jpg?raw=true \"Example_1\") |   ![A hand drawn treasure map](examples/magma_treasure.png?raw=true \"Example_3\")\n![A fallen tree is blocking a road](examples/magma_tree.jpg?raw=true \"Example_2\")   | ![A software architecture](examples/magma_oracle.png?raw=true \"Example_4\") \n\n\n ## Model design\n\n![MAGMA model design](examples/model.jpg?raw=true \"MAGMA model design\") \n\n\n## About the repository\n\nIn this repository we share the main parts of the codebase for training and inference of our MAGMA VL model. The main use of the repo is for downloading our pretrained weights and interacting with the model. We include a script for data parallel training with Deepspeed for finetuning our models or training a MAGMA model from scratch.\n\nNOTE: The freely available model from this repo is only a demo. For the latest multimodal and multilingual models from Aleph Alpha check out our [website](https://app.aleph-alpha.com \"Aleph Alpha\").\n\n## Installation\n\nMake sure PyTorch (Ver \u003e= 1.9.0) and Torchvision are installed. See https://pytorch.org/get-started/locally/.\n\nYou can pip install from the git repository with:\n\n```bash\npip install git+https://github.com/Aleph-Alpha/magma.git\n```\n\nMake sure that you also download the config:\n```\nmkdir configs; wget -O configs/MAGMA_v1.yml https://raw.githubusercontent.com/Aleph-Alpha/magma/master/configs/MAGMA_v1.yml\n```\n\nOr if you've cloned the repo, you can install all further requirements by:\n\n```bash\npip install -r requirements.txt\n```\n\n## Checkpoint\n\nWe also publish a model checkpoint that has been used for the publication. It is hosted on our infrastructure and downloads automatically. It can be downloaded manually here: https://bit.ly/aleph_alpha_magma_download \n\t\nThis checkpoint can also be [played around with on a space](https://huggingface.co/spaces/EleutherAI/magma) managed by [Heath Mitchell](https://github.com/Heath123), [AK](https://mobile.twitter.com/ak92501), and [Stella Biderman](https://stellabiderman.com). (This is a 3rd party space, not managed by Aleph Alpha.)\n\n## Loading a model for inference\n\nDownloads the checkpoint file into `checkpoint_path` if it's not already present.\n\n```python\nfrom magma import Magma\nfrom magma.image_input import ImageInput\n\nmodel = Magma.from_checkpoint(\n    config_path = \"configs/MAGMA_v1.yml\",\n    checkpoint_path = \"./mp_rank_00_model_states.pt\",\n    device = 'cuda:0'\n)\n\ninputs =[\n    ## supports urls and path/to/image\n    ImageInput('https://www.art-prints-on-demand.com/kunst/thomas_cole/woods_hi.jpg'),\n    'Describe the painting:'\n]\n\n## returns a tensor of shape: (1, 149, 4096)\nembeddings = model.preprocess_inputs(inputs)  \n\n## returns a list of length embeddings.shape[0] (batch size)\noutput = model.generate(\n    embeddings = embeddings,\n    max_steps = 6,\n    temperature = 0.7,\n    top_k = 0,\n)  \n\nprint(output[0]) ##  A cabin on a lake\n```\n\n## Converting datasets to our format\n\nTo convert an image-caption dataset to our dataset class `magma.datasets.ImgCptDataset`, we suggest:\n\n```python\nfrom magma.datasets.convert_datasets import convert_dataset\n\ndef my_dataset_iterator():\n    \"\"\"\n    Implement an iterator for your dataset that for every datapoint yields a tuple\n    image_path, {\"captions\": [...], \"metadata\": {...}, }, where image_path is the path to the image as a Path object, captions is a list of caption strings and metadata is an optional field.\n    \"\"\"\n\nif __name__ == \"__main__\":\n    convert_dataset(data_dir=\"/target/directory\", ds_iterator=my_dataset_iterator())\n\n```\n\n## How to train MAGMA\n\nRun the training with:\n\n```bash\ndeepspeed train.py --config path_to_my_config\n```\nTo continue training from a deepspeed checkpoint, provide the checkpoint directory in the \"load\" config parameter.\n\nWARNING: By default, instantiating magma via the init method instead of from_checkpoint loads the pretrained CLIP weights but not the pretrained gpt-j weights. For training MAGMA from scratch, download the gpt-j weights from this repo: https://github.com/finetuneanon/transformers and include them in the state dict after initializing the MAGMA model.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAleph-Alpha%2Fmagma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAleph-Alpha%2Fmagma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAleph-Alpha%2Fmagma/lists"}