{"id":16635792,"url":"https://github.com/eshansurendra/mag-bert","last_synced_at":"2026-04-26T07:32:47.646Z","repository":{"id":235906042,"uuid":"791503041","full_name":"eshansurendra/MAG-BERT","owner":"eshansurendra","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-25T03:41:13.000Z","size":13886,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-26T19:42:30.589Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eshansurendra.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-24T20:46:23.000Z","updated_at":"2024-05-16T05:15:19.000Z","dependencies_parsed_at":"2024-11-17T15:34:05.692Z","dependency_job_id":"c68dde12-bd0b-4c31-a7d9-37e2b661d164","html_url":"https://github.com/eshansurendra/MAG-BERT","commit_stats":{"total_commits":50,"total_committers":12,"mean_commits":4.166666666666667,"dds":0.52,"last_synced_commit":"10ef60d7c67642a63b86d8570ae4f193d43c87b1"},"previous_names":["eshansurendra/mag-bert"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/eshansurendra/MAG-BERT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eshansurendra%2FMAG-BERT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eshansurendra%2FMAG-BERT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eshansurendra%2FMAG-BERT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eshansurendra%2FMAG-BERT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eshansurendra","download_url":"https://codeload.github.com/eshansurendra/MAG-BERT/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eshansurendra%2FMAG-BERT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32289926,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T06:26:00.361Z","status":"ssl_error","status_checked_at":"2026-04-26T06:25:58.791Z","response_time":129,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T06:04:48.247Z","updated_at":"2026-04-26T07:32:47.624Z","avatar_url":"https://github.com/eshansurendra.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multimodal Adaptation Gate (MAG)\n\nOpen source code for ACL 2020 Paper: [Integrating Multimodal Information in Large Pretrained Transformers](https://www.aclweb.org/anthology/2020.acl-main.214.pdf)\n\nIf you use the model or results, please consider citing the research paper:\n```\n@inproceedings{rahman-etal-2020-integrating,\n    title = \"Integrating Multimodal Information in Large Pretrained Transformers\",\n    author = \"Rahman, Wasifur  and\n      Hasan, Md Kamrul  and\n      Lee, Sangwu  and\n      Bagher Zadeh, AmirAli  and\n      Mao, Chengfeng  and\n      Morency, Louis-Philippe  and\n      Hoque, Ehsan\",\n    booktitle = \"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics\",\n    month = jul,\n    year = \"2020\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://www.aclweb.org/anthology/2020.acl-main.214\",\n    doi = \"10.18653/v1/2020.acl-main.214\",\n    pages = \"2359--2369\",\n    abstract = \"\",\n}\n```\n\n## Getting started\n\n1. Configure `global_configs.py`\n\n   `global_configs.py` defines global constants for runnning experiments. Dimensions of data modality (text, acoustic, visual), cpu/gpu settings, and MAG's injection position. Default configuration is set to **MOSI**. For running experiments on **MOSEI** or on custom dataset, make sure that **ACOUSTIC_DIM** and **VISUAL_DIM** are set approperiately.\n\n   ```python\n   os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0\"\n   os.environ[\"WANDB_PROGRAM\"] = \"multimodal_driver.py\"\n\n   DEVICE = torch.device(\"cuda:0\")\n\n    # MOSI SETTING\n    ACOUSTIC_DIM = 74\n    VISUAL_DIM = 47\n    TEXT_DIM = 768\n\n    # MOSEI SETTING\n    # ACOUSTIC_DIM = 74\n    # VISUAL_DIM = 35\n    # TEXT_DIM = 768\n\n    # CUSTOM DATASET\n    # ACOUSTIC_DIM = ??\n    # VISUAL_DIM = ??\n    # TEXT_DIM = ??\n\n   XLNET_INJECTION_INDEX = 1\n   ```\n\n2. Download datasets\n   Inside `./datasets` folder, run `./download_datasets.sh` to download MOSI and MOSEI datasets\n\n3. Training MAG-BERT / MAG-XLNet on MOSI\n\n   First, install python dependancies using `pip install -r requirements.txt`\n\n   **Training scripts:**\n\n   - MAG-BERT `python multimodal_driver.py --model bert-base-uncased`\n   - MAG-XLNet `python multimodal_driver.py --model xlnet-base-cased`\n\n   By default, `multimodal_driver.py` will attempt to create a [Weights and Biases (W\u0026B)](https://www.wandb.com/) project to log your runs and results. If you wish to disable W\u0026B logging, set environment variable to `WANDB_MODE=dryrun`.\n\n4. Model usage\n\n   We would like to thank [huggingface](https://huggingface.co/) for providing and open-sourcing BERT / XLNet code for developing our models. Note that bert.py / xlnet.py are based on huggingface's implmentation.\n\n   **MAG**\n\n   ```python\n   from modeling import MAG\n\n   hidden_size, beta_shift, dropout_prob = 768, 1e-3, 0.5\n   multimodal_gate = MAG(hidden_size, beta_shift, dropout_prob)\n\n   fused_embedding = multimodal_gate(text_embedding, visual_embedding, acoustic_embedding)\n   ```\n\n   **MAG-BERT**\n\n   ```python\n   from bert import MAG_BertForSequenceClassification\n\n   class MultimodalConfig(object):\n       def __init__(self, beta_shift, dropout_prob):\n           self.beta_shift = beta_shift\n           self.dropout_prob = dropout_prob\n\n   multimodal_config = MultimodalConfig(beta_shift=1e-3, dropout_prob=0.5)\n   model = MAG_BertForSequenceClassification.from_pretrained(\n           'bert-base-uncased', multimodal_config=multimodal_config, num_labels=1,\n       )\n\n   outputs = model(input_ids, visual, acoustic, attention_mask, position_ids)\n   logits = outputs[0]\n   ```\n\n   **MAG-XLNet**\n\n   ```python\n   from xlnet import MAG_XLNetForSequenceClassification\n\n   class MultimodalConfig(object):\n       def __init__(self, beta_shift, dropout_prob):\n           self.beta_shift = beta_shift\n           self.dropout_prob = dropout_prob\n\n   multimodal_config = MultimodalConfig(beta_shift=1e-3, dropout_prob=0.5)\n   model = MAG_XLNet_ForSequenceClassification.from_pretrained(\n           'xlnet-base-cased', multimodal_config=multimodal_config, num_labels=1,\n       )\n\n   outputs = model(input_ids, visual, acoustic, attention_mask, position_ids)\n   logits = outputs[0]\n   ```\n\n   For MAG-BERT / MAG-XLNet usage, visual, acoustic are torch.FloatTensor of shape (batch_size, sequence_length, modality_dim).\n\n   input_ids, attention_mask, position_ids are torch.LongTensor of shape (batch_size, sequence_length). For more details on how these tensors should be formatted / generated, please refer to `multimodal_driver.py`'s `convert_to_features` method and [huggingface's documentation](https://huggingface.co/transformers/preprocessing.html)\n\n## Dataset Format\n\nAll datasets are saved under `./datasets/` folder and is encoded as .pkl file.\nFormat of dataset is as follows:\n\n```python\n{\n    \"train\": [\n        (words, visual, acoustic), label_id, segment,\n        ...\n    ],\n    \"dev\": [ ... ],\n    \"test\": [ ... ]\n}\n```\n\n- words (List[str]): List of words\n- visual (np.array): Numpy array of shape (sequence_len, VISUAL_DIM)\n- acoustic (np.array): Numpy array of shape (seqeunce_len, ACOUSTIC_DIM)\n- label_id (float): Label for data point\n- segment (Any): Unique identifier for each data point\n\nDataset is encoded as python dictionary and saved as .pkl file\n\n```python\nimport pickle as pkl\n\n# NOTE: Use 'wb' mode\nwith open('data.pkl', 'wb') as f:\n    pkl.dump(data, f)\n```\n\n## Contacts\n\n- Wasifur Rahman: rahmanwasifur@gmail.com\n- Sangwu Lee: sangwulee2@gmail.com\n- Kamrul Hasan: mhasan8@cs.rochester.edu\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feshansurendra%2Fmag-bert","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feshansurendra%2Fmag-bert","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feshansurendra%2Fmag-bert/lists"}