{"id":16271686,"url":"https://github.com/jianzhnie/multimodaltookit","last_synced_at":"2025-07-25T09:37:56.491Z","repository":{"id":103457900,"uuid":"425756158","full_name":"jianzhnie/MultimodalTookit","owner":"jianzhnie","description":" Incorporate Image, Text and  Tabular Data with HuggingFace Transformers","archived":false,"fork":false,"pushed_at":"2022-03-01T02:43:25.000Z","size":1025,"stargazers_count":12,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-13T21:46:21.993Z","etag":null,"topics":["image","machine-learning","multimodal","nlp","tabular-data","text","transformer","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jianzhnie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-11-08T08:31:44.000Z","updated_at":"2024-12-24T12:50:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"40877f5b-83b2-44da-8ca3-fd5d7c530a71","html_url":"https://github.com/jianzhnie/MultimodalTookit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jianzhnie/MultimodalTookit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianzhnie%2FMultimodalTookit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianzhnie%2FMultimodalTookit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianzhnie%2FMultimodalTookit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianzhnie%2FMultimodalTookit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jianzhnie","download_url":"https://codeload.github.com/jianzhnie/MultimodalTookit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianzhnie%2FMultimodalTookit/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266988235,"owners_count":24017313,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-25T02:00:09.625Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image","machine-learning","multimodal","nlp","tabular-data","text","transformer","transformers"],"created_at":"2024-10-10T18:14:25.489Z","updated_at":"2025-07-25T09:37:56.444Z","avatar_url":"https://github.com/jianzhnie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multimodal Transformers\nIncorporate Image, Tabular Data with HuggingFace Transformers\n\nA toolkit for incorporating multimodal data on top of text data for classification\nand regression tasks. It uses HuggingFace transformers as the base model for text features.\nThe toolkit adds a combining module that takes the outputs of the transformer in addition to categorical and numerical features\nto produce rich multimodal features for downstream classification/regression layers.\nGiven a pretrained transformer, the parameters of the combining module and transformer are trained based\non the supervised task. For a brief literature review, check out the accompanying [blog post](https://medium.com/georgian-impact-blog/how-to-incorporate-tabular-data-with-huggingface-transformers-b70ac45fcfb4) on Georgian's Impact Blog.\n\n![](https://drive.google.com/uc?export=view\u0026id=1kyExPDQNkg49NRYgcw2wk8xg4QtQ6Ppt)\n\n\n\n## Installation\nThe code was developed in Python 3.7 with PyTorch and transformers 3.1.\nThe multimodal specific code is in `multimodal_transformers` folder.\n```\npip install multimodal-transformers\n```\n\n## Supported Transformers\nThe following Hugging Face Transformers are supported to handle tabular data. See the documentation [here](https://multimodal-toolkit.readthedocs.io/en/latest/modules/model.html#module-multimodal_transformers.model.tabular_transformers).\n* [BERT](https://huggingface.co/transformers/v3.1.0/model_doc/bert.html) from Devlin et al.:\n[BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) (ACL 2019)\n* [ALBERT](https://huggingface.co/transformers/v3.1.0/model_doc/albert.html) from Lan et al.: [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations\n](https://arxiv.org/abs/1909.11942) (ICLR 2020)\n* [DistilBERT](https://huggingface.co/transformers/v3.1.0/model_doc/distilbert.html) from Sanh et al.:\n[DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108) (NeurIPS 2019)\n* [RoBERTa](https://huggingface.co/transformers/v3.1.0/model_doc/roberta.html)\nfrom Liu et al.: [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692)\n* [XLM](https://huggingface.co/transformers/v3.1.0/model_doc/xlm.html) from Lample et al.: [Cross-lingual Language Model Pretraining\n](https://arxiv.org/abs/1901.07291) (NeurIPS 2019)\n* [XLNET](https://huggingface.co/transformers/v3.1.0/model_doc/xlnet.html) from Yang et al.:\n[XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) (NeurIPS 2019)\n* [XLM-RoBERTa](https://huggingface.co/transformers/v3.1.0/model_doc/xlmroberta.html) from Conneau et al.:\n[Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) (ACL 2020)\n\n## Included Datasets\n\n### Text \u0026\u0026 TabularData\nThis repository also includes two kaggle datasets which contain text data and\nrich tabular features\n* [Women's Clothing E-Commerce Reviews](https://www.kaggle.com/nicapotato/womens-ecommerce-clothing-reviews) for Recommendation Prediction (Classification)\n* [Melbourne Airbnb Open Data](https://www.kaggle.com/tylerx/melbourne-airbnb-open-data) for Price Prediction (Regression)\n* [PetFindermy Adoption Prediction](https://www.kaggle.com/c/petfinder-adoption-prediction) for Pet Adoption Speed Prediction (Multiclass Classification)\n### Image \u0026\u0026 Tabular Data\n* [SIIM-ISIC Melanoma Classification](https://www.kaggle.com/c/siim-isic-melanoma-classification/overview)\n* [PetFindermy Adoption Prediction](https://www.kaggle.com/c/petfinder-adoption-prediction) for Pet Adoption Speed Prediction (Multiclass Classification)\n\n## Working Examples\nTo quickly see these models in action on say one of the above datasets with preset configurations\n```\n$ python main.py ./datasets/Melbourne_Airbnb_Open_Data/train_config.json\n```\n\nOr if you prefer command line arguments run\n```\npython main.py \\\n    --output_dir=./logs/Womens_Clothing_E-Commerce_Reviews \\\n    --task=classification \\\n    --combine_feat_method=individual_mlps_on_cat_and_numerical_feats_then_concat \\\n    --do_train \\\n    --overwrite_output_dir \\\n    --model_name_or_path=bert-base-uncased \\\n    --data_path=./datasets/Womens_Clothing_E-Commerce_Reviews \\\n    --column_info_path=./datasets/Womens_Clothing_E-Commerce_Reviews/column_info.json\n```\n`main.py` expects a `json` file detailing which columns in a dataset contain text,\ncategorical, or numerical input features. It also expects a path to the folder where\nthe data is stored as `train.csv`, and `test.csv`(and if given `val.csv`).For more details on the arguments see\n`multimodal_exp_args.py`.\n### Notebook Introduction\nTo see the modules come together in a notebook: \\\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/georgianpartners/Multimodal-Toolkit/blob/master/notebooks/text_w_tabular_classification.ipynb)\n\n## Included Methods\n| combine feat method |description | requires both cat and num features |\n|:--------------|:-------------------|:-------|\n| text_only | Uses just the text columns as processed by a HuggingFace transformer before final classifier layer(s). Essentially equivalent to HuggingFace's `ForSequenceClassification` models |  False |\n| concat | Concatenate transformer output, numerical feats, and categorical feats all at once before final classifier layer(s) | False |\n| mlp_on_categorical_then_concat | MLP on categorical feats then concat transformer output, numerical feats, and processed categorical feats before final classifier layer(s) | False (Requires cat feats)\n| individual_mlps_on_cat_and_numerical_feats_then_concat | Separate MLPs on categorical feats and numerical feats then concatenation of transformer output, with processed numerical feats, and processed categorical feats before final classifier layer(s). | False\n| mlp_on_concatenated_cat_and_numerical_feats_then_concat | MLP on concatenated categorical and numerical feat then concatenated with transformer output before final classifier layer(s) | True\n| attention_on_cat_and_numerical_feats | Attention based summation of transformer outputs, numerical feats, and categorical feats queried by transformer outputs before final classifier layer(s). | False\n| gating_on_cat_and_num_feats_then_sum | Gated summation of transformer outputs, numerical feats, and categorical feats before final classifier layer(s). Inspired by [Integrating Multimodal Information in Large Pretrained Transformers](https://www.aclweb.org/anthology/2020.acl-main.214.pdf) which performs the mechanism for each token. | False\n| weighted_feature_sum_on_transformer_cat_and_numerical_feats | Learnable weighted feature-wise sum of transformer outputs, numerical feats and categorical feats for each feature dimension before final classifier layer(s) | False\n### Simple baseline model\nIn practice, taking the categorical and numerical features as they are and just tokenizing them and just concatenating them to\nthe text columns as extra text sentences is a strong baseline. To do that here, just specify all the categorical and numerical\ncolumns as text columns and set `combine_feat_method` to `text_only`. For example for each of the included sample datasets in `./datasets`,\nin `train_config.json` change `combine_feat_method` to `text_only` and `column_info_path` to  `./datasets/{dataset}/column_info_all_text.json`.\n\nIn the experiments below this baseline corresponds to Combine Feat Method being `unimodal`.\n\n## Results\nThe following tables shows the results on the two included datasets's respective test sets, by running main.py\nNon specified parameters are the default.\n\n### Review Prediction\nSpecific training parameters can be seen in `datasets/Womens_Clothing_E-Commerce_Reviews/train_config.json`.\n\nThere are **2** text columns, **3** categorical columns, and **3** numerical columns.\n\nModel | Combine Feat Method |F1 | PR AUC\n--------|-------------|---------|-------\nBert Base Uncased | text_only | 0.957 | 0.992\nBert Base Uncased | unimodal | **0.968** | **0.995**\nBert Base Uncased | concat | 0.958 | 0.992\nBert Base Uncased | individual_mlps_on_cat_and_numerical_feats_then_concat | 0.959 | 0.992\nBert Base Uncased | attention_on_cat_and_numerical_feats | 0.959 | 0.992\nBert Base Uncased | gating_on_cat_and_num_feats_then_sum | 0.961 | 0.994\nBert Base Uncased | weighted_feature_sum_on_transformer_cat_and_numerical_feats | 0.962 | 0.994\n\n\n### Pricing Prediction\nSpecific training parameters can be seen in `datasets/Melbourne_Airbnb_Open_Data/train_config.json`.\n\nThere are **3** text columns, **74** categorical columns, and **15** numerical columns.\n\nModel | Combine Feat Method | MAE | RMSE |\n--------|-------------|---------|------- |\nBert Base Multilingual Uncased | text_only | 82.74 | 254.0 |\nBert Base Multilingual Uncased | unimodal | 79.34 | 245.2 |\nBert Base Uncased | concat | **65.68** | 239.3\nBert Base Multilingual Uncased | individual_mlps_on_cat_and_numerical_feats_then_concat | 66.73 | **237.3**\nBert Base Multilingual Uncased | attention_on_cat_and_numerical_feats | 74.72 |246.3\nBert Base Multilingual Uncased | gating_on_cat_and_num_feats_then_sum | 66.64 | 237.8\nBert Base Multilingual Uncased | weighted_feature_sum_on_transformer_cat_and_numerical_feats | 71.19 | 245.2\n\n\n### Pet Adoption Prediction\nSpecific training parameters can be seen in `datasets/PetFindermy_Adoption_Prediction`\nThere are **2** text columns, **14** categorical columns, and **5** numerical columns.\n\nModel | Combine Feat Method | F1_macro | F1_micro |\n--------|-------------|---------|------- |\nBert Base Multilingual Uncased | text_only | 0.088 | 0.281 |\nBert Base Multilingual Uncased | unimodal | 0.089 | 0.283 |\nBert Base Uncased | concat | 0.199 | 0.362\nBert Base Multilingual Uncased | individual_mlps_on_cat_and_numerical_feats_then_concat | 0.244 | 0.352\nBert Base Multilingual Uncased | attention_on_cat_and_numerical_feats | 0.254 | 0.375\nBert Base Multilingual Uncased | gating_on_cat_and_num_feats_then_sum | **0.275** | 0.375\nBert Base Multilingual Uncased | weighted_feature_sum_on_transformer_cat_and_numerical_feats | 0.266 | **0.380**\n\n\n\n## Reference\n- [MMF](https://github.com/facebookresearch/mmf)\nA modular framework for vision \u0026 language multimodal research from Facebook AI Research\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjianzhnie%2Fmultimodaltookit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjianzhnie%2Fmultimodaltookit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjianzhnie%2Fmultimodaltookit/lists"}