{"id":28676566,"url":"https://github.com/zjunlp/mkg_analogy","last_synced_at":"2025-07-24T18:13:30.988Z","repository":{"id":65766505,"uuid":"543100817","full_name":"zjunlp/MKG_Analogy","owner":"zjunlp","description":"[ICLR 2023] Multimodal Analogical Reasoning over Knowledge Graphs","archived":false,"fork":false,"pushed_at":"2024-07-28T09:30:47.000Z","size":29208,"stargazers_count":84,"open_issues_count":1,"forks_count":10,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-07-28T10:44:59.236Z","etag":null,"topics":["analogical-reasoning","analogy","computer-vision","dataset","iclr","iclr2023","kg","knowledge-graph","language-model","markg","mars","multimodal","natural-language-processing","pre-trained-language-models","prompt","reasoning"],"latest_commit_sha":null,"homepage":"https://zjunlp.github.io/project/MKG_Analogy/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zjunlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-29T12:06:35.000Z","updated_at":"2024-07-28T09:44:17.000Z","dependencies_parsed_at":"2024-06-26T03:37:03.550Z","dependency_job_id":null,"html_url":"https://github.com/zjunlp/MKG_Analogy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zjunlp/MKG_Analogy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FMKG_Analogy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FMKG_Analogy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FMKG_Analogy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FMKG_Analogy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zjunlp","download_url":"https://codeload.github.com/zjunlp/MKG_Analogy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FMKG_Analogy/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259732770,"owners_count":22903087,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analogical-reasoning","analogy","computer-vision","dataset","iclr","iclr2023","kg","knowledge-graph","language-model","markg","mars","multimodal","natural-language-processing","pre-trained-language-models","prompt","reasoning"],"created_at":"2025-06-13T23:05:12.229Z","updated_at":"2025-06-13T23:05:12.630Z","avatar_url":"https://github.com/zjunlp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MKG_Analogy\n\nCode and datasets for the ICLR2023 paper \"[Multimodal Analogical Reasoning over Knowledge Graphs](https://arxiv.org/pdf/2210.00312.pdf)\"\n- ❗New: We provide a Huggingface Demo at [https://huggingface.co/spaces/zjunlp/MKG_Analogy](https://huggingface.co/spaces/zjunlp/MKG_Analogy), have fun!\n- ❗New: We have released the Checkpoints at [Google Drive](https://drive.google.com/drive/folders/1ul9vC93t_e5t_fDj3zzgJKqPoPn_jgsX?usp=share_link) for reproducibility.\n- ❗New: We have released the Powerpoint at [ICLR2023_MKG_Analogy.pdf](resource/ICLR2023_MKG_Analogy.pdf).\n\n## Quick links\n* [MKG_Analogy](#MKG_Analogy)\n    * [Overview](#overview)\n    * [Requirements](#requirements)\n    * [Data Preparation](#data-collection-and-preprocessing)\n    * [Evaluate on Benchmark Mehods](#evaluate-on-benchmark-mehods)\n        * [Multimodal Knowledge Representation Methods](#multimodal-knowledge-representation-methods)\n        * [Transformer-based Methods](#transformer-based-methods)\n    * [Citation](#citation)\n\n\n## Overview\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"resource/mkg_analogy.gif\" width=\"45%\" height=\"45%\" /\u003e\n\u003c/div\u003e\n\nIn this work, we propose a new task of multimodal analogical reasoning over knowledge graph. A overview of the Multimodal Analogical Reasoning task can be seen as follows:\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"resource/task.png\" width=\"75%\" height=\"75%\" /\u003e\n\u003c/div\u003e\n\nWe provide a knowledge graph\nto support and further divide the task into single and blended patterns. Note that the relation marked\nby dashed arrows ($\\dashrightarrow$) and the text around parentheses under images are only for annotation and\nnot provided in the input.\n\n## Requirements\n\n```setup\npip install -r requirements.txt\n```\n\n## Data Collection and Preprocessing\n\nTo support the multimodal analogical reasoning task, we collect a multimodal knowledge graph dataset MarKG and a Multimodal Analogical ReaSoning dataset MARS. A visual outline of the data collection as shown in following figure:\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"resource/flowchart.png\" width=\"75%\" height=\"75%\" /\u003e\n\u003c/div\u003e\n\nWe collect the datasets follow below steps:\n1. Collect Analogy Entities and Relations\n2. Link to Wikidata and Retrieve Neighbors\n3. Acquire and Validate Images\n4. Sample Analogical Reasoning Data\n\nThe statistics of the two datasets are shown in following figures:\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"resource/MARS.png\" width=\"75%\" height=\"75%\" /\u003e\n\u003c/div\u003e\n\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"resource/MarKG.png\" width=\"75%\" height=\"75%\" /\u003e\n\u003c/div\u003e\n\nWe put the text data under `MarT/dataset/`, and the image data can be downloaded through the [Google Drive](https://drive.google.com/file/d/1AqnyrA05vKngfEbhw1mxY5qEoaqiKsC1/view?usp=share_link) or the [Baidu Pan(TeraBox) (code:7hoc)](https://pan.baidu.com/s/1WZvpnTe8m0m-976xRrH90g) and placed on `MarT/dataset/MARS/images`. Please refer to [MarT](MarT/dataset/README.md) for details.\n\nThe expected structure of files is:\n\n```\nMKG_Analogy\n |-- M-KGE\t# multimodal knowledge representation methods\n |    |-- IKRL_TransAE   \n |    |-- RSME\n |-- MarT\n |    |-- data          # data process functions\n |    |-- dataset\n |    |    |-- MarKG    # knowledge graph data\n |    |    |-- MARS     # analogical reasoning data\n |    |-- lit_models    # pytorch_lightning models\n |    |-- models        # source code of models\n |    |-- scripts       # running scripts\n |    |-- tools         # tool function\n |    |-- main.py       # main function\n |-- resources   # image resources\n |-- requirements.txt\n |-- README.md\n\n```\n\n## Evaluate on Benchmark Mehods\n\nWe select some baseline methods to establish the initial benchmark results on MARS, including multimodal knowledge representation methods (IKRL, TransAE, RSME), pre-trained vision-language models (VisualBERT, ViLBERT, ViLT, FLAVA) and a multimodal knowledge graph completion method (MKGformer).\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"resource/model.png\" width=\"75%\" height=\"75%\" /\u003e\n\u003c/div\u003e\n\nIn addition, we follow the structure-mapping theory to regard the Abudction-Mapping-Induction as explicit pipline steps for multimodal knowledge representation methods. As for transformer-based methods, we further propose MarT, a novel framework that implicitly combines these three steps to accomplish the multimodal analogical reasoning task end-to-end, which can avoid error propagation during analogical reasoning. The overview of the baseline methods can be seen in above figure.\n\n### Multimodal Knowledge Representation Methods\n#### 1. [IKRL](https://github.com/thunlp/IKRL)\n\nWe reproduce the IKRL models via TransAE framework, to evaluate on IKRL, running following code:\n```bash\ncd M-KGE/IKRL_TransAE\npython IKRL.py\n```\n\nYou can choose pre-train/fine-tune and TransE/ANALOGY  by modifing `finetune` and `analogy` parameters in `IKRL.py`, respectively.\n\n\n#### 2. [TransAE](https://github.com/ksolaiman/TransAE)\n\nTo evaluate on IKRL, running following code:\n```bash\ncd M-KGE/IKRL_TransAE\npython TransAE.py\n```\n\nYou can choose pre-train/fine-tune and TransE/ANALOGY  by modifing `finetune` and `analogy` parameters in `TransAE.py`, respectively.\n\n#### 3. [RSME](https://github.com/wangmengsd/RSME)\nWe only provide part of the data for RSME. To evaluate on RSME, you need to generate the full data by following scripts:\n```bash\ncd M-KGE/RSME\npython image_encoder.py  # -\u003e analogy_vit_best_img_vec.pickle\npython utils.py          # -\u003e img_vec_id_analogy_vit.pickle\n```\n\nFirstly, pre-train the models over MarKG:\n```bash\nbash run.sh\n```\nThen modify the `--checkpoint` parameter and fine-tune the models on MARS:\n```bash\nbash run_finetune.sh\n```\n\nMore training details about the above models can refer to their [offical repositories](https://github.com/wangmengsd/RSME).\n\n### Transformer-based Methods\n\nWe leverage the MarT framework for transformer-based models. MarT contains two steps: pre-train and fine-tune. \n\nTo train the models fast, we encode the image data in advance with this script (Note that the size of the encoded data is about 7GB):\n```bash\ncd MarT\npython tools/encode_images_data.py\n```\n\nTaking MKGformer as an example, first pre-train the model via following script:\n```bash\nbash scripts/run_pretrain_mkgformer.sh\n```\n\nAfter pre-training, fine-tune the model via following script:\n```bash\nbash scripts/run_finetune_mkgformer.sh\n```\n\n\u0026#x1F353; We provide the best checkpoints of transformer-based models during the fine-tuning and pre-training phrases at this [Google Drive](https://drive.google.com/drive/folders/1ul9vC93t_e5t_fDj3zzgJKqPoPn_jgsX?usp=share_link). Download them and add `--only_test` in `scripts/run_finetune_xxx.sh` for testing experiments.\n\n## Citation\n\nIf you use or extend our work, please cite the paper as follows:\n\n```bibtex\n@inproceedings{\nzhang2023multimodal,\ntitle={Multimodal Analogical Reasoning over Knowledge Graphs},\nauthor={Ningyu Zhang and Lei Li and Xiang Chen and Xiaozhuan Liang and Shumin Deng and Huajun Chen},\nbooktitle={The Eleventh International Conference on Learning Representations },\nyear={2023},\nurl={https://openreview.net/forum?id=NRHajbzg8y0P}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fmkg_analogy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzjunlp%2Fmkg_analogy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fmkg_analogy/lists"}