{"id":28676502,"url":"https://github.com/zjunlp/knowledge2data","last_synced_at":"2026-03-09T19:12:42.641Z","repository":{"id":297020528,"uuid":"901904981","full_name":"zjunlp/Knowledge2Data","owner":"zjunlp","description":"Spatial Knowledge Graph-Guided Synthesis for Multimodal LLMs","archived":false,"fork":false,"pushed_at":"2025-06-03T12:50:47.000Z","size":1583,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-06-13T23:04:51.985Z","etag":null,"topics":["generation","kg2data","knowledge-graph","large-language-models","multimodal-large-language-models","natural-language-processing","spatial-knowledge-graph","synthetic-data","visual-question-answering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zjunlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-11T14:34:00.000Z","updated_at":"2025-06-03T12:50:48.000Z","dependencies_parsed_at":"2025-06-03T22:46:40.037Z","dependency_job_id":"146a2adc-110c-4b7d-b290-0b1ee5f2e1a5","html_url":"https://github.com/zjunlp/Knowledge2Data","commit_stats":null,"previous_names":["zjunlp/knowledge2data"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zjunlp/Knowledge2Data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FKnowledge2Data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FKnowledge2Data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FKnowledge2Data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FKnowledge2Data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zjunlp","download_url":"https://codeload.github.com/zjunlp/Knowledge2Data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FKnowledge2Data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30308874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T17:35:44.120Z","status":"ssl_error","status_checked_at":"2026-03-09T17:35:43.707Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generation","kg2data","knowledge-graph","large-language-models","multimodal-large-language-models","natural-language-processing","spatial-knowledge-graph","synthetic-data","visual-question-answering"],"created_at":"2025-06-13T23:04:51.718Z","updated_at":"2026-03-09T19:12:42.635Z","avatar_url":"https://github.com/zjunlp.png","language":"Python","readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1 align=\"center\"\u003e 👉 Knowledge2Data 👈 \u003c/h1\u003e\n\u003cb\u003eSpatial Knowledge Graph-Guided Multimodal Synthesis\u003c/b\u003e\n\n[![Awesome](https://awesome.re/badge.svg)](https://github.com/zjunlp/Knowledge2Data) \n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"figs/figure1.gif\" width=\"90%\"\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/zjunlp/Knowledge2Data\"\u003eProject\u003c/a\u003e •\n  \u003ca href=https://arxiv.org/abs/2505.22633\u003ePaper\u003c/a\u003e •\n  \u003ca href=\"https://huggingface.co/datasets/zjunlp/Knowledge2Data\"\u003eHuggingFace\u003c/a\u003e •\n  \u003ca href=\"#overview\"\u003eOverview\u003c/a\u003e •\n  \u003ca href=\"#quickstart\"\u003eQuickstart\u003c/a\u003e •\n  \u003ca href=\"#citation\"\u003eCitation\u003c/a\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n## Table of Contents\n\n- \u003ca href=\"#news\"\u003eWhat's New\u003c/a\u003e •\n- \u003ca href=\"#overview\"\u003eOverview\u003c/a\u003e •\n- \u003ca href=\"#quickstart\"\u003eQuickstart\u003c/a\u003e •\n- \u003ca href=\"#citation\"\u003eCitation\u003c/a\u003e\n\n## 🔔News\n- **2025-11-01, Our paper has been ACCEPTED for publication as a REGULAR paper in the IEEE TASLP(Transactions on Audio, Speech and Language Processing).**\n- **2025-02-28, We release the paper.**\n---\n\n## 🌟Overview\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"figs/figure2.png\" width=\"90%\"\u003e\n\u003c/div\u003e\n\n\n## ⏩Quickstart\n### Data\nGet training data and test data from HuggingFace: https://huggingface.co/datasets/zjunlp/Knowledge2Data\n### Installation\n```\ngit clone https://github.com/zjunlp/Knowledge2Data\ncd Knowledge2Data\nconda create -n skg python==3.9\nconda activate skg\npip install -r requirements.txt\n```\n\n### Download the models\n#### Download the following models from HuggingFace\n\n| 🎯 Model Name                 | 🤗 HuggingFace                                                            |\n|-------------------------------|---------------------------------------------------------------------------|\n| Diffusers-generation-text-box | [gligen/diffusers-generation-text-box](https://huggingface.co/gligen/diffusers-generation-text-box) |\n| Sam-vit-base                  | [stabilityai/stable-diffusion-xl-refiner-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0)       |\n| Stable-diffusion-xl-refiner   | [facebook/sam-vit-base](https://huggingface.co/facebook/sam-vit-base)      |\n\n### Export the environment variables.\n```shell\ncd src\nexport OPENAI_API_KEY=\"YOUR_API_KEY\"\nexport SKG_HF_MODELS=\"LOCAL_HUGGINGFACE_MODELS_DIR\"\n```\n### Generate Spatial KG and multimodal synthetic data.\n#### Execute script to generate Spatial KG.\n```shell\nsh run_skg.sh\n```\nYou can also customize objects and their spatial relationships to form Spatial KG. Save the file format as a JSON file similar to \"src/data/skg_demo.json\".\n#### Execute script to multimodal synthetic data.\n```shell\nsh run_data.sh\n```\nFor custom data, only the input file parameters \"--input_file\" need to be modified.\n\nYou can find generated data in \"src/data\" and images in \"src/img_generations\" as default.\nIf you want to generate more data, you can modify the parameters including \"--num_scenes\" ([generate_scenes.py](src%2Fgenerate_scenes.py)) and \"--repeats\" ([generate_images.py](src%2Fgenerate_images.py)).\n\n## 🌻Acknowledgement\n\nThis project is based on open-source projects including [LLM-groundedDiffusion](https://github.com/TonyLianLong/LLM-groundedDiffusion). Thanks for their great contributions!\n\n### 🚩Citation\n\nPlease cite the following paper if you use this project in your work.\n\n```bibtex\n@misc{xue2025spatialknowledgegraphguidedmultimodal,\n      title={Spatial Knowledge Graph-Guided Multimodal Synthesis}, \n      author={Yida Xue and Zhen Bi and Jinnan Yang and Jungang Lou and Huajun Chen and Ningyu Zhang},\n      year={2025},\n      eprint={2505.22633},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2505.22633}, \n}\n```\n\n---\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fknowledge2data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzjunlp%2Fknowledge2data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fknowledge2data/lists"}