{"id":24396800,"url":"https://github.com/ai4ce/msg","last_synced_at":"2025-10-11T16:16:33.346Z","repository":{"id":261333211,"uuid":"872617375","full_name":"ai4ce/MSG","owner":"ai4ce","description":"[NeurIPS2024] Multiview Scene Graph (topologically representing a scene from unposed images by interconnected place and object nodes)","archived":false,"fork":false,"pushed_at":"2025-01-09T17:12:54.000Z","size":38632,"stargazers_count":110,"open_issues_count":3,"forks_count":6,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T08:07:39.872Z","etag":null,"topics":["computer-vision","deep-learning","multiview","neurips-2024","scene-graph","scene-representations","visual-navigation"],"latest_commit_sha":null,"homepage":"https://ai4ce.github.io/MSG/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ai4ce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-14T18:51:22.000Z","updated_at":"2025-03-29T07:04:03.000Z","dependencies_parsed_at":"2024-11-06T01:33:35.514Z","dependency_job_id":"76c876dd-4c1b-4f3c-a619-5af54b98fd7e","html_url":"https://github.com/ai4ce/MSG","commit_stats":null,"previous_names":["ai4ce/msg"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4ce%2FMSG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4ce%2FMSG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4ce%2FMSG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4ce%2FMSG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ai4ce","download_url":"https://codeload.github.com/ai4ce/MSG/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247457799,"owners_count":20941906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","multiview","neurips-2024","scene-graph","scene-representations","visual-navigation"],"created_at":"2025-01-19T21:58:30.609Z","updated_at":"2025-10-11T16:16:28.291Z","avatar_url":"https://github.com/ai4ce.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multiview Scene Graph (NeurIPS 2024)\nThis is the official implementation of \n\u003e[Multiview Scene Graph](https://ai4ce.github.io/MSG/) \n\u003e\n\u003eJuexiao Zhang, Gao Zhu, Sihang Li, Xinhao Liu, Haorui Song, Xinran Tang, Chen Feng\n\u003e\n\u003e New York University\n\n### [**[Project page]**](https://ai4ce.github.io/MSG) **|** [**[Paper]**](https://arxiv.org/abs/2410.11187)\n\n![teaser](./media/teaser.jpg)\n\u003cdiv style=\"text-align: center;\"\u003e\n  \u003cimg src=\"media/scene1.gif\" alt=\"Description of GIF\"\u003e\n\u003c/div\u003e\n\n## Implementations\n### Requirements\n\nFirst, setup the environment by running\n```shell\ngit clone https://github.com/ai4ce/MSG.git\ncd msg\nconda create --name msg python=3.11.8\nconda activate msg\npip install -r requirements.txt\n```\nThis `requirements.txt` contains minimum dependencies estimated by running `pipreqs`.\n\n*Alternatively*, to fully replicate the environment you can also run:\n```shell\ngit clone https://github.com/ai4ce/MSG.git\ncd msg\nconda env create -f environment.yml\nconda activate msg\n```\n### Data and weights\n\nMSG data is converted from Apple's [ARKitScenes](https://github.com/apple/ARKitScenes) by transforming its 3D annotations to 2D.\nThe converted dataset can be found at this [Dataset Hub](https://huggingface.co/datasets/ai4ce/MSG) on Huggingface.\nWe have also kept the code snippets for data convertion in `data_preprocess`.\n\nTo use the data, download and unzip the data to `./data/msg`\n- [ ] TODO: specify the data usage. \n\n```shell\nmkdir -p data/msg\n```\n\nWe also provide pretrained [checkpoint](https://huggingface.co/datasets/ai4ce/MSG) of our AoMSG model in the same hub.\n\nTo use the checkpoint, download it to `./exp-results/aomsg`\n- [ ] TODO: specify the checkpoint usage\n\n```shell\nmkdir -p exp-results/aomsg\n\n```\n\n### Inference\n\nTo do inference with the pretrained weights, run:\n\n```shell\npython inference.py --experiment inference\n```\nwhich loads configurations from the file `./configs/experiments/inference.yaml`, where the dataset path and the evaluation checkpoint are specified.\nYou can also specify them via arguments which will overwrite the YAML configs. For example:\n```shell\npython inference.py --experiment inference \\\n--dataset_path PATH/TO/DATASET \\\n--eval_output_dir PATH/TO/MODEL/CHECKPOINT \\\n--eval_chkpt CHECKPOINT/FILE\n```\n\nAdditional to inference, you can also leverage MSG for topological localization. Please see `localization.py` for details.\n\n### Training\n\nTo train the AoMSG model for MSG:\n```shell\npython train.py --experiment aomsg\n```\n\nTo train the SepMSG baselines:\n```shell\npython train.py --experiment sepmsg\n```\nPlease refer to the respective configuration files `./configs/experiments/aomsg.yaml` and `./configs/experiments/sepmsg.yaml` for the detailed settings.\n\nTo resume training of a pretrained checkpoint, set `resume=True` and specify the `resume_path` to the checkpoint in the corresponding YAML configuration files.\n\n\nFor evaluation, simply change the script while keep the same `experiment` configuration, in which `eval_output_dir` and `eval_chkpt` are specified.\n```shell\n# evaluate AoMSG\npython eval.py --experiment aomsg \n# evaluate SepSMG\npython eval.py --experiment sepmsg \n# evaluate SepMSG-direct, which directly use features from froze backbone for MSG\npython eval.py --experiment direct \n```\n\n\u003e **NOTE:**\n\u003e \n\u003e This release focuses on the implementation of MSG. Object detection dependency is not included. \n\u003e To use detection results instead of groundtruth detection, we can specify detection results in files and give the `result_path` as is illustrated in `./configs/experiments/aomsg_gdino.yaml` where detection results obtained from [GroundingDINO](https://github.com/IDEA-Research/GroundingDINO) is used.\n\u003e \n\u003e This means you need to run detection separately and save the results to a path. In the data hub we provide the gdino results for convenience. In the future release, we may include a version incorporating online detection.\n\n## BibTex\n```\n@inproceedings{\nzhang2024multiview,\ntitle={Multiview Scene Graph},\nauthor={Juexiao Zhang and Gao Zhu and Sihang Li and Xinhao Liu and Haorui Song and Xinran Tang and Chen Feng},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=1ELFGSNBGC}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai4ce%2Fmsg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fai4ce%2Fmsg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai4ce%2Fmsg/lists"}