{"id":21107212,"url":"https://github.com/3dtopia/openlrm","last_synced_at":"2025-04-12T06:13:30.295Z","repository":{"id":243962424,"uuid":"733883451","full_name":"3DTopia/OpenLRM","owner":"3DTopia","description":"An open-source impl. of Large Reconstruction Models","archived":false,"fork":false,"pushed_at":"2024-05-06T09:51:00.000Z","size":14293,"stargazers_count":1087,"open_issues_count":31,"forks_count":63,"subscribers_count":29,"default_branch":"main","last_synced_at":"2025-04-12T06:13:23.593Z","etag":null,"topics":["3d","aigc","computer-vision","generation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/3DTopia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-20T10:52:01.000Z","updated_at":"2025-04-11T10:39:30.000Z","dependencies_parsed_at":"2024-06-12T08:40:16.346Z","dependency_job_id":null,"html_url":"https://github.com/3DTopia/OpenLRM","commit_stats":null,"previous_names":["3dtopia/openlrm"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3DTopia%2FOpenLRM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3DTopia%2FOpenLRM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3DTopia%2FOpenLRM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3DTopia%2FOpenLRM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/3DTopia","download_url":"https://codeload.github.com/3DTopia/OpenLRM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525138,"owners_count":21118619,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","aigc","computer-vision","generation"],"created_at":"2024-11-20T00:36:56.089Z","updated_at":"2025-04-12T06:13:30.275Z","avatar_url":"https://github.com/3DTopia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenLRM: Open-Source Large Reconstruction Models\n\n[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-yellow.svg)](LICENSE)\n[![Weight License](https://img.shields.io/badge/Weight%20License-CC%20By%20NC%204.0-red)](LICENSE_WEIGHT)\n[![LRM](https://img.shields.io/badge/LRM-Arxiv%20Link-green)](https://arxiv.org/abs/2311.04400)\n\n[![HF Models](https://img.shields.io/badge/Models-Huggingface%20Models-bron)](https://huggingface.co/zxhezexin)\n[![HF Demo](https://img.shields.io/badge/Demo-Huggingface%20Demo-blue)](https://huggingface.co/spaces/zxhezexin/OpenLRM)\n\n\u003cimg src=\"assets/rendered_video/teaser.gif\" width=\"75%\" height=\"auto\"/\u003e\n\n\u003cdiv style=\"text-align: left\"\u003e\n    \u003cimg src=\"assets/mesh_snapshot/crop.owl.ply00.png\" width=\"12%\" height=\"auto\"/\u003e\n    \u003cimg src=\"assets/mesh_snapshot/crop.owl.ply01.png\" width=\"12%\" height=\"auto\"/\u003e\n    \u003cimg src=\"assets/mesh_snapshot/crop.building.ply00.png\" width=\"12%\" height=\"auto\"/\u003e\n    \u003cimg src=\"assets/mesh_snapshot/crop.building.ply01.png\" width=\"12%\" height=\"auto\"/\u003e\n    \u003cimg src=\"assets/mesh_snapshot/crop.rose.ply00.png\" width=\"12%\" height=\"auto\"/\u003e\n    \u003cimg src=\"assets/mesh_snapshot/crop.rose.ply01.png\" width=\"12%\" height=\"auto\"/\u003e\n\u003c/div\u003e\n\n## News\n\n- [2024.03.13] Update [training code](openlrm/runners/train) and release [OpenLRM v1.1.1](https://github.com/3DTopia/OpenLRM/releases/tag/v1.1.1).\n- [2024.03.08] We have released the core [blender script](scripts/data/objaverse/blender_script.py) used to render Objaverse images.\n- [2024.03.05] The [Huggingface demo](https://huggingface.co/spaces/zxhezexin/OpenLRM) now uses `openlrm-mix-base-1.1` model by default. Please refer to the [model card](model_card.md) for details on the updated model architecture and training settings.\n- [2024.03.04] Version update v1.1. Release model weights trained on both Objaverse and MVImgNet. Codebase is majorly refactored for better usability and extensibility. Please refer to [v1.1.0](https://github.com/3DTopia/OpenLRM/releases/tag/v1.1.0) for details.\n- [2024.01.09] Updated all v1.0 models trained on Objaverse. Please refer to [HF Models](https://huggingface.co/zxhezexin) and overwrite previous model weights.\n- [2023.12.21] [Hugging Face Demo](https://huggingface.co/spaces/zxhezexin/OpenLRM) is online. Have a try!\n- [2023.12.20] Release weights of the base and large models trained on Objaverse.\n- [2023.12.20] We release this project OpenLRM, which is an open-source implementation of the paper [LRM](https://arxiv.org/abs/2311.04400).\n\n## Setup\n\n### Installation\n```\ngit clone https://github.com/3DTopia/OpenLRM.git\ncd OpenLRM\n```\n\n### Environment\n- Install requirements for OpenLRM first.\n  ```\n  pip install -r requirements.txt\n  ```\n- Please then follow the [xFormers installation guide](https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers) to enable memory efficient attention inside [DINOv2 encoder](openlrm/models/encoders/dinov2/layers/attention.py).\n\n## Quick Start\n\n### Pretrained Models\n\n- Model weights are released on [Hugging Face](https://huggingface.co/zxhezexin).\n- Weights will be downloaded automatically when you run the inference script for the first time.\n- Please be aware of the [license](LICENSE_WEIGHT) before using the weights.\n\n| Model | Training Data | Layers | Feat. Dim | Trip. Dim. | In. Res. | Link |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| openlrm-obj-small-1.1 | Objaverse | 12 | 512 | 32 | 224 | [HF](https://huggingface.co/zxhezexin/openlrm-obj-small-1.1) |\n| openlrm-obj-base-1.1 | Objaverse | 12 | 768 | 48 | 336 | [HF](https://huggingface.co/zxhezexin/openlrm-obj-base-1.1) |\n| openlrm-obj-large-1.1 | Objaverse | 16 | 1024 | 80 | 448 | [HF](https://huggingface.co/zxhezexin/openlrm-obj-large-1.1) |\n| openlrm-mix-small-1.1 | Objaverse + MVImgNet | 12 | 512 | 32 | 224 | [HF](https://huggingface.co/zxhezexin/openlrm-mix-small-1.1) |\n| openlrm-mix-base-1.1 | Objaverse + MVImgNet | 12 | 768 | 48 | 336 | [HF](https://huggingface.co/zxhezexin/openlrm-mix-base-1.1) |\n| openlrm-mix-large-1.1 | Objaverse + MVImgNet | 16 | 1024 | 80 | 448 | [HF](https://huggingface.co/zxhezexin/openlrm-mix-large-1.1) |\n\nModel cards with additional details can be found in [model_card.md](model_card.md).\n\n### Prepare Images\n- We put some sample inputs under `assets/sample_input`, and you can quickly try them.\n- Prepare RGBA images or RGB images with white background (with some background removal tools, e.g., [Rembg](https://github.com/danielgatis/rembg), [Clipdrop](https://clipdrop.co)).\n\n### Inference\n- Run the inference script to get 3D assets.\n- You may specify which form of output to generate by setting the flags `EXPORT_VIDEO=true` and `EXPORT_MESH=true`.\n- Please set default `INFER_CONFIG` according to the model you want to use. E.g., `infer-b.yaml` for base models and `infer-s.yaml` for small models.\n- An example usage is as follows:\n\n  ```\n  # Example usage\n  EXPORT_VIDEO=true\n  EXPORT_MESH=true\n  INFER_CONFIG=\"./configs/infer-b.yaml\"\n  MODEL_NAME=\"zxhezexin/openlrm-mix-base-1.1\"\n  IMAGE_INPUT=\"./assets/sample_input/owl.png\"\n\n  python -m openlrm.launch infer.lrm --infer $INFER_CONFIG model_name=$MODEL_NAME image_input=$IMAGE_INPUT export_video=$EXPORT_VIDEO export_mesh=$EXPORT_MESH\n  ```\n\n### Tips\n- The recommended PyTorch version is `\u003e=2.1`. Code is developed and tested under PyTorch `2.1.2`.\n- If you encounter CUDA OOM issues, please try to reduce the `frame_size` in the inference configs.\n- You should be able to see `UserWarning: xFormers is available` if `xFormers` is actually working.\n\n## Training\n\n### Configuration\n- We provide a sample accelerate config file under `configs/accelerate-train.yaml`, which defaults to use 8 GPUs with `bf16` mixed precision.\n- You may modify the configuration file to fit your own environment.\n\n### Data Preparation\n- We provide the core [Blender script](scripts/data/objaverse/blender_script.py) used to render Objaverse images.\n- Please refer to [Objaverse Rendering](https://github.com/allenai/objaverse-rendering) for other scripts including distributed rendering.\n\n### Run Training\n- A sample training config file is provided under `configs/train-sample.yaml`.\n- Please replace data related paths in the config file with your own paths and customize the training settings.\n- An example training usage is as follows:\n\n  ```\n  # Example usage\n  ACC_CONFIG=\"./configs/accelerate-train.yaml\"\n  TRAIN_CONFIG=\"./configs/train-sample.yaml\"\n\n  accelerate launch --config_file $ACC_CONFIG -m openlrm.launch train.lrm --config $TRAIN_CONFIG\n  ```\n\n### Inference on Trained Models\n- The inference pipeline is compatible with huggingface utilities for better convenience.\n- You need to convert the training checkpoint to inference models by running the following script.\n\n  ```\n  python scripts/convert_hf.py --config \u003cYOUR_EXACT_TRAINING_CONFIG\u003e convert.global_step=null\n  ```\n\n- The converted model will be saved under `exps/releases` by default and can be used for inference following the [inference guide](https://github.com/3DTopia/OpenLRM?tab=readme-ov-file#inference).\n\n## Acknowledgement\n\n- We thank the authors of the [original paper](https://arxiv.org/abs/2311.04400) for their great work! Special thanks to Kai Zhang and Yicong Hong for assistance during the reproduction.\n- This project is supported by Shanghai AI Lab by providing the computing resources.\n- This project is advised by Ziwei Liu and Jiaya Jia.\n\n## Citation\n\nIf you find this work useful for your research, please consider citing:\n```\n@article{hong2023lrm,\n  title={Lrm: Large reconstruction model for single image to 3d},\n  author={Hong, Yicong and Zhang, Kai and Gu, Jiuxiang and Bi, Sai and Zhou, Yang and Liu, Difan and Liu, Feng and Sunkavalli, Kalyan and Bui, Trung and Tan, Hao},\n  journal={arXiv preprint arXiv:2311.04400},\n  year={2023}\n}\n```\n\n```\n@misc{openlrm,\n  title = {OpenLRM: Open-Source Large Reconstruction Models},\n  author = {Zexin He and Tengfei Wang},\n  year = {2023},\n  howpublished = {\\url{https://github.com/3DTopia/OpenLRM}},\n}\n```\n\n## License\n\n- OpenLRM as a whole is licensed under the [Apache License, Version 2.0](LICENSE), while certain components are covered by [NVIDIA's proprietary license](LICENSE_NVIDIA). Users are responsible for complying with the respective licensing terms of each component.\n- Model weights are licensed under the [Creative Commons Attribution-NonCommercial 4.0 International License](LICENSE_WEIGHT). They are provided for research purposes only, and CANNOT be used commercially.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3dtopia%2Fopenlrm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F3dtopia%2Fopenlrm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3dtopia%2Fopenlrm/lists"}