{"id":20054839,"url":"https://github.com/thu-ml/crm","last_synced_at":"2025-05-15T13:07:39.463Z","repository":{"id":226917093,"uuid":"769954254","full_name":"thu-ml/CRM","owner":"thu-ml","description":"[ECCV 2024] Single Image to 3D Textured Mesh in 10 seconds with Convolutional Reconstruction Model.","archived":false,"fork":false,"pushed_at":"2024-11-28T09:04:06.000Z","size":4594,"stargazers_count":642,"open_issues_count":25,"forks_count":54,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-05-07T17:39:13.072Z","etag":null,"topics":["3d","aigc","diffusion-models","generative-model","multiview","reconstruction"],"latest_commit_sha":null,"homepage":"https://ml.cs.tsinghua.edu.cn/~zhengyi/CRM/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thu-ml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-10T14:28:50.000Z","updated_at":"2025-05-05T03:48:48.000Z","dependencies_parsed_at":"2024-12-14T11:02:26.986Z","dependency_job_id":"0bdba9aa-3949-4da1-9dca-633b4b98ffa2","html_url":"https://github.com/thu-ml/CRM","commit_stats":null,"previous_names":["thu-ml/crm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thu-ml%2FCRM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thu-ml%2FCRM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thu-ml%2FCRM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thu-ml%2FCRM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thu-ml","download_url":"https://codeload.github.com/thu-ml/CRM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254346624,"owners_count":22055808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","aigc","diffusion-models","generative-model","multiview","reconstruction"],"created_at":"2024-11-13T12:43:58.211Z","updated_at":"2025-05-15T13:07:34.454Z","avatar_url":"https://github.com/thu-ml.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Convolutional Reconstruction Model\n\nOfficial implementation for *CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model*.\n\n**CRM is a feed-forward model which can generate 3D textured mesh in 10 seconds.**\n\n## [Project Page](https://ml.cs.tsinghua.edu.cn/~zhengyi/CRM/) | [Arxiv](https://arxiv.org/abs/2403.05034) | [HF-Demo](https://huggingface.co/spaces/Zhengyi/CRM) | [Weights](https://huggingface.co/Zhengyi/CRM)\n\nhttps://github.com/thu-ml/CRM/assets/40787266/8b325bc0-aa74-4c26-92e8-a8f0c1079382\n\n## Try CRM 🍻\n* Try CRM at [Huggingface Demo](https://huggingface.co/spaces/Zhengyi/CRM).\n* Try CRM at [Replicate Demo](https://replicate.com/camenduru/crm). Thanks [@camenduru](https://github.com/camenduru)! \n\n## Install\n\n### Step 1 - Base\n\nInstall package one by one, we use **python 3.9**\n\n```bash\npip install torch==1.13.0+cu117 torchvision==0.14.0+cu117 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu117\npip install torch-scatter==2.1.1 -f https://data.pyg.org/whl/torch-1.13.1+cu117.html\npip install kaolin==0.14.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-1.13.1_cu117.html\npip install -r requirements.txt\n```\n\nbesides, one by one need to install xformers manually according to the official [doc](https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers) (**conda no need**), e.g.\n\n```bash\npip install ninja\npip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers\n```\n\n### Step 2 - Nvdiffrast\n\nInstall nvdiffrast according to the official [doc](https://nvlabs.github.io/nvdiffrast/#installation), e.g.\n\n```bash\npip install git+https://github.com/NVlabs/nvdiffrast\n```\n\n\n\n## Inference\n\nWe suggest gradio for a visualized inference.\n\n```\ngradio app.py\n```\n\n![image](https://github.com/thu-ml/CRM/assets/40787266/4354d22a-a641-4531-8408-c761ead8b1a2)\n\nFor inference in command lines, simply run\n```bash\nCUDA_VISIBLE_DEVICES=\"0\" python run.py --inputdir \"examples/kunkun.webp\"\n```\nIt will output the preprocessed image, generated 6-view images and CCMs and a 3D model in obj format.\n\n**Tips:** (1) If the result is unsatisfatory, please check whether the input image is correctly pre-processed into a grey background. Otherwise the results will be unpredictable.\n(2) Different from the [Huggingface Demo](https://huggingface.co/spaces/Zhengyi/CRM), this official implementation uses UV texture instead of vertex color. It has better texture than the online demo but longer generating time owing to the UV texturing.\n\n## Train\nWe provide training script for multivew generation and their data requirements.\nTo launch a simple one instance overfit training of multivew gen:\n```shell\naccelerate launch $accelerate_args train.py --config configs/nf7_v3_SNR_rd_size_stroke_train.yaml \\\n    config.batch_size=1 \\\n    config.eval_interval=100\n```\nTo launch a simple one instance overfit training of CCM gen:\n```shell\naccelerate launch $accelerate_args train_stage2.py --config configs/stage2-v2-snr_train.yaml \\\n    config.batch_size=1 \\\n    config.eval_interval=100\n```\n\n### data prepare\nTo specify the data dir modify the following params in the configs/xxxx.yaml\n```yaml\n    base_dir: \u003cpath to multiview piexl image basedir\u003e\n    xyz_base: \u003cpath to related CCM image basedir\u003e\n    caption_csv: \u003cpath to caption.csv\u003e\n```\nThe file tree of basedirs should satisfy as following:\n```shell\nbase_dir\n├── uid1\n│   ├── 000.png\n│   ├── 001.png\n│   ├── 002.png\n│   ├── 003.png\n│   ├── 004.png\n│   ├── 005.png\n├── uid2\n....\n\nxyz_base\n├── uid1\n│   ├── xyz_new_000.png\n│   ├── xyz_new_001.png\n│   ├── xyz_new_002.png\n│   ├── xyz_new_003.png\n│   ├── xyz_new_004.png\n│   └── xyz_new_005.png\n├── uid2\n....\n```\nThe `train_example` dir shows a minimal case of train data and `caption.csv` file.\n\n\n\n## Todo List\n- [x] Release inference code.\n- [x] Release pretrained models.\n- [ ] Optimize inference code to fit in low memery GPU.\n- [x] Upload training code.\n\n## Acknowledgement\n- [ImageDream](https://github.com/bytedance/ImageDream)\n- [nvdiffrast](https://github.com/NVlabs/nvdiffrast)\n- [kiuikit](https://github.com/ashawkey/kiuikit)\n- [GET3D](https://github.com/nv-tlabs/GET3D)\n\n## Citation\n\n```\n@article{wang2024crm,\n  title={CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model},\n  author={Zhengyi Wang and Yikai Wang and Yifei Chen and Chendong Xiang and Shuo Chen and Dajiang Yu and Chongxuan Li and Hang Su and Jun Zhu},\n  journal={arXiv preprint arXiv:2403.05034},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthu-ml%2Fcrm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthu-ml%2Fcrm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthu-ml%2Fcrm/lists"}