{"id":13563582,"url":"https://github.com/apple/ml-gmpi","last_synced_at":"2025-04-05T15:10:07.122Z","repository":{"id":47079149,"uuid":"515737529","full_name":"apple/ml-gmpi","owner":"apple","description":"[ECCV 2022, Oral Presentation] Official PyTorch implementation of GMPI ","archived":false,"fork":false,"pushed_at":"2024-07-25T17:06:56.000Z","size":832,"stargazers_count":343,"open_issues_count":1,"forks_count":32,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-03-29T14:11:19.893Z","etag":null,"topics":["3d","eccv2022","generative-adversarial-network","generative-model","multiplane-images","neural-rendering","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apple.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-19T20:49:58.000Z","updated_at":"2025-02-24T22:20:59.000Z","dependencies_parsed_at":"2024-08-01T13:19:36.092Z","dependency_job_id":null,"html_url":"https://github.com/apple/ml-gmpi","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-gmpi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-gmpi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-gmpi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-gmpi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apple","download_url":"https://codeload.github.com/apple/ml-gmpi/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247353749,"owners_count":20925329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","eccv2022","generative-adversarial-network","generative-model","multiplane-images","neural-rendering","pytorch"],"created_at":"2024-08-01T13:01:20.979Z","updated_at":"2025-04-05T15:10:07.106Z","avatar_url":"https://github.com/apple.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eGenerative Multiplane Images (GMPI)\u003c/h1\u003e\n\u003cp align=\"center\"\u003eMaking a 2D GAN 3D-Aware\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003cb\u003eECCV 2022 (Oral)\u003c/b\u003e\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"90%\" src=\"media/teaser.jpg\"/\u003e\n\u003c/p\u003e\n\n**Generative Multiplane Images: Making a 2D GAN 3D-Aware, ECCV 2022 (Oral).**\u003cbr\u003e\n[Xiaoming Zhao](https://xiaoming-zhao.com/), [Fangchang Ma](https://fangchangma.github.io/), [David Güera Cobo](https://scholar.google.com/citations?user=bckYvFkAAAAJ\u0026hl=en), [Zhile Ren](https://jrenzhile.com/), [Alexander G. Schwing](https://www.alexander-schwing.de/), and [Alex Colburn](https://www.colburn.org/). \n\n### [Project Page](https://xiaoming-zhao.github.io/projects/gmpi/) | [Paper](https://arxiv.org/abs/2207.10642)\n\n## Table of Contents\n\n- [Environment Setup](#environment-setup)\n- [Using Pretrained Checkpoints](#using-pretrained-checkpoints)\n  - [Render a Single Image](#render-a-single-image)\n  - [Render a Video](#render-a-video)\n  - [Extract a Mesh](#extract-a-mesh)\n- [Training and Evaluation](#training-and-evaluation)\n- [Citation](#citation)\n- [License](#license)\n\n## Environment Setup\n\nThis code has been tested on Ubuntu 18.04 with CUDA 10.2.\n\n```\nconda env create -f environment.yml\n```\n\n## Using Pretrained Checkpoints\n\n### Download Checkpoints\n```bash\ncd /path/to/this/repo\nexport GMPI_ROOT=$PWD\n```\n\nPlease download our pretrained checkpoints from [the release page](https://github.com/apple/ml-gmpi/releases) or [this link](https://drive.google.com/drive/folders/1MEIjen0XOIW-kxEMfBUONnKYrkRATSR_?usp=sharing) and place them under `${GMPI_ROOT}/ckpts`. The structure should be:\n```\n.\n+-- ckpts\n|  +-- gmpi_pretrained\n|  |  +-- FFHQ256\n|  |  +-- FFHQ512\n|  |  +-- FFHQ1024\n|  |  +-- AFHQCat\n|  |  +-- MetFaces\n```\n\nWe use the following variables for illustration purposes.\n```bash\n# This can be FFHQ256, FFHQ512, FFHQ1024, AFHQCat, or MetFaces\nexport DATASET_NAME=FFHQ1024\nexport OUTPUT_DIR=${GMPI_ROOT}/ckpts/gmpi_pretrained/${DATASET_NAME}\n\n# Set this to your favourate seed\nexport SEED=589\n\n# - When psi = 1.0 there is no truncation, which is used for quantitative results in the paper.\n# - To obtain better qualitative results, use psi \u003c 1.0.\nexport TRUNCATION_PSI=1.0\n```\n\n### Render a Single Image\nThe following command renders an image `${OUTPUT_DIR}/rendered.png`, along with:\n- `mpi_alpha.png`: alpha maps for all planes,\n- `mpi_rgb.png`: the same RGB texture for all planes,\n- `mpi_rgba.png`: RGB-alpha images for all planes.\n\n```bash\nconda activate gmpi \u0026\u0026 \\\nexport PYTHONPATH=${GMPI_ROOT}:${GMPI_ROOT}/gmpi/models:$PYTHONPATH \u0026\u0026 \\\npython ${GMPI_ROOT}/gmpi/eval/vis/render_video.py \\\n--ckpt_path ${OUTPUT_DIR}/generator.pth \\\n--output_dir ${OUTPUT_DIR} \\\n--seeds ${SEED} \\\n--nplanes 96 \\\n--truncation_psi ${TRUNCATION_PSI} \\\n--exp_config ${OUTPUT_DIR}/config.pth \\\n--render_single_image 1\n```\n\nNote: We use `nplanes = 96` in the paper for reporting quantitative and qualitative results, but GMPI is able to produce high-quality results even with 32 planes. Use a small `nplanes` (e.g., 32) if your run into CUDA out-of-memoory errors. \n\n### Render a Video\nThe following command renders a video in `${OUTPUT_DIR}`, along with:\n- `video_rgb.mp4`: video for the RGB rendering,\n- `video_depth.mp4`: video for the depth rendering.\n\n```bash\nconda activate gmpi \u0026\u0026 \\\nexport PYTHONPATH=${GMPI_ROOT}:${GMPI_ROOT}/gmpi/models:$PYTHONPATH \u0026\u0026 \\\npython ${GMPI_ROOT}/gmpi/eval/vis/render_video.py \\\n--ckpt_path ${OUTPUT_DIR}/generator.pth \\\n--output_dir ${OUTPUT_DIR} \\\n--seeds ${SEED} \\\n--nplanes 96 \\\n--truncation_psi ${TRUNCATION_PSI} \\\n--exp_config ${OUTPUT_DIR}/config.pth \\\n--render_single_image 0 \\\n--horizontal_cam_move 1\n```\n\nNotes:\n- Set `nplanes` to some small number (e.g., 32) if your run into CUDA out-of-memoory errors.\n- Set `horizontal_cam_move` to 0 if you want a video with vertical camera motion.\n\n### Extract a Mesh\nThe following command produces a mesh `${OUTPUT_DIR}/mesh_${TRUNCATION_PSI}.ply`.\n\n```bash\nconda activate gmpi \u0026\u0026 \\\nexport PYTHONPATH=${GMPI_ROOT}:${GMPI_ROOT}/gmpi/models:$PYTHONPATH \u0026\u0026 \\\npython ${GMPI_ROOT}/gmpi/eval/vis/extract_mesh.py \\\n--ckpt_path ${OUTPUT_DIR}/generator.pth \\\n--dataset ${DATASET_NAME} \\\n--save_dir ${OUTPUT_DIR} \\\n--exp_config ${OUTPUT_DIR}/config.pth \\\n--stylegan2_sanity_check 0 \\\n--truncation_psi ${TRUNCATION_PSI} \\\n--seed ${SEED} \\\n--chunk_n_planes -1\n```\n\nNotes:\n- Set `chunk_n_planes` to some small positive numbers (e.g., 64) if your run into CUDA out-of-memoory errors.\n- It may take several minutes to extract a mesh with Marching Cubes, for resolutions of 512 or above. \n\n## Training and Evaluation\nPlease refer to [TRAIN_EVAL.md](./docs/TRAIN_EVAL.md) for more details.\n\n## Citation\n\u003eXiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G. Schwing, and Alex Colburn. Generative Multiplane Images: Making a 2D GAN 3D-Aware. ECCV 2022.\n```\n@inproceedings{zhao-gmpi2022,\n  title = {Generative Multiplane Images: Making a 2D GAN 3D-Aware},\n  author = {Xiaoming Zhao and Fangchang Ma and David Güera and Zhile Ren and Alexander G. Schwing and Alex Colburn},\n  booktitle = {ECCV},\n  year = {2022},\n}\n```\n\n## License\nThis sample code is released under the [LICENSE](LICENSE) terms.\n\n### Disclaimer\nSome of this software was built on Nvidia codebase, as noted within the applicable files, and such Nvidia code is available under its own terms at https://github.com/NVlabs/stylegan2-ada-pytorch.  The authors of this software are not responsible for the contents of third-party websites.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-gmpi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapple%2Fml-gmpi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-gmpi/lists"}