{"id":23097579,"url":"https://github.com/google/stereo-magnification","last_synced_at":"2025-08-16T12:32:47.760Z","repository":{"id":53879274,"uuid":"152299547","full_name":"google/stereo-magnification","owner":"google","description":"Code accompanying the SIGGRAPH 2018 paper \"Stereo Magnification: Learning View Synthesis using Multiplane Images\"","archived":true,"fork":false,"pushed_at":"2019-07-19T19:49:07.000Z","size":1406,"stargazers_count":384,"open_issues_count":23,"forks_count":90,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-05-08T21:51:43.551Z","etag":null,"topics":["computer-graphics","computer-vision","deep-learning","multiplane-images","stereo-magnification","view-synthesis"],"latest_commit_sha":null,"homepage":"https://people.eecs.berkeley.edu/~tinghuiz/projects/mpi/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-10-09T18:16:50.000Z","updated_at":"2024-03-23T06:17:10.000Z","dependencies_parsed_at":"2022-08-29T22:20:59.753Z","dependency_job_id":null,"html_url":"https://github.com/google/stereo-magnification","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fstereo-magnification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fstereo-magnification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fstereo-magnification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fstereo-magnification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google","download_url":"https://codeload.github.com/google/stereo-magnification/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230038854,"owners_count":18163310,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-graphics","computer-vision","deep-learning","multiplane-images","stereo-magnification","view-synthesis"],"created_at":"2024-12-16T22:58:25.180Z","updated_at":"2024-12-16T22:58:38.414Z","avatar_url":"https://github.com/google.png","language":"Python","funding_links":[],"categories":["Datasets"],"sub_categories":["Arxiv"],"readme":"# Stereo Magnification: Learning View Synthesis using Multiplane Images\n\nThis code accompanies the paper\n\nStereo Magnification: Learning View Synthesis using Multiplane Images\\\nTinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, Noah Snavely\\\nSIGGRAPH 2018\n\n*Please note that this is not an officially supported Google product.*\n\n## Training the MPI model\n\nThe entry python script for training is train.py. The input flags are\nspecified in two places: 1) `train.py` and 2) `stereomag/loader.py`.\n\nThe input flag `which_color_pred` specifies how to predict the color image at\neach MPI plane:\n\n`bg` [default] - Our default model. The network predicts: 1) weights for blending\n    the background and foreground (reference source image) color images at each\n    plane, 2) the alphas at each plane. 3) a background color image\n`fgbg` - Instead of using the reference source as the foreground image, the\n    network predicts an extra foreground image for blending with the background\n`alpha_only` - No color image (or blending weights) is predicted by the network.\n    The reference source image is used as the color image at each MPI plane.\n`single` - The network predicts a single color image shared for all MPI planes.\n`all` - The network directly outputs the color image at each MPI plane.\n\nYou can also specify which loss to use for training: `pixel` or `vgg` (i.e., the\n`perceptual loss` as measured by differences in VGG features). Note that when\nusing the VGG loss, you will need to download the pre-trained VGG model\n`imagenet-vgg-verydeep-19.mat` available at\n\nhttp://www.vlfeat.org/matconvnet/pretrained/#downloading-the-pre-trained-models\n\nThe path to this file can be set by the `vgg_model_file` flag in `train.py`.\n\n## Testing the MPI model\n\nThe entry python script for testing the models is `test.py`.\n\nOne could specify what to output to disk by concatenating one or more of the\nfollowing (e.g. with '_'): `src_images`, `ref_image`, `tgt_image`, `psv`, `fgbg`, `poses`,\n`intrinsics`, `blend_weights`, `rgba_layers`.\n\n`psv` - the plane sweep volume used as input to the network.\n`fgbg` - foreground and background color images (only valid when\n    `which_color_pred` is either `fgbg` or `bg`)\n`blend_weights` - weights for blending foreground and backgroud color images (only\n    valid when `which_color_pred` is either `fgbg` or `bg`)\n\n## Quantitative evaluation\n\n`evaluate.py` contains sample code for evaluating the view synthesis performance\nbased on the SSIM and PSNR metrics. It assumes that each scene result folder\ncontains a ground-truth target image `tgt_image_*.png` and the synthesized image\n`output_image_*.png`. The script will output a text file summarizing the metrics\ninside the folder FLAGS.result_root.\n\n## Pre-trained models from SIGGRAPH'18 paper\n\nOur pre-trained model can be downloaded into the `models` subdirectory by\nrunning the script `bash scripts/download_model.sh`.\n\n## Running the model on a single image pair\n\nTo run a trained model on a single image pair to generate an MPI, use\n`mpi_from_images.py`. This tool assumes images with the same orientation (as\nwith a rectified stereo pair), but allows for specifying the (x, y, z) offset\nbetween the images. \n\nYou can find example input stereo pairs and command lines for generating results\nin the `examples` directory.\n\n(You must first download the pretrained model or train your own model and place in the `models/` subdirectory)\n\n## Reference examples and results\n\nFor reference, you can find additional example input stereo pairs, as well as corresponding output multi-plane images and view synthesis results used in the paper in this [Google drive link](https://drive.google.com/open?id=1CZGJxRl0GK0js0MbL7cn7tHtdRrtnjOB) (772 MB).\n\n## RealEstate10K dataset\n\nWe have released the [RealEstate10K dataset](https://google.github.io/realestate10k/) suitable for training and testing the MPI model. Note that due to data restrictions, this is not the same version used in our SIGGRAPH'18 paper. However, we are working on updating the results using this public version.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Fstereo-magnification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle%2Fstereo-magnification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Fstereo-magnification/lists"}