{"id":21229313,"url":"https://github.com/ndrplz/semiparametric","last_synced_at":"2025-07-10T15:32:01.526Z","repository":{"id":53533699,"uuid":"175432350","full_name":"ndrplz/semiparametric","owner":"ndrplz","description":"[TPAMI 2020] Generating Novel Views of Vehicles via Semi-parametric Guidance. A semi-parametric approach for synthesizing novel views of a rigid object from a single monocular image.","archived":false,"fork":false,"pushed_at":"2021-10-12T22:56:27.000Z","size":69351,"stargazers_count":49,"open_issues_count":4,"forks_count":9,"subscribers_count":5,"default_branch":"master","last_synced_at":"2023-10-20T21:59:57.482Z","etag":null,"topics":["computer-vision","convolutional-neural-networks","deep-learning","image-synthesis","machine-learning","novel-viewpoint-synthesis","pascal3d","semi-parametric","semi-parametric-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ndrplz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-13T14:01:48.000Z","updated_at":"2023-10-20T21:59:57.483Z","dependencies_parsed_at":"2022-08-20T13:01:05.037Z","dependency_job_id":null,"html_url":"https://github.com/ndrplz/semiparametric","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrplz%2Fsemiparametric","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrplz%2Fsemiparametric/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrplz%2Fsemiparametric/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrplz%2Fsemiparametric/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ndrplz","download_url":"https://codeload.github.com/ndrplz/semiparametric/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225645510,"owners_count":17501694,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","convolutional-neural-networks","deep-learning","image-synthesis","machine-learning","novel-viewpoint-synthesis","pascal3d","semi-parametric","semi-parametric-learning"],"created_at":"2024-11-20T23:26:38.901Z","updated_at":"2024-11-20T23:26:39.499Z","avatar_url":"https://github.com/ndrplz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Semi-parametric Object Synthesis\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"gifs/teaser_car.gif\"/ alt=\"Rotating cars\"\u003e\n\u003c/p\u003e\n\n\n## Paper\n\nThis code accompanies the following **TPAMI 2020** paper:\n\n\u003e A. Palazzi, L. Bergamini, S. Calderara and R. Cucchiara, \"Warp and Learn: Novel Views Generation for Vehicles and Other Objects,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2020.3030701\n\navailable in Open Access **[here](https://ieeexplore.ieee.org/document/9222285)**. \n\nIf you find the code / data in this repository useful for your research work, please cite the respective publication.\n\n\u003cp align=\"center\"\u003e\n \u003ctable\u003e\n  \u003ctr\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"https://arxiv.org/abs/1907.10634\" target=\"_blank\"\u003e\u003cimg src=\"imgs/paper_thumb.png\" width=\"160px\"/\u003e\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n  \u003ctd\u003e\u003cpre\u003e  \n@article{palazzi2020warp,\n  title={Warp and Learn: Novel Views Generation for Vehicles and Other Objects},\n  author={Palazzi, Andrea and Bergamini, Luca and Calderara, Simone and Cucchiara, Rita},\n  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},\n  year={2020},\n  publisher={IEEE},\n  doi={10.1109/TPAMI.2020.3030701},\n  ISSN={1939-3539}\n}\n\u003c/pre\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e \n\u003c/p\u003e\n\n## Abstract\n\nIn this work we introduce a new semi-parametric approach for synthesizing novel views of a vehicle starting from a single monocular image. Differently from *parametric* (i.e. entirely learning-based) methods, we show how *a-priori* geometric knowledge about the object and the 3D world can be successfully integrated into a deep learning based image generation framework. As this geometric component is not learnt, we call our approach *semi-parametric*.\n\nIn particular, we exploit man-made object symmetry and piece-wise planarity to integrate rich *a-priori* visual information into the novel viewpoint synthesis process. An Image Completion Network (ICN) is then trained to generate a realistic image starting from this geometric guidance. \n\nThis careful blend between parametric and non-parametric components allows us to i) operate in a real-world scenario, ii) preserve high-frequency visual information such as textures, iii) handle truly arbitrary 3D roto-translations of the input and iv) perform shape transfer to completely different 3D models.\n\nEventually, we show that our approach can be easily complemented with synthetic data and extended to other rigid objects with completely different topology, even in presence of concave structures and holes (e.g. chairs). A comprehensive experimental analysis against state-of-the-art competitors shows the efficacy of our method both from a quantitative and a perceptive point of view.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"imgs/architecture.png\"/ alt=\"Model architecture\"\u003e\n\u003c/p\u003e\n\n---\n\n## Code\n\n### Install\n\nRun the following in a fresh Python 3.6 environment to install all dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\nCode was tested on Ubuntu linux only (16.04, 17.04).\n\n### How to run\n\nTo run the demo code, please download and unzip all the data from [this shared directory](https://drive.google.com/open?id=1ZEDvmKjcEHsPTtQX9GQ8GWecKqfSZsGw) in a `\u003cdata_root\u003e` of your choice.\n\nThe entry point is [`run_rotate.py`](https://github.com/iccv19sub265/semiparametric/blob/master/run_rotate.py). The script expects as mandatory arguments the object class, pascal dataset, pre-trained weights and 3D models dir.\n\nFor the *car* class it can be run as follows:\n```bash\npython run_rotate.py car \u003cdata_root\u003e/pascal_car \u003cdata_root\u003e/car_icn.pth \u003cdata_root\u003e/car_cads --device cpu\n```\nreplace *chair* with *car* to run on the chair class.\n\n### Description and usage\n\nYou should now see a GUI like the following:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"imgs/viewport.png\"/ alt=\"Viewport\" width=\"80%\"\u003e\n\u003c/p\u003e\n\nThe GUI is composed of two windows: the *viewport* and the *output* one.\n\nWhile the focus is on the *viewport*, keyboard can be used to move around the object in spherical coordinates. [Here](https://github.com/iccv19sub265/semiparametric/blob/master/help.txt) the full list of commands is provided. While you move, the *output* shows both Image Completion Network (ICN) inputs (2.5D sketches, appearance prior) and network prediction. Please refer to Sec.3 of the paper for details.\n\n*Notice*: it may happen that when starting the program, open3D does not render anything. This is an initialization issue. In case this happens, just focus on the *viewport* and press spacebar a couple of times until you see both windows rendered properly.\n\n---\n\n## Supplementary animations\n\n### Results for *chair* class\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"gifs/teaser_chair.gif\"/ alt=\"Rotating cars\"\u003e\n\u003c/p\u003e\n\n### Extreme viewpoint transformations (see Sec. 4)\n\n**Due to its *semi-parametric* nature, our method can handle extreme viewpoint changes.**\n\nManipulating radius        |  Manipulation elevation   | Arbitrary rototranslation\n:-------------------------:|:-------------------------:|:-------------------------:\n![](gifs/gif_zoom.gif)     |![](gifs/gif_elevation.gif)     |  ![](gifs/gif_pickup.gif)\n\nChairs backflip                      |\n:-----------------------------------:|\n![](gifs/chair_rotate_and_back.gif)  |\n\n---\n\n## Datasets ([link](https://drive.google.com/open?id=1Dd6Gas3PF9hJoSa_1mkk64mA8fV4vNoK) for download)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"imgs/synth_dataset_banner.png\"/ alt=\"Viewport\" width=\"90%\"\u003e\n\u003c/p\u003e\n\n**We release two datasets of 3D models (*cars*, *chairs*) with annotated 3D keypoints.**\n\n#### Dataset structure\nCurrently, there are 59 annotated models for car and 73 for chair.\n\n3D models come from [ShapeNet](https://www.shapenet.org/) and have been converted in `.ply` format (with colors).\n\nEach example of the datasets is composed of the following components:\n* One `.ply` file containing the 3D model mesh and colors.\n* One `.yaml` file containing the 3D keypoints annotation\n* One `.jpg` image of the model thumbnail.\n\n#### Annotation\nAnnotated keypoints are the ones in [Pascal3D+](http://cvgl.stanford.edu/projects/pascal3d.html): 12 for cars and 10 for chairs.\n\nCar keypoints: *front wheel*, *back wheel*, *upper windshield*, *upper rearwindow*, *front light*, *back trunk* (2x, left and right).\n\nChair keypoints: *back upper*, *seat upper*, *seat lower*, *leg upper*, *leg lower* (2x, left and right).\n\n#### Terms of use\n\nWe believe that research should be as open as possible and we are happy if these datasets can be helpful for your research too. If you use these data, please cite [our research work](https://github.com/ndrplz/semiparametric#paper).\n\nSince 3D models come from ShapeNet database, if you use this dataset you agree to ShapeNet [terms of use](https://www.shapenet.org/terms).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndrplz%2Fsemiparametric","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fndrplz%2Fsemiparametric","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndrplz%2Fsemiparametric/lists"}