{"id":19184212,"url":"https://github.com/princeton-vl/rel3d_render","last_synced_at":"2025-06-14T16:34:51.507Z","repository":{"id":108917440,"uuid":"317970424","full_name":"princeton-vl/Rel3D_Render","owner":"princeton-vl","description":"Code for rendering images for NeurRIPS 2020 paper \"Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3D\"","archived":false,"fork":false,"pushed_at":"2024-12-15T00:33:25.000Z","size":27032,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-08T00:03:32.723Z","etag":null,"topics":["3d-data","neurips-2020","relation-detection","rendering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/princeton-vl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-12-02T19:33:13.000Z","updated_at":"2025-03-17T12:51:51.000Z","dependencies_parsed_at":"2025-04-20T05:40:52.287Z","dependency_job_id":null,"html_url":"https://github.com/princeton-vl/Rel3D_Render","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/princeton-vl%2FRel3D_Render","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/princeton-vl%2FRel3D_Render/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/princeton-vl%2FRel3D_Render/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/princeton-vl%2FRel3D_Render/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/princeton-vl","download_url":"https://codeload.github.com/princeton-vl/Rel3D_Render/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252973680,"owners_count":21834107,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-data","neurips-2020","relation-detection","rendering"],"created_at":"2024-11-09T11:06:44.462Z","updated_at":"2025-05-08T00:04:12.464Z","avatar_url":"https://github.com/princeton-vl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"rel3d.gif\" align=\"right\" width=\"30%\"/\u003e\n\n[**Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3D**](https://arxiv.org/pdf/2012.01634.pdf)\n[Ankit Goyal](http://imankgoyal.github.io), [Kaiyu Yang](https://www.cs.princeton.edu/~kaiyuy/), [Dawei Yang](http://www-personal.umich.edu/~ydawei/), [Jia Deng](https://www.cs.princeton.edu/~jiadeng/) \u003cbr/\u003e\n***Neural Information Processing Systems (NeuRIPS), 2020 (Spotlight)***\n\n*For downloading the dataset and reproducing the results in the paper, use [the main repository](https://github.com/princeton-vl/Rel3D). This repository contains code for rendering images using the 3D data.*\n\n## Getting Started\n\n#### Clone Repository\nFirst clone the repository. We would refer to the directory containing the code as `Rel3D_Render`.\n\n```\ngit clone git@github.com:princeton-vl/Rel3D_Render.git\n```\n\n#### Requirements\nThe code is tested on Linux OS with Python version **3.7**. We recommend using a machine with a GPU for faster rendering.\n\n#### Install Libraries\nWe recommend you to first install [Anaconda](https://anaconda.org/) and create a virtual environment.\n```\nconda create -n rel3d_render python=3.7 -y\n```\n\nActivate the virtual environment and install standard libraries. Make sure you are in `Rel3D_Render`.\n```\nconda activate rel3d_render\nconda install opencv -y\nconda install scipy -y\nconda install pillow -y\n```\n\nInstall third-party dependencies including [trimesh](https://github.com/mikedh/trimesh) and [fcl](https://github.com/BerkeleyAutomation/python-fcl):\n```bash\nconda install trimesh -c conda-forge -y\npip install fcl\npip install networkx\n```\n\nDownload [blender](https://www.blender.org). Our code is tested with blender version 2.81. You can either use the following commands or download blender manually. In case you are using the following commands, make sure you are in the `Rel3D_Render` folder. In case you download blender manually (i.e. by not using the following commands), you would need to update this line (https://github.com/princeton-vl/Rel3D_Render/blob/master/config.py#L4) with the path where you downloaded blender.\n```bash\nwget https://download.blender.org/release/Blender2.81/blender-2.81-linux-glibc217-x86_64.tar.bz2\ntar -jxvf blender-2.81-linux-glibc217-x86_64.tar.bz2\nrm -rf blender-2.81-linux-glibc217-x86_64.tar.bz2\n```\n\n## Download Data\nMake sure you are in `Rel3D_Render`. download.sh script can be used for downloading the data. It also places them at the correct locations. First, use the following command to provide execute permission to the download.sh script.\n\n```\nchmod +x download.sh\n```\n\nThe shapes are required for rendering. The following command downloads them and places them at the correct location.\n```\n./download.sh shapes\n```\n\n(Optional) You can download our pre-generated data using the following command. It places the data in the `data/20200223`. For each sample there is a `.pkl`, `.png` and `.tiff` file. The `.png` and `.tiff` files store rgb and depth respectively at 720X1280 resolution. Information about object masks, bounding box, and surface normal is stored in the `.pkl` file.\n```\n./download.sh data_raw\n```\n**If you get error while executing the above command, you can manually download the data using this [link](https://drive.google.com/uc?id=1ArpgHg8uj90zfpf0ytYDW4S-K9DPLkuI). While using the link, make sure you are logged into your Google Accout. After downloading the zip file, you need to extract it and place the extracted `20200223` folder inside the `data` folder.**\n\n## Code\nThe main script for rendering is `render_scene.py`. The following command can be used to render the entire dataset.\n```\npython render_scene.py --output-folder \u003coutput_path\u003e render --denoising\n```\n\nThe `render_scene.py` can take the following arguments. They are defined [here](https://github.com/princeton-vl/Rel3D_Render/blob/master/render_scene.py#L216-L239). Note that one can use any suitable combination of these arguments.\n\n- Whether to use image denoising is decided by the `denoising` argument. We recommend always using the `--denoising` argument.\n\n- The height and width of the generated images can be changed using the `height` and `width` arguments which can be passed to the `render_scene.py`. For example, the following command can be used to render images of size 100X100.\n  ```\n  python render_scene.py --output-folder \u003coutput_path\u003e --height 100 --width 100 render --denoising\n  ```\n\n- Since rendering the entire dataset on a single machine can take a long time, one can parallelize the process by dividing the dataset into chunks and running multiple processes at the same time. To do so, one can use the `array-index` and `array-total` arguments. `array-total` specifies the number of chunks and  `array-index` specifies the index of the current chunk. `array-index` should change from `0` to `array-total - 1`. For example, the following command can be used to render images of chunk `0` when the data is divided into `10` chunks.\n  ```\n  python render_scene.py --output-folder \u003coutput_path\u003e render --denoising --array-total 10 --array-index 0\n  ```\n\n- To skip pre-generated images, one can use the `skip` argument. For example,\n  ```\n  python render_scene.py --output-folder \u003coutput_path\u003e render --denoising --skip\n  ```\n\n- `start` and `end` can be used to specify which chunk of the data to render. If they are unspecified, the entire dataset is rendered.\n\n- `sample-count` decides the number of samples `blender` uses for rendering. We recommend using the default value. A larger sample count improves the image quality but reduces rendering speed.\n\nWe also provide code for extracting the `3D features` which we used in our MLP baseline (Table 1, Column 8-9). These features can be extracted using the `transforms.py` script with the following command.\n```\npython transforms.py --output-folder \u003coutput_path\u003e --img-path \u003cimg_path\u003e\n```\nHere, `\u003coutput_path\u003e` is the folder where the `.pkl` files are stored and `\u003cimg_path\u003e` is the path of the image for which we want to extract the features. For example, in case one downloaded our pre-generated data using `./download.sh data_raw`, one could use the command `python transforms.py --output-folder ./data/20200223 --img-path \"./data/20200223/behind - Wheel_wheel_3 - Bike_97bb8c467f0efadaf2b5d20b1742ee75 - initialState cam10 rgb.png\"` to extract features for `behind - Wheel_wheel_3 - Bike_97bb8c467f0efadaf2b5d20b1742ee75 - initialState cam10 rgb.png` image.\n\n\nIf you find our research useful, consider citing it:\n```\n@article{goyal2020rel3d,\n  title={Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3D},\n  author={Goyal, Ankit and Yang, Kaiyu and Yang, Dawei and Deng, Jia},\n  journal={Advances in Neural Information Processing Systems},\n  volume={33},\n  year={2020}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprinceton-vl%2Frel3d_render","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprinceton-vl%2Frel3d_render","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprinceton-vl%2Frel3d_render/lists"}