{"id":18621388,"url":"https://github.com/ethz-asl/ucsa_neural_rendering","last_synced_at":"2025-09-09T06:42:57.265Z","repository":{"id":171909809,"uuid":"647013583","full_name":"ethz-asl/ucsa_neural_rendering","owner":"ethz-asl","description":"[CVPR 2023] Unsupervised Continual Semantic Adaptation through Neural Rendering","archived":false,"fork":false,"pushed_at":"2023-10-18T13:13:25.000Z","size":1478,"stargazers_count":39,"open_issues_count":1,"forks_count":2,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-06-07T14:44:47.187Z","etag":null,"topics":["3d-scene-understanding","continual-learning","cvpr2023","nerf","neural-rendering","scannet","semantic-segmentation","semantic-understanding"],"latest_commit_sha":null,"homepage":"https://ethz-asl.github.io/ucsa_neural_rendering/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ethz-asl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-05-29T21:48:42.000Z","updated_at":"2025-05-28T22:41:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"31238c79-cb46-4353-950a-a8f06e10dfe6","html_url":"https://github.com/ethz-asl/ucsa_neural_rendering","commit_stats":null,"previous_names":["ethz-asl/ucsa_neural_rendering"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ethz-asl/ucsa_neural_rendering","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-asl%2Fucsa_neural_rendering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-asl%2Fucsa_neural_rendering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-asl%2Fucsa_neural_rendering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-asl%2Fucsa_neural_rendering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ethz-asl","download_url":"https://codeload.github.com/ethz-asl/ucsa_neural_rendering/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-asl%2Fucsa_neural_rendering/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274254981,"owners_count":25250552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-09T02:00:10.223Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-scene-understanding","continual-learning","cvpr2023","nerf","neural-rendering","scannet","semantic-segmentation","semantic-understanding"],"created_at":"2024-11-07T04:11:28.433Z","updated_at":"2025-09-09T06:42:57.206Z","avatar_url":"https://github.com/ethz-asl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eUnsupervised Continual Semantic Adaptation through Neural Rendering\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cstrong\u003e\u003ca href=\"https://scholar.google.com/citations?user=Asc7j9oAAAAJ\u0026hl=en\u0026oi=ao\"\u003eZhizheng Liu\u003c/a\u003e\u003c/strong\u003e*, \u003cstrong\u003e\u003ca href=\"https://scholar.google.com/citations?user=qwSANZoAAAAJ\u0026hl=en\u0026oi=ao\"\u003eFrancesco Milano\u003c/a\u003e\u003c/strong\u003e*, \u003cstrong\u003e\u003ca href=\"https://scholar.google.com/citations?user=e5uPDzcAAAAJ\u0026hl=en\u0026oi=ao\"\u003eJonas Frey\u003c/a\u003e\u003c/strong\u003e, \u003cstrong\u003e\u003ca href=\"https://asl.ethz.ch/\"\u003eRoland Siegwart\u003c/a\u003e\u003c/strong\u003e, \u003cstrong\u003e\u003ca href=\"https://hermannblum.net/\"\u003eHermann Blum\u003c/a\u003e\u003c/strong\u003e, \u003cstrong\u003e\u003ca href=\"https://n.ethz.ch/~cesarc/\"\u003eCesar Cadena\u003c/a\u003e\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003ch2 align=\"center\"\u003eCVPR 2023\u003c/h2\u003e\n\u003ch3 align=\"center\"\u003e\u003ca href=\"https://arxiv.org/abs/2211.13969\"\u003ePaper\u003c/a\u003e | \u003ca href=\"https://www.youtube.com/watch?v=XfNLsl8ATNY\"\u003eVideo\u003c/a\u003e | \u003ca href=\"https://ethz-asl.github.io/ucsa_neural_rendering/\"\u003eProject Page\u003c/a\u003e\u003c/h3\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"\"\u003e\n    \u003cimg src=\"./assets/teaser.png\" alt=\"Unsupervised Continual Semantic Adaptation through Neural Rendering\" width=\"90%\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\nWe present a framework to improve semantic scene understanding for agents that are deployed across a _sequence of scenes_. In particular, our method performs unsupervised continual semantic adaptation by jointly training a _2-D segmentation model_ and a _Semantic-NeRF network_.\n\n- Our framework allows successfully adapting the 2-D segmentation model across _multiple, previously unseen scenes_ and with _no ground-truth supervision_, reducing the domain gap in the new scenes and improving on the initial performance of the model.\n- By rendering training and novel views, the pipeline can effectively _mitigate forgetting_ and even _gain additional knowledge_ about the previous scenes.\n\n## Table of Contents\n\n1. [Installation](#installation)\n2. [Running experiments](#running-experiments)\n3. [Citation](#citation)\n4. [Acknowledgements](#acknowledgements)\n5. [Contact](#contact)\n\n## Installation\n\n### Workspace setup\n\nWe recommend configuring your workspace with a conda environment. You can then install the project and its dependencies as follows. The instructions were tested on Ubuntu 20.04 and 22.04, with CUDA 11.3.\n\n- Clone this repo to a folder of your choice, which in the following we will refer to with the environmental variable `REPO_ROOT`:\n  ```bash\n  export REPO_ROOT=\u003cFOLDER_PATH_HERE\u003e\n  cd ${REPO_ROOT};\n  git clone git@github.com:ethz-asl/nr_semantic_segmentation.git\n  ```\n- Create a conda environment and install [PyTorch](https://pytorch.org/), [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn) and other dependencies:\n\n  ```bash\n  conda create -n nr4seg python=3.8\n  conda activate nr4seg\n  python -m pip install --upgrade pip\n\n  # For CUDA 11.3.\n  conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch\n  # Install tiny-cuda-nn\n  pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch\n\n  pip install -r requirements.txt\n\n  python setup.py develop\n  ```\n\n### Setting up the dataset\n\nWe use the [ScanNet v2](http://www.scan-net.org/) [1] dataset for our experiments.\n\n\u003e [1] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner, \"ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes\", in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 2432-2443, 2017.\n\n#### Dataset download\n\n- To get started, visit the official [ScanNet](https://github.com/ScanNet/ScanNet#scannet-data) dataset website to obtain the data downloading permission and script. We keep all the ScanNet data in the `${REPO_ROOT}/data/scannet` folder. You may use a symbolic link if necessary (_e.g._, `ln -s \u003cYOUR_DOWNLOAD_FOLDER\u003e ${REPO_ROOT}/data/scannet`).\n- As detailed in the paper, we use scenes `0000` to `0009` to perform continual semantic adaptation, and a subset of data from the remaining scenes (`0010` to `0706`) to pre-train the segmentation network.\n  - The data from the pre-training scenes are conveniently provided already in the correct format by the ScanNet dataset, as a `scannet_frames_25k.zip` file. You can download this file using the official download script that you should have received after requesting access to the dataset, specifying the `--preprocessed_frames` flag. Once downloaded, the content of the file should be extracted to the subfolder `${REPO_ROOT}/data/scannet/scannet_frames_25k`.\n  - For the scenes used to perform continual semantic adaptation, the full data are required. To obtain them, run the official download script, specifying through the flag `--id` the scene to download (_e.g._, `--id scene0000_00` to download scene `0000`) and including the `--label_map` flag, to download also the label mapping file `scannetv2-labels.combined.tsv` (cf. [here](https://github.com/ScanNet/ScanNet#labels)). The downloaded data should be stored in the subfolder `${REPO_ROOT}/data/scannet/scans`. Next, extract all the sensor data (depth images, color images, poses, intrinsics) using the [SensReader](https://github.com/ScanNet/ScanNet/tree/master/SensReader/python) tool provided by ScanNet, for each of the downloaded scenes from `0000` to `0009`. For instance, for scene `0000`, run\n    ```bash\n    python2 reader.py --filename ${REPO_ROOT}/data/scannet/scans/scene0000_00/scene0000_00.sens --output_path ${REPO_ROOT}/data/scannet/scans/scene0000_00 --export_depth_images --export_color_images --export_poses --export_intrinsics\n    ```\n    To obtain the raw labels (for evaluation purposes) for each of the continual adaptation scenes, also extract the content of the `sceneXXXX_XX_2d-label-filt.zip` file, so that a `${REPO_ROOT}/data/scannet/scans/sceneXXXX_XX/label-filt` folder is created.\n  - Copy the `scannetv2-labels.combined.tsv` file to each scene folder under `${REPO_ROOT}/data/scannet/scans`, as well as to the subfolder `${REPO_ROOT}/data/scannet/scannet_frames_25k`.\n\n  - At the end of the process, the `${REPO_ROOT}/data` folder should contain _at least_ the following data, structured as below:\n\n    ```shell\n    scannet\n      scannet_frames_25k\n        scene0010_00\n          color\n            000000.jpg\n            ...\n            XXXXXX.jpg\n          label\n            000000.png\n            ...\n            XXXXXX.png\n        ...\n        ...\n        scene0706_00\n          ...\n        scannetv2-labels.combined.tsv\n      scans\n        scene0000_00\n          color\n            000000.jpg\n            ...\n            XXXXXX.jpg\n          depth\n            000000.png\n            ...\n            XXXXXX.png\n          label-filt\n            000000.png\n            ...\n            XXXXXX.png\n          pose\n            000000.txt\n            ...\n            XXXXXX.txt\n          intrinsics\n            intriniscs_color.txt\n            intrinsics_depth.txt\n          scannetv2-labels.combined.tsv\n        ...\n        scene0009_00\n          ...\n    ```\n\n    You may define the data subfolders differently by adjusting the `scannet` and `scannet_frames_25k` fields in [`cfg/env/env.yml`](./cfg/env/env.yml). You may also define several config files and set the configuration to use by specifying the `ENV_WORKSTATION_NAME` environmental variable before running the code (_e.g._, `export ENV_WORKSTATION_NAME=\"gpu_machine\"` to use the config in `cfg/env/gpu_machine.yml`).\n\n- Copy the files [`split.npz`](./cfg/dataset/scannet/split.npz) and [`split_cl.npz`](./cfg/dataset/scannet/split_cl.npz) from the `${REPO_ROOT}/cfg/dataset/scannet/` folder to the `${REPO_ROOT}/data/scannet/scannet_frames_25k` folder. These files contain the indices of the samples that define the train/validation splits used in pre-training and to form the replay buffer in continual adaptation, to ensure reproducibility.\n\n#### Dataset pre-processing\n\nAfter organizing the ScanNet files as detailed above, run the following script to pre-process the files:\n\n```bash\nbash run_scripts/preprocess_scannet.sh\n```\n\nAfter pre-processing, the folder structure for each `sceneXXXX_XX` from `scene0000_00` to `scene0009_00` should look as follows:\n\n```shell\n  sceneXXXX_XX\n    color\n      000000.jpg\n      ...\n      XXXXXX.jpg\n    color_scaled\n      000000.jpg\n      ...\n      XXXXXX.jpg\n    depth\n      000000.png\n      ...\n      XXXXXX.png\n    label_40\n      000000.png\n      ...\n      XXXXXX.png\n    label_40_scaled\n      000000.png\n      ...\n      XXXXXX.png\n    label-filt\n      000000.png\n      ...\n      XXXXXX.png\n    pose\n      000000.txt\n      ...\n      XXXXXX.txt\n    intrinsics\n      intriniscs_color.txt\n      intrinsics_depth.txt\n    scannetv2-labels.combined.tsv\n    transforms_test.json\n    transforms_test_scaled_semantics_40_raw.json\n    transforms_train.json\n    transforms_train_scaled_semantics_40_raw.json\n```\n\n## Running experiments\n\nBy default, the data produced when running the code is stored in the `${REPO_ROOT}/experiments` folder. You can modify this by changing the `results` field in [`cfg/env/env.yml`](./cfg/env/env.yml).\n\n### DeepLabv3 pre-training\n\nTo pre-train the DeepLabv3 segmentation network on scenes `0010` to `0706`, run the following script:\n\n```bash\nbash run_scripts/pretrain.sh --exp cfg/exp/pretrain_scannet_25k_deeplabv3.yml\n```\n\nAlternatively, we provide a pre-trained DeepLabv3 [checkpoint](https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/637142/best-epoch143-step175536.ckpt), which you may download to the `${REPO_ROOT}/ckpts` folder.\n\n### One-step experiments\n\nThis Section contains instruction on how to perform one-step adaptation experiments (cf. Sec. 4.4 in the main paper).\n\n#### Fine-tuning\n\nFor fine-tuning, NeRF pseudo-labels should first be generated by running NeRF-only training:\n\n```bash\nbash run_scripts/one_step_nerf_only_train.sh\n```\n\nNext, run\n\n```bash\nbash run_scripts/one_step_finetune_train.sh\n```\n\nto fine-tune DeepLabv3 with the NeRF pseudo-labels. Please make sure the variable `prev_exp_name` defined in the [fine-tuning script](./run_scripts/one_step_finetune_train.sh) matches the variable `name` in the [NeRF-only script](./run_scripts/one_step_nerf_only_train.sh).\n\n#### Joint-training\n\nTo perform one-step joint training, run\n\n```bash\nbash run_scripts/one_step_joint_train.sh\n```\n\n### Multi-step experiments\n\nTo perform multi-step adaptation experiments (cf. Sec. 4.5 in the main paper), run the following commands:\n\n```bash\n# Using training views for replay.\nbash run_scripts/multi_step.sh --exp cfg/exp/multi_step/cl_base.yml\n# Using novel views for \"replay\".\nbash run_scripts/multi_step.sh --exp cfg/exp/multi_step/cl_base_novel_viewpoints.yml\n```\n\n### Logging\n\nBy default, we use [WandB](https://wandb.ai/site) to log our experiments. You can initialize WandB logging by running\n\n```bash\nwandb init -e ${YOUR_WANDB_ENTITY}\n```\nin the terminal. Alternatively, you can disable all logging by defining `export WANDB_MODE=disabled` before launching the experiments.\n\n### Seeding\n\nTo obtain the variances of the results, we run the above experiments multiple times with different seeds by specifying `--seed` in the argument.\n\n## Citation\n\nIf you find our code or paper useful, please cite:\n\n```bibtex\n@inproceedings{Liu2023UnsupervisedContinualSemanticAdaptationNR,\n  author    = {Liu, Zhizheng and Milano, Francesco and Frey, Jonas and Siegwart, Roland and Blum, Hermann and Cadena, Cesar},\n  title     = {Unsupervised Continual Semantic Adaptation through Neural Rendering},\n  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n  year      = {2023}\n}\n```\n\n## Acknowledgements\n\nParts of the NeRF implementation are adapted from [torch-ngp](https://github.com/ashawkey/torch-ngp), [Semantic-NeRF](https://github.com/Harry-Zhi/semantic_nerf/), and [Instant-NGP](https://github.com/NVlabs/instant-ngp).\n\n## Contact\n\nContact [Zhizheng Liu](mailto:liuzhi@student.ethz.ch) and [Francesco Milano](mailto:francesco.milano@mavt.ethz.ch) for questions, comments, and reporting bugs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fethz-asl%2Fucsa_neural_rendering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fethz-asl%2Fucsa_neural_rendering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fethz-asl%2Fucsa_neural_rendering/lists"}