{"id":18573563,"url":"https://github.com/prs-eth/resdepth","last_synced_at":"2025-04-10T07:32:19.209Z","repository":{"id":57431218,"uuid":"376057187","full_name":"prs-eth/ResDepth","owner":"prs-eth","description":"[ISPRS Journal of Photogrammetry and Remote Sensing, 2022] ResDepth: A Deep Residual Prior For 3D Reconstruction From High-resolution Satellite Images","archived":false,"fork":false,"pushed_at":"2022-07-06T08:32:34.000Z","size":90829,"stargazers_count":72,"open_issues_count":0,"forks_count":17,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-01-29T17:10:20.183Z","etag":null,"topics":["3d-reconstruction","dsm","satellite-imagery","scene-refinement","u-net"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prs-eth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-11T14:50:34.000Z","updated_at":"2024-01-01T07:39:01.000Z","dependencies_parsed_at":"2022-08-26T09:11:22.988Z","dependency_job_id":null,"html_url":"https://github.com/prs-eth/ResDepth","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FResDepth","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FResDepth/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FResDepth/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FResDepth/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prs-eth","download_url":"https://codeload.github.com/prs-eth/ResDepth/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223430255,"owners_count":17143625,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","dsm","satellite-imagery","scene-refinement","u-net"],"created_at":"2024-11-06T23:10:53.022Z","updated_at":"2024-11-06T23:10:53.623Z","avatar_url":"https://github.com/prs-eth.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ResDepth: A Deep Residual Prior For 3D Reconstruction From High-resolution Satellite Images\n\n![ResDepth](docs/teaser.png?raw=true)\n\nThis repository provides the code to train and evaluate ResDepth, an efficient and easy-to-use neural architecture for \nlearned DSM refinement from satellite imagery. It represents the official implementation of the paper:\n\n### [ResDepth: A Deep Residual Prior For 3D Reconstruction From High-resolution Satellite Images](https://doi.org/10.1016/j.isprsjprs.2021.11.009)\n*[Corinne Stucker](https://prs.igp.ethz.ch/content/specialinterest/baug/institute-igp/photogrammetry-and-remote-sensing/en/group/people/person-detail.html?persid=179102), \n[Konrad Schindler](https://prs.igp.ethz.ch/group/people/person-detail.schindler.html)*\n\n\u003e **Abstract:** *Modern optical satellite sensors enable high-resolution stereo reconstruction from space. But the challenging imaging conditions when observing the Earth from space push stereo matching to its limits. In practice, the resulting digital surface models (DSMs) are fairly noisy and often do not attain the accuracy needed for high-resolution applications such as 3D city modeling. Arguably, stereo correspondence based on low-level image similarity is insufficient and should be complemented with a-priori knowledge about the expected surface geometry beyond basic local smoothness. To that end, we introduce ResDepth, a convolutional neural network that learns such an expressive geometric prior from example data. ResDepth refines an initial, raw stereo DSM while conditioning the refinement on the images. I.e., it acts as a smart, learned post-processing filter and can seamlessly complement any stereo matching pipeline. In a series of experiments, we find that the proposed method consistently improves stereo DSMs both quantitatively and qualitatively. We show that the prior encoded in the network weights captures meaningful geometric characteristics of urban design, which also generalize across different districts and even from one city to another. Moreover, we demonstrate that, by training on a variety of stereo pairs, ResDepth can acquire a sufficient degree of invariance against variations in imaging conditions and acquisition geometry.*\n\n\n## Requirements\n\nThis code has been developed and tested on Ubuntu 18.04 with Python 3.7, PyTorch 1.9, and GDAL 2.2.3. \nIt may work for other setups but has not been tested thoroughly.\n\nOn Ubuntu 18.04, `gdal` can be installed with `apt-get`:\n```shell\nsudo apt update\nsudo apt install libgdal-dev gdal-bin\n```\n\n\n## Setup\n\nBefore proceeding, make sure that GDAL is installed and set up correctly.\n\nTo create a [Python virtual environment](https://docs.python.org/3/tutorial/venv.html) and install the required \ndependencies, please run:\n```bash\ngit clone https://github.com/stuckerc/ResDepth.git\ncd ResDepth\npython3 -m venv tmp/resdepth\nsource tmp/resdepth/bin/activate\n(resdepth) $ pip install --upgrade pip setuptools wheel\n(resdepth) $ pip install -r requirements.txt\n```\nin your working directory. Next, use the following command to install GDAL in the previously created virtual environment:\n```bash\n(resdepth) $ pip install --global-option=build_ext --global-option=\"-I/usr/include/gdal\" GDAL==`gdal-config --version`\n```\n\n\n## Quick Start\n\nResDepth follows a residual learning strategy, i.e., it is trained to refine an imperfect input DSM by regressing a \nper-pixel correction to the height, using both the DSM and ortho-rectified panchromatic (stereo) images as input. \n\n\n### Data Preparation\n\nWe assume that the initial surface reconstruction has been generated with existing multi-view stereo matching and/or \ndepth map fusion techniques. Furthermore, we assume that the images have already been ortho-rectified with the help of \nthe initial surface estimate. Please note that this repository does not provide any functionality to perform data \npre-processing (initial surface reconstruction, ortho-rectification) nor image pair selection.\n\nWhen preparing your data as input for ResDepth, make sure to meet the following requirements:\n* The initial reconstruction and the reference model are parametrized as digital surface models (DSMs, i.e., height \nfields in raster format).\n* The initial DSM and ground truth DSM are reprojected to the same horizontal coordinate system (for instance, the \nlocal UTM zone). The pixel values of the initial DSM and the ground truth DSM must refer to the same vertical coordinate \nsystem or otherwise be vertically aligned.\n* ResDepth accepts ground truth DSMs that exhibit NoData cells. However, NoData cells in the initial DSM are prohibited. \nHence, you might have to interpolate the initial DSM to replace any NoData cells.  \n* The initial DSM is used to ortho-rectify the panchromatic satellite views.\n* The initial DSM, the ground truth DSM, and the ortho-images are stored in GeoTIFF raster format. Furthermore, all \nrasters must be co-registered, cropped to the same spatial (rectangular) extent, and exhibit the same spatial resolution \n(i.e., pixel-aligned rasters).  \n\n\u003e  **Note**: Throughout our experiments, we use DSMs with a grid spacing of 0.25 m. By construction, ResDepth is \n\u003egeneric and can be trained to refine any DSM. However, if the spatial resolution deviates from our setting, it might be \n\u003erequired to adapt the tile size ```tile_size``` of the DSM patches and/or the depth ```depth``` of the U-Net  \n\u003e(number of downsampling and upsampling layers). \n\n\nFor evaluation, ResDepth accepts the following additional rasters as input:\n* *Ground truth mask*: A raster in GeoTIFF format. A pixel value of 1 indicates a pixel with valid reference height, \nwhereas a pixel value of 0 indicates a pixel with invalid reference height. This raster is used to mask out areas with \nevident temporal differences between the initial and ground truth DSM due to construction activities.\n* *Building mask*: A raster in GeoTIFF format. A pixel with a value of 1 indicates a building pixel, and a pixel with a \nvalue of 0 is a terrain pixel. If a building mask is provided, the quantitative errors are computed for buildings and \nterrain separately. Before calculating the object-specific metrics, the building mask is dilated by two pixels to avoid \naliasing at vertical walls.\n* *Water mask*: A raster in GeoTIFF format. A pixel with a value of 1 indicates a water pixel, and a pixel with a value \nof 0 is a non-water pixel. Water bodies are excluded from the evaluation only if a building mask is provided.  \n* *Forest mask*: A raster in GeoTIFF format. A pixel with a value of 1 indicates a forest pixel, and a pixel with a value \nof 0 is a non-forest pixel. Densely forested areas are excluded from the evaluation only if a building mask is provided.\n\n\u003e **Note**: All rasters (DSMs, ortho-images, mask rasters) must be co-registered and cropped to the same spatial \n\u003e(rectangular) extent. Furthermore, the spatial resolution must be the same.\n\n\n### Data Structure\n\nWe do not expect a particular structure of how the data is stored. The path to every initial DSM, to the corresponding \nground truth DSM, and possibly to raster masks must be listed in the configuration file (see below). The ortho-images \nand the definition of the image pairs must be provided as text files.\n\n\n#### Image List\n\nPrepare a text file ```imagelist.txt``` that lists the absolute paths to the pre-computed ortho-rectified satellite \nimages (one file path per line):\n```shell\npath/to/ortho-image1.tif\npath/to/ortho-image2.tif\npath/to/ortho-image3.tif\npath/to/ortho-image4.tif\npath/to/ortho-image5.tif\npath/to/ortho-image6.tif\npath/to/ortho-image7.tif\n...\n```\n\nIt is possible to use the same image list for training, validation, and testing.\n\n\u003e **Note**: The filename of every image listed in the image list has to be unique (irrespective of the absolute file \n\u003epath due to the definition of the image pair list, see below).\n\n\n#### Image Pair List\n\nPrepare a text file ```pairlist.txt``` that comprises a comma-separated list of filenames, where every line defines one \nimage pair. If multiple image pairs are specified, each pair needs to be of equal length (i.e., the same number of  \nimages per image pair).\n\n***Example image pair list for ResDepth-stereo***: The following image pair list defines a single stereo pair composed \nof the images ```ortho-image1.tif``` and ```ortho-image2.tif```. The absolute paths to the ortho-images will be derived \nby matching the image filenames listed in ```pairlist.txt``` and ```imagelist.txt```.\n```shell\northo-image1.tif, ortho-image2.tif\n```\n\n***Example image pair list for ResDepth-stereo, generalized across viewpoints***: To train a ResDepth-stereo network \nthat generalizes across variations in acquisition geometry and the images' radiometry, one simply has to provide \nmultiple stereo pairs in the image pair list, for example:\n```shell\northo-image1.tif, ortho-image2.tif\northo-image2.tif, ortho-image5.tif\northo-image4.tif, ortho-image5.tif\n```\n\nIn this example, ResDepth-stereo will be trained using the three image pairs (```ortho-image1.tif```, ```ortho-image2.tif```), \n(```ortho-image2.tif```, ```ortho-image5.tif```), and (```ortho-image4.tif```, ```ortho-image5.tif```).\n\n\n***Example image pair list for ResDepth-mono***: The following example shows an image pair list to train ResDepth-mono  \nusing the single image ```ortho-image1.tif``` as guidance:\n```shell\northo-image1.tif, \n```\n\n***Example image pair list for ResDepth-0***: The network variant ResDepth-0 does not leverage any satellite views. \nTherefore, to train or evaluate ResDepth-0, neither the image list ```imagelist.txt``` nor the image pair list \n```pairlist.txt``` have to be provided.\n\n\n## Training\n\nTo train ResDepth, run the script ```train.py``` with a JSON configuration file as the unique argument:\n```shell\n(resdepth) $ python train.py config.json\n```\n\nThe configuration file ```config.json``` specifies the input data and the output directory (see below for details). \nFurthermore, it configures the model architecture, hyperparameters, and training settings. All parameters and their \ndefault settings are described in ```./lib/config.py```.\n\nTo monitor and visualize the training process, you can start a tensorboard session with:\n```shell\n(resdepth) $ tensorboard --logdir \u003ctboard_log_dir\u003e\n```\n\n\n## Evaluation\n\nTo evaluate the ResDepth prior, run the script ```test.py``` with a JSON configuration file as the unique argument:\n```shell\n(resdepth) $ python test.py config_test.json\n```\n\nThe configuration file ```config_test.json``` specifies the input data, the model architecture of ResDepth and its \nweights, and the output directory (see below for details).\n\nThe script ```test.py``` uses a tiling-based strategy to refine the given input DSM. First, it cuts the DSM into \na regular grid of overlapping tiles, where the tile size amount to ```tile_size``` and the stride to ```0.5*tile_size```. \nThe DSM patches are then individually refined. Lastly, the refined DSM patches are merged to output a single refined \nDSM raster.\nIf multiple images (ResDepth-mono) or image pairs (ResDepth-stereo) are provided in the image pair list, the same \ninitial DSM is refined multiple times by using every image (pair) once for guidance. Finally, the error metrics are \nreported both over all predictions and for every prediction separately.\n\n\n## Configuration File: Training\n\nThe following example shows the bare minimum JSON configuration file ```config.json``` to train ResDepth. It consists of \ntwo objects ```datasets``` and ```output``` with mandatory and optional name-value pairs that need to be completed by \nthe user. \n```json\n{\n  \"datasets\": [\n    {\n      \"name\": \"my_dataset\",\n      \"raster_gt\": \"path/to/ground_truth_DSM.tif\",\n      \"raster_in\": \"path/to/initial_DSM.tif\",\n      \"path_image_list\": \"path/to/imagelist.txt\",\n      \"path_pairlist_training\": \"path/to/pairlist_training.txt\",\n      \"path_pairlist_validation\": \"path/to/pairlist_validation.txt\",\n      \"area_type\": \"train+val\",\n      \"allocation_strategy\": \"5-crossval_vertical\",\n      \"test_stripe\": 1,\n      \"crossval_training\": false,\n      \"n_training_samples\": 20000\n    }\n    ],\n  \"output\": {\n    \"suffix\": \"\",\n    \"output_directory\": \"path/to/output_directory\",\n    \"tboard_log_dir\": \"path/to/tboard_log_directory\"\n  }\n}\n```\n\n### Input Data\n\nThe ```datasets``` object defines a list of objects with mandatory and optional key-value pairs. Every object in the list \ndescribes a dataset, i.e., a (rectangular) geographic region for which an initial DSM, a corresponding ground truth DSM, \nand ortho-images are available. For training, ResDepth expects at least one training dataset and one validation dataset \n(i.e., the list is composed of two objects). Alternatively, ResDepth accepts one (or multiple) dataset(s) split into \nmutually exclusive stripes for training, validation, and testing (i.e., the list is composed of at least one object). \nEvery dataset (object in the list) has the following mandatory key-value pairs:\n\n* ```raster_in```: str, path to the initial DSM.\n* ```raster_gt```: str, path to the corresponding ground truth DSM.\n* ```path_image_list```: str, path to the image list ```imagelist.txt```.\n* ```path_pairlist_training```: str, path to the image pair list ```pairlist_training.txt```, which defines the image \npairs used for training.\n* ```path_pairlist_validation```: str, path to the image pair list ```pairlist_validation.txt```, which defines the image \npairs used for validation (the image pairs used for training and validation can be equal or different from each other).\n* ```area_type```: str, choose among ['train', 'val', 'train+val']. The parameter ```area_type``` specifies whether the \ndataset/region is used for training ('train'), for validation ('val'), or for both ('train+val').\n\nThe keys ```path_image_list```, ```path_pairlist_training```, and ```path_pairlist_validation``` are not required for \nResDepth-0.\n\nAdditionally, the user can specify the following optional key-value pairs:\n* ```name```: str, optional dataset identifier.\n* ```allocation_strategy```: str, choose among ['5-crossval_vertical', '5-crossval_horizontal', 'entire']. If the parameter \n```allocation_strategy``` is set to 'entire', the entire dataset/region will be used for either training or validation \n(depending on the value of ```area_type```). Set the parameter ```allocation_strategy``` to '5-crossval_vertical' to \nsplit the geographic region into five equally large and mutually exclusive vertical stripes (north-south oriented). \nSimilarly, set ```allocation_strategy``` to '5-crossval_horizontal' to split the geographic region into five equally \nlarge and mutually exclusive horizontal stripes (west-east oriented).\n* ```test_stripe```: int, choose among [0, 1, 2, 3, 4]. The parameter ```test_stripe``` is relevant if the value of \n```allocation_strategy``` is in ['5-crossval_vertical', '5-crossval_horizontal']. It determines which of the five stripes \nis reserved for testing. By definition, the validation stripe is located to the right/bottom (east/south) of the test \nstripe (cyclic order). The remaining stripes are used for training.\n* ```crossval_training```: bool, flag to activate (True) or deactivate (False) cross-validation. To perform cross-validation, \nset ```crossval_training``` to True, ```area_type``` to 'train+val', and choose among ['5-crossval_vertical', '5-crossval_horizontal'] \nto specify ```allocation_strategy```. The stripe with index ```test_stripe``` is used as validation region and all other \nstripes as training regions.\n* ```n_training_samples```: int, number of DSM patches randomly sampled from the training region. This parameter  \nhas to be specified only if ```area_type``` is in ['train', 'train+val'].\n\n\u003e **Warning**: At runtime, all rasters (DSMs and ortho-images listed in ```imagelist.txt```) are loaded to memory.\n\n\n### Output Settings\n\nThe name of the results folder consists of the code execution day and time and an optional suffix ```YYYY-MM-DD_HH-MM_${suffix}```. \nThe ```output``` object consists of the following key-value pairs:  \n* ```suffix```: str, optional suffix appended to the name of the results folder.\n* ```output_directory```: str, output directory. All log files will be stored in the directory \n```${output_directory}/YYYY-MM-DD_HH-MM_${suffix}``` and the PyTorch checkpoints in \n```${output_directory}/YYYY-MM-DD_HH-MM_${suffix}/checkpoints``` (saved intermediate and final model weights). \n* ```tboard_log_dir```: str, output directory. All tensorboard checkpoint files (event files) will be stored in the \ndirectory ```${tboard_log_dir}/YYYY-MM-DD_HH-MM_${suffix}```.\n\n\nThe results directory is structured as follows:\n```shell\n${output_directory}/YYYY-MM-DD_HH-MM_${suffix}/\n├── checkpoints\n│   ├── Model_after_${save_model_rate}_epochs.pth\n│   ├── Model_after_${2*save_model_rate}_epochs.pth\n│   ├── ...\n│   └── Model_best.pth\n├── config.json\n├── config.json.orig\n├── DSM_normalization_parameters.p\n├── Image_normalization_parameters.p\n├── model_config.json\n├── run.log\n└── training.log\n```\n\nIt stores the following files:\n* ```config.json.orig```: JSON file, configuration file used to invoke ```train.py```.\n* ```config.json```: validated and augmented JSON file, the combination of ```config.json.orig``` and the default settings \nspecified in ```./lib/config.py```.\n* ```model_config.json```: JSON file, stores the model architecture of ResDepth.\n* ```DSM_normalization_parameters.p```: pickle file, stores the DSM normalization parameters.\n* ```Image_normalization_parameters.p```: pickle file, stores the satellite image normalization parameters.\n* ```run.log```: log file, console output.\n* ```training.log```: log file, training progress.\n* ```checkpoints/Model_after_${save_model_rate}_epochs.pth```: model weights after ```save_model_rate``` training epochs.\n* ```checkpoints/Model_best.pth```: model weights of the training epoch with lowest validation loss.\n\n\n### Switching between ResDepth-stereo, ResDepth-mono, and ResDepth-0\n\nThe content of the image pair list ```pairlist.txt``` determines whether ResDepth is trained using stereo information \nor a single image as guidance. To train ResDepth-0, neither the image list ```imagelist.txt``` nor the image pair list \n```pairlist.txt``` have to be provided.\n\nIn addition to modifying the image pair list ```pairlist.txt```, one must also adjust the network architecture \naccordingly. For ResDepth-stereo, add the following settings to your JSON configuration file:\n```json\n  \"model\": {\n    \"input_channels\": \"geom-stereo\"\n  },\n  \"stereopair_settings\": {\n    \"use_all_stereo_pairs\": true,\n    \"permute_images_within_pair\": true\n  }\n```\n\nThese are the default settings to train ResDepth-stereo, which generalizes across variations in acquisition geometry and \nimaging conditions. Ideally, the image pair list specified by ```path_pairlist_training``` comprises more than one image \npair. Set ```permute_images_within_pair``` to False if the goal is to train a ResDepth-stereo prior tailored to the \nspecific image characteristics and acquisition geometry of a ***single*** image pair.\n\nFor ResDepth-mono, please specify:\n```json\n  \"model\": {\n    \"input_channels\": \"geom-mono\"\n  }\n```\n\nSimilarly, for ResDepth-0, specify:\n```json\n  \"model\": {\n    \"input_channels\": \"geom\"\n  }\n```\n\nLastly, we also provide the option to train a U-Net variant that directly regresses a DSM from an ortho-rectified stereo\npair[^1]:\n```json\n  \"model\": {\n    \"input_channels\": \"stereo\",\n    \"outer_skip\": false\n  }\n```\n\n\n### Changing the Default Model and Training Settings (Hyperparameters, Optimizer, Learning Rate Scheduler)\n\nWe provide a detailed description of all parameters and their default settings in ```./lib/config.py```. Most likely, the \nparameters ```depth``` and ```tile_size``` have to be fine-tuned if the spatial resolution of the DSMs deviates from 0.25 m \n(our setting). Add the parameters that you wish to modify to your JSON configuration file to overwrite the respective \ndefault setting.\n\n\n### Templates\n\nWe provide several template files in the directory ```./configs/``` to train ResDepth-0, ResDepth-mono, and ResDepth-stereo \non a single dataset. Furthermore, we provide a template for the generalized ResDepth-stereo variant.\n\n\n\n## Configuration File: Evaluation\n\nIn the following, we show an example JSON configuration file ```config_test.json``` to evaluate ResDepth:\n```json\n{\n  \"datasets\": [\n    {\n      \"name\": \"my_dataset\",\n      \"raster_gt\": \"path/to/ground_truth_DSM.tif\",\n      \"raster_in\": \"path/to/initial_DSM.tif\",\n      \"path_image_list\": \"path/to/imagelist.txt\",\n      \"path_pairlist\": \"path/to/pairlist_test.txt\",\n      \"mask_ground_truth\": \"path/to/ground_truth_mask.tif\",\n      \"mask_building\": \"path/to/building_mask.tif\",\n      \"mask_water\": \"path/to/water_mask.tif\",\n      \"mask_forest\": \"path/to/forest_mask.tif\",\n      \"area_type\": \"test\",\n      \"allocation_strategy\": \"5-crossval_vertical\",\n      \"test_stripe\": 1,\n      \"crossval_training\": false\n    }\n  ],\n  \"model\": {\n    \"weights\": \"${output_directory}/YYYY-MM-DD_HH-MM_${suffix}/checkpoints/Model_best.pth\",\n    \"architecture\": \"${output_directory}/YYYY-MM-DD_HH-MM_${suffix}/model_config.json\",\n    \"normalization_geom\": \"${output_directory}/YYYY-MM-DD_HH-MM_${suffix}/DSM_normalization_parameters.p\",\n    \"normalization_image\": \"${output_directory}/YYYY-MM-DD_HH-MM_${suffix}/Image_normalization_parameters.p\"\n  },\n  \"general\": {\n    \"tile_size\": 256\n  },\n  \"output\": {\n    \"directory\": \"path/to/results/folder\"\n  }\n}\n```\n\nThe key-value pairs of the ```datasets``` object are equal to those used for the training configuration file. \nAdditionally, the user can specify the file paths of ground truth masks, building masks, water masks, and forest masks \n(see Section 'Data Preparation' above). To evaluate cross-validation, set ```crossval_training``` to True and \n```area_type``` to 'val'. Furthermore, use the same values for ```allocation_strategy``` and ```test_stripe``` as \nduring training.\n\nThe ```model``` object specifies the model weights, the model architecture, and the parameters used for data normalization. \nThe directory ```${output_directory}/YYYY-MM-DD_HH-MM_${suffix}``` corresponds to the output folder of the training \nscript ```train.py```. Note that ```normalization_image``` is required for ResDepth-mono and ResDepth-stereo but not \nfor ResDepth-0. \n\nFinally, it is essential to set the same tile size ```tile_size``` as used during training.\n\n\n## Pretrained Models\n\nWe provide the checkpoints of two ResDepth-stereo models used to test geographical generalization between Berlin and \nZurich (see Section *Geographical Generalization Across Cities* in the paper). Furthermore, we provide the checkpoint \nof our ResDepth-stereo multi-city model (see Section *Multi-city Model* in the paper).\n\nTo download these models, please run:\n```shell\n(resdepth) $ bash ./scripts/download_pretrained_models.sh\n```\n\nAdditionally, we provide all the models used in the ablation studies (see Section *Influence of Image Guidance* in \nthe paper).\n\nTo download these models, please run:\n```shell\n(resdepth) $ bash ./scripts/download_pretrained_models_ablations.sh\n```\n\nAll the models will be downloaded and extracted to ```./logs/pretrained_models/``` and ```./logs/pretrained_models_ablations/```.\n\n\n## Demo\n\nDue to the commercial nature of VHR imagery, we cannot share our complete datasets. In this demo, we thus provide two \nvariants of DSM refinement using a single DSM patch of 256\u0026times;256 pixels only (64\u0026times;64 m in world coordinates).\n\n### Example Dataset\n\nTo download the demo, please run:\n```shell\n(resdepth) $ bash ./scripts/download_demo.sh\n```\n\nThe data, configuration files, and pretrained models will be downloaded and extracted to ```./demo/```.\n\nThe data is stored in ```./demo/data/```:\n```shell\n./demo/data/\n├── dsm\n│   ├── DSM_Zurich_ground_truth.tif\n│   └── DSM_Zurich_initial.tif\n├── image_selection\n│   ├── imagelist.txt\n│   ├── pairlist_simple.txt\n│   └── pairlist_generalized.txt\n├── mask\n│   └── Zurich_building_mask.tif\n└── satellite_views\n    ├── 15MAR17102414-P1BS-502980289080_01_P002.tif\n    ├── 17OCT07103158-P1BS-501653123070_01_P002.tif\n    ├── 18APR10105411-P1BS-502091706050_01_P002.tif\n    ├── 18JAN29104120-P1BS-502980288020_01_P005.tif\n    └── 18MAR24105605-P1BS-501687882040_02_P006.tif\n```\n\nWe provide an initial DSM patch ```DSM_Zurich_initial.tif```, the corresponding ground truth DSM patch \n```DSM_Zurich_ground_truth.tif```, and a building mask ```Zurich_building_mask.tif```. The example DSM is located \nin the test stripe of the region ZUR1 in Zurich. Furthermore, we provide five ortho-images stored in the subdirectory \n```./demo/data/satellite_views/```. The subdirectory ```./demo/data/image_selection/``` comprises the image list \n```imagelist.txt``` and two pair lists, where ```pairlist_simple.txt``` defines a single stereo pair and \n```pairlist_generalized.txt``` two stereo pairs.\n\n\n### Pretrained Models\n\nThe pretrained model weights are stored in ```./demo/models/```:\n```shell\n./demo/models/\n├── ResDepth-stereo\n│   ├── checkpoints\n│   │    └── Model_best.pth\n│   ├── DSM_normalization_parameters.p\n│   ├── Image_normalization_parameters.p\n│   └── model_config.json\n└── ResDepth-stereo_generalized\n    ├── checkpoints\n    │    └── Model_best.pth\n    ├── DSM_normalization_parameters.p\n    ├── Image_normalization_parameters.p\n    └── model_config.json    \n```\n\nThe subdirectory ```./demo/models/ResDepth-stereo/``` contains the pretrained weights of a ResDepth-stereo prior. \nWe used the training stripes of the region ZUR1 in Zurich and the single stereo pair listed in \n```./demo/data/image_selection/pairlist_simple.txt``` for training.\n\nSimilarly, the subdirectory ```./demo/models/ResDepth-stereo_generalized/``` specifies the pretrained weights of a \nResDepth-stereo prior that has learned to generalize to unseen viewing directions, lighting conditions, and urban styles. \nFor training, we sampled training data from all training stripes in ZUR1, ZUR2, and ZUR3 and leveraged multiple stereo \npairs that are different from the ones listed in ```./demo/data/image_selection/pairlist_generalized.txt```. \n\n\n### Running the Demo\n\nUse the available test configuration ```./demo/configs/config_simple.json``` to refine the initial DSM using the simple \nResDepth-stereo prior (tailored to the specific image characteristics and acquisition geometry of the training image pair):\n```shell\n(resdepth) $ python test.py ./demo/configs/config_simple.json\n```\n\nTo refine the initial DSM using the generalized ResDepth-stereo prior, use the test configuration \n```./demo/configs/config_generalized.json``` and run:\n```shell\n(resdepth) $ python test.py ./demo/configs/config_generalized.json\n```\n\n### Results\n\nThe refined DSMs will be stored in ```./demo/results/ResDepth-stereo/``` and \u003cnobr\u003e```./demo/results/ResDepth-stereo_generalized/```\u003c/nobr\u003e.\nFor visualization, you can use Python visualization packages or off-the-shelf DSM visualization software such as \n[Quick Terrain Reader](https://appliedimagery.com/download/) or [planlauf/TERRAIN](https://planlaufterrain.com/).\n\nAs reference, ```./demo/results_expected/``` stores the expected results.\n\nPreview of the results:\n![Demo](docs/DSMs_all.gif)\n\n## Contact\n\nIf you run into any problems or have questions, please contact [Corinne Stucker](mailto:corinne.stucker@geod.baug.ethz.ch).\n\n\n## Citations\n\nIf you find this code or work helpful, please cite:\n```\n@article{stucker2022resdepth,\ntitle = {{ResDepth}: A deep residual prior for 3D reconstruction from high-resolution satellite images},\nauthor = {Stucker, Corinne and Schindler, Konrad},\njournal = {ISPRS Journal of Photogrammetry and Remote Sensing},\nvolume = {183},\npages = {560--580},\nyear = {2022}\n}\n```\n\n\n[^1]: C. Stucker, K. Schindler, ResDepth: Learned residual stereo reconstruction, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020, pp. 707-716.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprs-eth%2Fresdepth","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprs-eth%2Fresdepth","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprs-eth%2Fresdepth/lists"}