{"id":20516353,"url":"https://github.com/nianticlabs/acezero","last_synced_at":"2025-05-15T09:03:09.920Z","repository":{"id":235202434,"uuid":"788936421","full_name":"nianticlabs/acezero","owner":"nianticlabs","description":"[ECCV 2024 - Oral] ACE0 is a learning-based structure-from-motion approach that estimates camera parameters of sets of images by learning a multi-view consistent, implicit scene representation.","archived":false,"fork":false,"pushed_at":"2025-01-29T14:31:13.000Z","size":20577,"stargazers_count":752,"open_issues_count":1,"forks_count":49,"subscribers_count":59,"default_branch":"main","last_synced_at":"2025-04-14T14:59:58.797Z","etag":null,"topics":["3d-reconstruction","camera-relocalization","computer-vision","eccv","eccv2024","machine-learning","pose-estimation","sfm","structure-from-motion","visual-relocalization"],"latest_commit_sha":null,"homepage":"https://nianticlabs.github.io/acezero/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nianticlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-19T11:33:53.000Z","updated_at":"2025-04-08T08:30:24.000Z","dependencies_parsed_at":"2024-11-22T22:01:54.307Z","dependency_job_id":"ee6ea735-9b12-4ef9-b9b4-4c13b64747a7","html_url":"https://github.com/nianticlabs/acezero","commit_stats":null,"previous_names":["nianticlabs/acezero"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Facezero","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Facezero/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Facezero/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Facezero/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nianticlabs","download_url":"https://codeload.github.com/nianticlabs/acezero/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254310513,"owners_count":22049468,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","camera-relocalization","computer-vision","eccv","eccv2024","machine-learning","pose-estimation","sfm","structure-from-motion","visual-relocalization"],"created_at":"2024-11-15T21:28:23.867Z","updated_at":"2025-05-15T09:03:04.905Z","avatar_url":"https://github.com/nianticlabs.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# ACE0 (ACE Zero)\n\nThis repository contains the code associated to the ACE0 paper:\n\u003e **Scene Coordinate Reconstruction: \nPosing of Image Collections via Incremental Learning of a Relocalizer**\n\u003e \n\u003e [Eric Brachmann](https://ebrach.github.io/), \n\u003e [Jamie Wynn](https://scholar.google.com/citations?user=ASP-uu4AAAAJ\u0026hl=en), \n\u003e [Shuai Chen](https://chenusc11.github.io/), \n\u003e [Tommaso Cavallari](https://scholar.google.it/citations?user=r7osSm0AAAAJ\u0026hl=en), \n\u003e [Áron Monszpart](https://amonszpart.github.io/), \n\u003e [Daniyar Turmukhambetov](https://dantkz.github.io/about/), and \n\u003e [Victor Adrian Prisacariu](https://www.robots.ox.ac.uk/~victor/)\n\u003e \n\u003e ECCV 2024, Oral\n\nFor further information please visit:\n\n- [Project page (with a method overview and videos)](https://nianticlabs.github.io/acezero)\n- [Arxiv](https://arxiv.org/abs/2404.14351)\n\nTable of contents:\n\n- [Installation](#installation)\n- [Usage](#usage)\n    - [Basic Usage](#basic-usage)\n    - [Visualization Capabilities](#visualization-capabilities)\n    - [Advanced Use Cases](#advanced-use-cases)\n      - [Refine Existing Poses](#refine-existing-poses)\n      - [Start From a Partial Reconstruction](#start-from-a-partial-reconstruction)\n      - [Self-Supervised Relocalization](#self-supervised-relocalization)\n      - [Train NeRF models or Gaussian splats](#train-nerf-models-or-gaussian-splats)\n    - [Utility Scripts](#utility-scripts)\n- [Benchmark](#benchmark)\n  - [Nerfacto](#nerfacto)\n  - [Splatfacto](#splatfacto)\n- [Paper Experiments](#paper-experiments)\n  - [7-Scenes](#7-scenes)\n  - [Mip-NeRF 360](#mip-nerf-360)\n  - [Tanks and Temples](#tanks-and-temples)\n- [Frequently Asked Questions](#frequently-asked-questions)\n- [References](#publications)\n\n## Installation\n\nThis code uses PyTorch and has been tested on Ubuntu 20.04 with a V100 Nvidia GPU, although it should reasonably run \nwith other Linux distributions and GPUs as well. Note our [FAQ](#frequently-asked-questions) if you want to run ACE0 on GPUs with less memory. \n\nWe provide a pre-configured [`conda`](https://docs.conda.io/en/latest/) environment containing all required dependencies\nnecessary to run our code.\nYou can re-create and activate the environment with:\n\n```shell\nconda env create -f environment.yml\nconda activate ace0\n```\n\n**All the following commands in this file need to run from the repository root and in the `ace0` environment.**\n\nACE0 represents a scene using an [ACE](https://nianticlabs.github.io/ace/) scene coordinate regression model.\nIn order to register cameras to the scene, it relies on the RANSAC implementation of the DSAC* paper (Brachmann and\nRother, TPAMI 2021), which is written in C++.\nAs such, you need to build and install the C++/Python bindings of those functions.\nYou can do this with:\n\n```shell\ncd dsacstar\npython setup.py install\ncd ..\n```\n\nHaving done the steps above, you are ready to experiment with ACE0!\n\n**Important note:** the first time you run ACE0, the script may ask you to confirm that you are happy to download the ZoeDepth depth estimation code and its pretrained weights from GitHub.\nSee [this link](https://github.com/isl-org/ZoeDepth) for its license and details.\nACE0 uses that model to estimate the depth for the seed images.\nIt can be replaced, please see the [FAQ](#frequently-asked-questions) section below for details.\n\n## Docker\n\nIf you would prefer to run Ace0 in a docker container, you can start it with:\n\n```shell  \ndocker-compose up -d \n```\n\nYou can then shell into the container with the following command: \n\n```shell  \ndocker exec -it acezero /bin/bash\n```\n\nFrom there you can follow the Gaussian Splatting tutorial described at the bottom of the README [here.](#frequently-asked-questions) Make sure to add your images to the volume defined in ```docker-compose.yml```\n\n## Usage\n\nWe explain how to run ACE0 to reconstruct images from scratch, with and without knowledge about the image intrinsics.\nWe also explain how to use ACE0 to refine existing poses, or to initialise reconstruction with a subset of poses.\nFurthermore, we cover the visualization capabilities of ACE0, including export of the reconstruction as a video and as \n3D models.\n\n### Basic Usage\n\nIn the minimal case, you can run ACE0 on a set of images as defined by a glob pattern. \n\n```shell\n# running on a set of images with default parameters\npython ace_zero.py \"/path/to/some/images/*.jpg\" result_folder\n```\n\nNote the quotes around the glob pattern to ensure it is passed to the ACE0 script rather than being expanded by the shell.\n\nIf you want to run ACE0 on a video, you can extract frames from the video and run ACE0 on the extracted frames, see our [Utility Scripts](#utility-scripts).\n\nThe ACE0 script will call ACE training (`train_ace.py`) and camera registration (`register_mapping.py`) in a loop until \nall images have been registered to the scene representation, or there is no change between iterations.\n\nThe result of an ACE0 reconstruction is the `poses_final.txt` in the result folder. \nThese files contain the estimated image poses in the following format:\n```\nfilename qw qx qy qz x y z focal_length confidence\n``` \n`filename` is the image file relative to the repository root.\n`qw qx qy qz ` is the camera rotation as a quaternion, and `x y z` is the camera translation.\nCamera poses are world-to-camera transformations, using the OpenCV camera convention.\n`focal_length` is the focal length estimated by ACE0 or set externally (see below).\n`confidence` is the reliability of an estimate. \nIf the confidence is less than 1000, it should be treated as unreliable and possibly ignored.\n\nThe pose files can be used e.g. to train a Nerfacto or Splatfacto model, using our benchmarking scripts, see [Benchmarking](#benchmark).\nOur benchmarking scripts also allow you to only convert our pose files to the format required by Nerfstudio, without running the benchmark itself.\n\n\u003cdetails\u003e\n\u003csummary\u003eOther content of the result folder explained.\u003c/summary\u003e\n\nThe result folder will contain files such as the following:\n\n- `iterationX.pt`: The ACE scene model (the MLP network) at iteration X. Output of `train_ace.py` in iteration X.\n- `iterationX.txt`: Training statistics of the ACE model at iteration X, e.g. loss values, pose statistics, etc. See `ace_trainer.py`. Output of `train_ace.py` in iteration X.\n- `poses_iterationX_preliminary.txt`: Poses of cameras after the mapping iteration but before relocalization. Contains poses refined by the MLP, rather than poses re-estimated by RANSAC. Output of `train_ace.py` in iteration X. \n- `poses_iterationX.txt`: Final poses of iteration X, after relocalization, i.e. re-estimated by RANSAC. Output of `register_mapping.py` in iteration X.\n- `poses_final.txt`: The final poses of the images in the scene. Corresponds to the poses of the last relocalisation iteration, i.e. the output of the last `register_mapping.py` call.\n- `pc_final.ply`: An ACE0 point cloud of the scene, for visualisation or initialisation of Gaussian splats. This output is optional and triggered using the `--export_point_cloud True` option of `ace_zero.py`.\n\u003c/details\u003e\n\n#### Setting Calibration Parameters\n\nUsing default parameters, ACE0 will estimate the focal length of the images, starting from a heuristic value (70% of the image diagonal.)\nIf you have a better estimate of the focal length, you can provide it as an initialisation parameter.\n\n```shell\n# running ACE0 with an initial guess for the focal length\npython ace_zero.py \"/path/to/some/images/*.jpg\" result_folder --use_external_focal_length \u003cfocal_length\u003e\n```\n\nUsing the call above, ACE0 will refine the focal length throughout the reconstruction process.\nIf you are confident that your focal length value is correct, you can disable focal length refinement.\n\n```shell\n# running ACE0 with a fixed focal length\npython ace_zero.py \"/path/to/some/images/*.jpg\" result_folder --use_external_focal_length \u003cfocal_length\u003e --refine_calibration False\n```\n\n**Note:** The current implementation of ACE0 supports only a single focal length value shared by all images. \nACE0 currently also does assume that the principal point is at the image center, and pixels are square and unskewed.\nChanging these assumptions should be possible, but requires some implementation effort.\n\n### Visualization Capabilities\n\nACE0 can visualize the reconstruction process as a video. \n\n```shell\n# running ACE0 with visualisation enabled\npython ace_zero.py \"/path/to/some/images/*.jpg\" result_folder --render_visualization True\n```\n\nWith visualisation enabled, ACE0 will render individual frames in a subfolder `renderings` and call `ffmpeg` at the end.\nThe visualisation will be saved as a video in the results folder, named `reconstruction.mp4`.\n\n\u003cdetails\u003e\n\u003csummary\u003eOther content of the renderings folder explained.\u003c/summary\u003e\n\n* `frame_N.png`: The Nth frame of the video.\n* `iterationX_mapping.pkl`: The visualisation buffer of the mapping call in iteration X. It stores the 3D point cloud of the scene, the last rendering camera for a smooth transition, and the last frame index.\n* `iterationX_register.pkl`: The visualisation buffer of the relocalization call in iteration X.\n\u003c/details\u003e\n\n**Note that this will slow down the reconstruction considerably.**\nAlternatively, you can run without visualisation enabled and export the final reconstruction as a 3D model, see [Utility Scripts](#utility-scripts).\n\n### Advanced Use Cases\n\nYou can combine the ACE0 meta script with custom calls to `train_ace.py` and `register_mapping.py` to cater to more advanced use cases.\n\n* `train_ace.py`: Trains an ACE model on a set of images with corresponding poses.\n* `register_mapping.py`: Estimates poses of images in a scene given an ACE model.\n* `ace_zero.py`: Can start from an existing ACE model.\n\nYou are free to switch image sets between the calls to these functions.\nWe provide some examples of advanced use cases that also cover some of the experiments in our paper.\n\n#### Refine Existing Poses\n\nIf you have an initial guess of all image poses, you can use ACE to refine them quickly.\nWe combine a single ACE mapping call with pose refinement enabled, and a single relocalization call.\n\n```shell\n# running ACE mapping with pose refinement enabled\npython train_ace.py \"/path/to/some/images/*.jpg\" result_folder/ace_network.pt --pose_files \"/path/to/some/images/*.txt\" --pose_refinement mlp --pose_refinement_wait 5000 --use_external_focal_length \u003cfocal_length\u003e --refine_calibration False\n\n# re-estimate poses of all images\npython register_mapping.py \"/path/to/some/images/*.jpg\" result_folder/ace_network.pt --use_external_focal_length \u003cfocal_length\u003e --session ace_network\n```\n\nIn this example, ACE takes the existing poses in the [7-Scenes](#7-scenes) format as input: one text file per image with the camera-to-world pose stored as a 4x4 matrix.\nThe option `--pose_refinement mlp` enables pose refinement using a refinement network.\nThe option `--pose_refinement_wait 5000` freezes poses for the first 5000 iterations which increases the stability if you are mapping from scratch with pose refinement.\n\nAfter calling `register_mapping.py`, the result folder will contain the refined poses in `poses_ace_network.txt`.\n\nNote that the example above assumes a known, fixed focal length. If you let ACE refine the calibration, you need to pass the refined focal length of `train_ace.py` to `register_mapping.py`.\nPlease see `scripts/reconstruct_7scenes_warmstart.sh` for a complete example where we refine KinectFusion poses with ACE.\n\n#### Start From a Partial Reconstruction\n\nIf you have pose estimates for subsets of images, you can use ACE0 to complete the reconstruction.\nFirst, you call ACE mapping on the subset of images with poses which results in an ACE scene model.\nYou pass this model to ACE0, which will then register the remaining images to the scene.\n\n```shell\n# running ACE mapping on a subset of images wit poses\npython train_ace.py \"/images/with/poses/*.jpg\" result_folder/iteration0_seed0.pt --pose_files \"/poses/of/images/*.txt\" --use_external_focal_length \u003cfocal_length\u003e --refine_calibration False\n\n# running ACE0 with the ACE model as a seed, and the complete set of images\npython ace_zero.py \"/all/images/*.jpg\" result_folder --seed_network result_folder/iteration0_seed0.pt --use_external_focal_length ${focal_length} --refine_calibration False\n```\n\nACE0 will store the final poses in `poses_final.txt` in the result folder, containing poses of all images.\nNote that the example above assumes a known, fixed focal length.\nYou can also let ACE or ACE0 estimate or refine the focal length, but you need to take care of passing the correct focal length between the calls.\n\nPlease see `scripts/reconstruct_t2_training_videos_warmstart.sh` for a complete example where we reconstruct the Tanks and Temples training scenes starting from a partial reconstruction by COLMAP. More information about this example in [Tanks and Temples](#tanks-and-temples).\n\n#### Self-Supervised Relocalization\n\nYou can use ACE0 to map a set of images, and call `register_mapping.py` on a different set of images to relocalize them.\nHere, ACE0 would run on the set of mapping images, while `register_mapping.py` would run on the set of query images.\n\n```shell\n# running ACE0 on the mapping images\npython ace_zero.py \"/path/to/mapping/images/*.jpg\" result_folder --use_external_focal_length \u003cfocal_length\u003e --refine_calibration False\n\n# running relocalization on the query images\npython register_mapping.py \"/path/to/query/images/*.jpg\" result_folder/iterationX.pt --use_external_focal_length \u003cfocal_length\u003e --session query\n```\n\nYou need to point `register_mapping.py` to the ACE model from the last mapping iteration (e.g. `iterationX.pt`). \nThe relocalization results will be stored in `poses_query.txt`.\nNote that ACE0 reconstructions are only approximately metric. \nIf you compare the query poses to ground truth, you need to fit a similarity transform first.\nWe provide a script for doing that.\n\n```shell\npython eval_poses.py result_folder/poses_query.txt \"/path/to/ground/truth/poses/*.txt\"\n```\n\nMore information about the evaluation script can be found under [Utility Scripts](#utility-scripts).\n\n#### Train NeRF models or Gaussian splats\n\nSee [Benchmarking](#benchmark) for instructions on how to use Nerfstudio on top of ACE0.\n\n### Utility Scripts\n\n#### Video to Dataset\n\nWe provide a script for extracting frames from MP4 videos via ffmpeg.\n\n```shell\npython datasets/video_to_dataset.py datasets\n```\n\nThe script looks for all MP4 files in the target folder (here `datasets`) and extracts frames into a subfolder `datasets/video_\u003cmp4_file_name\u003e` for each video.\n\n#### Export 3D Scene as Point Cloud\n\nWe provide a script for exporting ACE point clouds from a network and a pose file.\n\n```shell\npython export_point_cloud.py point_cloud_out.txt --network /path/to/ace_network.pt --pose_file /path/to/poses_final.txt\n````\n\nThe script can either write out TXT of PLY files, decided by the file extension of the output file you specify. \nIf the output file has a .txt extension, the script will write the point cloud into a text file in the format `x y z r g b` per line for each point.\nIf the output file has a .ply extension, the script will write the point cloud into a binary PLY file.\nBoth formats can be imported into most 3D software, e.g. Meshlab, CloudCompare, etc.\nThe PLY format is understood by Nerfstudio for initialisation of Gaussian splats.\nNote, you can also point the script to an existing visualization buffer, `result_folder/renderings/iterationX_mapping.pkl`, which already contains the point cloud so it does not have to be re-generated.\n\nPoint clouds can be exported either using OpenGL or OpenCV coordinate conventions. Nerfstudio expects OpenCV coordinates.\nThe script can extract sparse or dense point clouds. The sparse point clouds have more filters applied and look cleaner.\nThe dense point clouds tend to work better for Gaussian splatting if you have a lot of images (2000+) as they cover more of the background. \n\n#### Export Cameras as Mesh\n\nWe provide a script for exporting an ACE pose file to PLY showing the cameras.\n\n```shell\npython export_cameras.py /path/to/ace/pose_file.txt /path/to/output.ply\n```\n\nThe script will color-code the cameras by their confidence value. \nThe PLY format can be imported into most 3D software, e.g. Meshlab, CloudCompare, etc.\n\n#### Evaluate Poses Against (Pseudo) Ground Truth\n\nWe provide a script that measures the pose error of a set of estimated poses against a set of ground truth poses.\n\n```shell\npython eval_poses.py /path/to/ace/pose_file.txt \"/path/to/ground/truth/poses/*.txt\"\n```\n\nThe ground truth poses are given as a glob pattern, where each file contains the pose of a single image as a 4x4 camera-to-world transformation (e.g. as provided by the 7-Scenes dataset).\nCorrespondence between ACE estimates and ground truth files will be established via alphabetical order of the image filenames.\n\nBy default, the script will calculate the percentage of poses below 5cm and 5 degrees error, as well as median rotation and translation errors.\nSince ACE0 poses are only approximately metric and in an arbitrary reference frame, the script will fit a similarity transform between estimates and ground truth before calculating error.\nThis behaviour can be disabled with the appropriate command line flags.\n\n## Benchmark\n\nWe evaluate the ACE0 pose quality using novel view synthesis via Nerfstudio.\n\n**Note:** All paper results were produced with Nerfstudio v0.3.4. Since then, we updated this repository to support newer version of Nerfstudio.\nWe verified that benchmark results did not change significantly when updating to Nerfstudio v1.1.4. \nHowever, if you observe benchmarking inconsistencies w.r.t. the paper, we advise to first down-grade to Nerfstudio v0.3.4 and checkout our code using the \"eccv_2024_checkpoint\" git tag.\n\n### Nerfacto\n\nIn our paper, we benchmark the ACE0 reconstruction by training a Nerfacto model and measuring PSNR on a dataset-specific training/test split of images.\nTo setup the benchmark, follow the instructions in the [Benchmark README](benchmarks/README.md).\n\nNote that the benchmark lives in its own conda environment, so you have to change environments between reconstruction and benchmarking.\n\nThe benchmark takes an ACE0 pose file and fits a Nerfacto model. Optionally, you can also use our benchmarking scripts to generate the input files for Nerfstudio without running the benchmark, see the `--no_run_nerfstudio` flag.\n\nIf you do run the benchmark, it will apply a 1/8 split of the images by default to calculate PSNR.\nThe scripts we provide for our [Paper Experiments](#paper-experiments) do optionally run the benchmark on each dataset using the correct split.\n\nSince the benchmarking results are stored in a nested structure, we provide a script to extract the PSNR values:\n\n```shell\n# show the benchmark results of all scenes as sub-folders in the provided top-level folder\npython scripts/show_benchmark_results.py /path/to/top/level/results/folder\n```\n\nThe script assumes a folder structure where each scene is a sub-folder in a dataset-specific top-level folder.\nE.g. `benchmark/7scenes` contains sub-folders `chess`, `fire`, `heads`, etc.\n\nAfter running benchmarking on a reconstruction, you can load the NeRF model using Nerfstudio's viewer, render videos etc.\n\n```shell\nns-viewer --load-config /path/to/nerf/config.yaml\n```\n\n### Splatfacto\n\nTraining Gaussian splats with Splatfacto is very similar to training a Nerfacto model.\nSplatfacto additionally needs a point cloud to initialise the splats, which you can export using one of our [utility scripts](#export-3d-scene-as-point-cloud) or by running ACE0 with `--export_point_cloud True`.\nOur benchmarking scripts will look for a file `pc_final.ply` to pass to Nerfstudio.\nNote that Nerfstudio will also proceed if no `pc_final.ply` file is found, but the splats will be initialised uniformly which can result in very poor quality.\nYou can check for the following warning in the Nerfstudio log to see if the point cloud was expected but missing:\n```\nWarning: load_3D_points set to true but no point cloud found.\n```\n\nOtherwise, you just have to run our benchmarking scripts with `--method splatfacto`, see the [Benchmark README](benchmarks/README.md) for details.\nThe script `show_benchmark_results.py` mentioned in the previous section also has the `--method splatfacto` option to show benchmarking metrics of Splatfacto models.\n\nNote that all our scripts for the [Paper Experiments](#paper-experiments) support both Nerfacto and Splatfacto benchmarking.\nThe method can just be toggled at the top of each script. \nWe recommend to take a look and use these scripts as blueprints for your own experiments.\n\n## Paper Experiments\n\nWe provide scripts to run the main experiments of the paper.\nWe also provide pre-computed results for all these experiments, along with the corresponding visualizations, in the respective sections below.\n\n### 7-Scenes\n\nSetup the dataset.\n\n```shell\n# setup the 7-Scenes dataset in the datasets folder\ncd datasets\n# download and unpack the dataset\npython setup_7scenes.py\n# back to root directory\ncd ..\n```\nThe script can optionally convert the dataset to the ACE format, download alternative pseudo ground truth poses, calibrate depth maps, etc.\nHowever, it is not required for the ACE0 experiments.\n\n(Optional for the benchmark) Create a benchmarking train/test split for the 7-Scenes dataset, see the [Benchmark README](benchmarks/README.md) for details.\n\n```shell\npython scripts/create_splits_7scenes.py datasets/7scenes split_files\n```\n\nReconstruct each scene (corresponding to \"ACE0\" in Table 1, left).\n\n```shell\nbash scripts/reconstruct_7scenes.sh\n```\n\nBy default, the script will run with benchmarking enabled (make sure you set it up, see [Nerfacto Benchmark](#benchmark)), using Nerfacto and with visualisation disabled. \nFlip the appropriate flags in the script to change this behaviour, e.g. to train Gaussian splats instead of NeRF models. \nThe ACE0 reconstruction files will be stored in `reconstructions/7scenes` while the benchmarking results will be stored in `benchmark/7scenes`.\nTo show the benchmarking results, call:\n\n```shell\npython scripts/show_benchmark_results.py benchmark/7scenes\n```\n\nTo refine KinectFusion poses using ACE (corresponding to \"KF+ACE0\" in Table 1, left), run:\n\n```shell\nbash scripts/reconstruct_7scenes_warmstart.sh\n# show the benchmark results\npython scripts/show_benchmark_results.py benchmark/7scenes_warmstart\n```\n\nFind pre-computed poses and reconstruction videos for 7-Scenes [here](https://storage.googleapis.com/niantic-lon-static/research/acezero/results_ace0_7scenes.tar.gz). \nThese results are from a different run of ACE0 than the one we used for the paper results, but PSNR values are very close (\u0026plusmn; 0.1dB PSNR on average).\n\nFor some experiments in the paper (see right side of Table 1), we run ACE0 and baselines on a subset of images for each scene.\nWe provide the lists of images, together with how they have been split for the view synthesis benchmark here: [200 images per scene](https://storage.googleapis.com/niantic-lon-static/research/acezero/splits_7s_200frames.tar.gz) and [50 images per scene](https://storage.googleapis.com/niantic-lon-static/research/acezero/splits_7s_50frames.tar.gz).\n\n### Mip-NeRF 360\n\nSetup the dataset.\n\n```shell\n# setup the Mip-NeRF 360 dataset in the datasets folder\ncd datasets\n# download and unpack the dataset\npython setup_mip360.py\n# back to root directory\ncd ..\n```\nThe script can optionally convert the COLMAP ground truth to the ACE format, but it is not required for the ACE0 experiments.\n\n(Optional for the benchmark) Create a benchmarking train/test split for the Mip-NeRF 360 dataset, see the [Benchmark README](benchmarks/README.md) for details. \nThis uses a slightly different 1/8 split than the default benchmark split.\n\n```shell\npython scripts/create_splits_mip360.py datasets/mip360 split_files\n```\n\nReconstruct each scene (corresponding to \"ACE0\" in Table 2 (b)).\n\n```shell\nbash scripts/reconstruct_mip360.sh\n```\n\nBy default, the script will run with benchmarking enabled (make sure you set it up, see [Nerfacto Benchmark](#benchmark)), using Nerfacto and with visualisation disabled. \nFlip the appropriate flags in the script to change this behaviour, e.g. to train Gaussian splats instead of NeRF models.  \nThe ACE0 reconstruction files will be stored in `reconstructions/mip360` while the benchmarking results will be stored in `benchmark/mip360`.\nTo show the benchmarking results, call:\n\n```shell\npython scripts/show_benchmark_results.py benchmark/mip360\n```\n\nFind pre-computed poses and reconstruction videos for the Mip-NerF 360 dataset [here](https://storage.googleapis.com/niantic-lon-static/research/acezero/results_ace0_mip360.tar.gz). \nThese results are from a different run of ACE0 than the one we used for the paper results, but PSNR values are very close (\u0026plusmn; 0.1dB PSNR on average).\n\n### Tanks and Temples\n\nYou have to manually [download the dataset](https://www.tanksandtemples.org/download/).\nOur dataset script assumes you downloaded the group archives into `datasets/t2` without unpacking them:\n```\ndatasets/t2/training.zip\ndatasets/t2/training_videos.zip\ndatasets/t2/intermediate.zip\ndatasets/t2/intermediate_videos.zip\ndatasets/t2/advanced.zip\ndatasets/t2/advanced_videos.zip\n```\n\nSetup the dataset.\n\n```shell\n# setup the T\u0026T dataset in the datasets folder\ncd datasets\n# unpack the dataset\npython setup_t2.py\n# back to root directory\ncd ..\n```\n\nOptionally, the script can download and setup COLMAP ground truth poses, and convert them to the ACE format.\nThis is required for the ACE0 experiments which reconstruct the dataset videos starting from a sparse COLMAP reconstruction.\nCall the script with `--with-colmap`.\nThis will create an additional folder `t2_colmap` in the datasets folder where each scene folder not only has the image \nfiles, but also corresponding `*_pose.txt` files with COLMAP poses as 4x4, camera-to-world transformations.\nAlso, per scene, a single `focal_length.txt` file is created with the COLMAP focal length estimate.\n\nWe provide scripts for Tanks and Temples separated by scene group, i.e. training, intermediate, and advanced.\nThe following explanations are for the training group, but the scripts for the intermediate and advanced groups are similar.\n\nReconstruct each scene from a few hundred images (corresponding to \"ACE0\" in Table 3, left).\n\n```shell\nbash scripts/reconstruct_t2_training.sh\n```\n\nBy default, the script will run with benchmarking enabled (make sure you set it up, see [Nerfacto Benchmark](#benchmark)), using Nerfacto and with visualisation disabled. \nFlip the appropriate flags in the script to change this behaviour, e.g. to train Gaussian splats instead of NeRF models. \nThe ACE0 reconstruction files will be stored in `reconstructions/t2_training` while the benchmarking results will be stored in `benchmark/t2_training`.\nTo show the benchmarking results, call:\n\n```shell\npython scripts/show_benchmark_results.py benchmark/t2_training\n```\n\nNote that no benchmarking split files need to be generated for Tanks and Temples. \nThe benchmark will apply a default 1/8 split.\n\nTo reconstruct the full videos of each scene (corresponding to \"ACE0\" in Table 3, right), call:\n```shell\nbash scripts/reconstruct_t2_training_videos.sh\n# show benchmarking results\npython scripts/show_benchmark_results.py benchmark/t2_training_videos\n```\n\nTo reconstruct the full videos of each scene starting from a COLMAP reconstruction (corresponding to \"Sparse COLMAP + ACE0\" in Table 3, left), call:\n```shell\nbash scripts/reconstruct_t2_training_videos_warmstart.sh\n# show benchmarking results\npython scripts/show_benchmark_results.py benchmark/t2_training_videos_warmstart\n```\n\nNote that the last experiment assumes that you set up the dataset with `--with-colmap`.\nThe code will first call ACE mapping on the images with COLMAP poses to create an initial scene model.\nThis model is then passed to ACE0 which will use it as a seed for the full video reconstruction.\nIn this example, we trust the focal length estimate of COLMAP and keep it fixed throughout the reconstruction.\n\nFind pre-computed poses and reconstruction videos for Tanks and Temples here: [Training scenes](https://storage.googleapis.com/niantic-lon-static/research/acezero/results_ace0_t2_training.tar.gz), [Intermediate scenes](https://storage.googleapis.com/niantic-lon-static/research/acezero/results_ace0_t2_intermediate.tar.gz), [Advanced scenes](https://storage.googleapis.com/niantic-lon-static/research/acezero/results_ace0_t2_advanced.tar.gz). \nThese results are from a different run of ACE0 than the one we used for the paper results, but PSNR values are very close (\u0026plusmn; 0.3dB PSNR on average).\n\n## Frequently Asked Questions\n\n**Q: I want Gaussian splats from my images. What do I need to do?**\n\nPrepare ACE0 as explained in the beginning of this document: Create the ACE0 environment, compile the DSAC* bindings, and [setup Nerfstudio](benchmarks/README.md).\nThen, run the following commands on your image set:\n\n```\n# activate our conda environment\nconda activate ace0\n\n# run ACE0 reconstruction with point cloud export\npython ace_zero.py \"/path/to/some/images/*.jpg\" result_folder --export_point_cloud True\n\n# switch to the Nerfstudio conda environment\nconda activate nerfstudio\n\n# convert the ACE0 output to a Nerfstudio compatible format and run Splatfacto training (also runs evaluation but it's fast)\npython -m benchmarks.benchmark_poses --pose_file result_folder/poses_final.txt --output_dir benchmark_folder --images_glob_pattern \"/path/to/some/images/*.jpg\" --method splatfacto \n\n# view the Gaussian splats\nns-viewer --load-config benchmark_folder/nerf_data/nerf_for_eval/splatfacto/run/config.yaml\n```\n\n**Q: I run out of GPU memory during the ACE0 reconstruction. What can I do?**\n\n**A:** All experiments in the paper were performed with 16GB of GPU memory (e.g. NVIDIA V100/T4) and the default settings should work with such a GPU.\nThe bulk of the memory is used by the ACE training buffer (up to ~8GB). \nYou can run ACE0 with the flag `--training_buffer_cpu True` to keep the training buffer on the CPU at the expense of reconstruction speed.\nWith that option, ACE0 should require ~1GB of GPU memory.\n\n**Q: I have an image collection with various images sizes, aspect ratios and intrinsics. Can I use ACE0?**\n\n**A:** No. ACE0 assumes that all images share their intrinsics, particularly the focal length.\nThis is a limitation of the current implementation, rather than the method. \nSupporting images with varying intrinics should work, but would require some implementation effort, particularly in `refine_calibration.py`. \n\n**Q: Does ACE0 estimate intrinsics other than the focal length?**\n\n**A:** No. ACE0 assumes that the principal point is at the image center, and pixels are square and unskewed.\nThe focal length, shared by all images, is the only intrinsic parameter estimated and/or refined by ACE0.\n\n**Q: I have images from a complex camera model. e.g. with severe image distortion. Can I use ACE0?**\n\n**A:** No. The scene coordinate regression network might be able to remove some distortion, but presumably not much.\nThe reprojection loss of ACE and the RANSAC pose estimator assume a pinhole camera model. \nThese parts would need to implement a camera distortion model. \nIf the distortion parameters are known, we would recommend to undistort the images before passing them to ACE0.\n\n**Q: How can I run ACE0 with depth other than ZoeDepth estimates?**\n\n**A:** If you have pre-calculated depth maps, you can call `ace_zero.py` with `--depth_files \"/path/to/depths/*.png\"`.\nIn this case, ACE0 will use the provided depth maps for the seed images instead of estimating depth.\nOtherwise, the functions `get_depth_model()` and `estimate_depth()` in `dataset_io.py` can be adapted to use a depth estimator other than ZoeDepth.\nNote that we found the impact of the depth estimation model to be rather small in our experiments.\n\n**Q: Is ACE0 able to reconstruct from a small set of sparse views?**\n\n**A:** It can work but this scenario is challenging for ACE0. \nWe expect other methods, and even COLMAP, to work much better in this case.\nACE0 relies on images having sufficient visual overlap, particularly when registering new images to the reconstruction.\nYou can lower the registration threshold when running `ace_zero.py` via `--registration_confidence` setting it to 300 or 100 - but at some point ACE0 will get unstable.\nACE0 shines if you have dense coverage of a scene, and reconstruct it from many images in reasonable time.\n\n## Publications\n\nIf you use ACE0 or parts of its code in your own work, please cite:\n\n```\n@inproceedings{brachmann2024acezero,\n    title={Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer},\n    author={Brachmann, Eric and Wynn, Jamie and Chen, Shuai and Cavallari, Tommaso and Monszpart, {\\'{A}}ron and Turmukhambetov, Daniyar and Prisacariu, Victor Adrian},\n    booktitle={ECCV},\n    year={2024},\n}\n```\n\nThis code builds on the ACE relocalizer and uses the DSAC* pose estimator. Please consider citing:\n\n```\n@inproceedings{brachmann2023ace,\n    title={Accelerated Coordinate Encoding: Learning to Relocalize in Minutes using RGB and Poses},\n    author={Brachmann, Eric and Cavallari, Tommaso and Prisacariu, Victor Adrian},\n    booktitle={CVPR},\n    year={2023},\n}\n\n@article{brachmann2021dsacstar,\n  title={Visual Camera Re-Localization from {RGB} and {RGB-D} Images Using {DSAC}},\n  author={Brachmann, Eric and Rother, Carsten},\n  journal={TPAMI},\n  year={2021}\n}\n```\n\nACE0 estimates depth of seed images using ZoeDepth. Please consider citing:\n\n```\n@article{bhat2023zoedepth,\n  title={Zoe{D}epth: Zero-shot transfer by combining relative and metric depth},\n  author={Bhat, Shariq Farooq and Birkl, Reiner and Wofk, Diana and Wonka, Peter and M{\\\"u}ller, Matthias},\n  journal={arXiv},\n  year={2023}\n}\n```\n\nThis repository relies on Nerfstudio for benchmarking. \nPlease consider citing according to [their docs](https://docs.nerf.studio/#citation).\n\n\n## License\n\nCopyright © Niantic, Inc. 2024. Patent Pending.\nAll rights reserved.\nPlease see the [license file](LICENSE) for terms.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Facezero","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnianticlabs%2Facezero","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Facezero/lists"}