{"id":20464503,"url":"https://github.com/bytedance/particle-sfm","last_synced_at":"2025-04-12T06:09:33.861Z","repository":{"id":46754502,"uuid":"515423970","full_name":"bytedance/particle-sfm","owner":"bytedance","description":"ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild. ECCV 2022.","archived":false,"fork":false,"pushed_at":"2025-02-06T09:07:52.000Z","size":2755,"stargazers_count":317,"open_issues_count":5,"forks_count":26,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-04-12T06:09:22.326Z","etag":null,"topics":["dynamic-scenes","motion-segmentation","research","structure-from-motion","video-correspondence"],"latest_commit_sha":null,"homepage":"http://b1ueber2y.me/projects/ParticleSfM/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bytedance.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-19T03:41:50.000Z","updated_at":"2025-04-10T12:32:35.000Z","dependencies_parsed_at":"2024-04-09T03:27:53.729Z","dependency_job_id":"0cae9fc6-376f-4e3f-9c0f-ac08dbae1c73","html_url":"https://github.com/bytedance/particle-sfm","commit_stats":{"total_commits":22,"total_committers":7,"mean_commits":3.142857142857143,"dds":0.7272727272727273,"last_synced_commit":"c6b7848d4a4a5d0783cc48024cd0140705fdd2c7"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2Fparticle-sfm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2Fparticle-sfm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2Fparticle-sfm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2Fparticle-sfm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bytedance","download_url":"https://codeload.github.com/bytedance/particle-sfm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525139,"owners_count":21118619,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dynamic-scenes","motion-segmentation","research","structure-from-motion","video-correspondence"],"created_at":"2024-11-15T13:15:25.951Z","updated_at":"2025-04-12T06:09:33.826Z","avatar_url":"https://github.com/bytedance.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ParticleSfM\n\u003cp align=\"center\"\u003e\n\u003ch3\u003e\u003ca href=\"https://arxiv.org/abs/2207.09137\"\u003ePaper\u003c/a\u003e | \u003ca href=\"https://youtu.be/KfFCdOFE6tA\"\u003eVideo\u003c/a\u003e | \u003ca href=\"http://b1ueber2y.me/projects/ParticleSfM/\"\u003eProject Page\u003c/a\u003e\u003c/h3\u003e\n  \u003cdiv align=\"center\"\u003e\u003c/div\u003e\n\u003c/p\u003e\n\nCode release for our ECCV 2022 paper \"ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild.\" by [Wang Zhao](https://thuzhaowang.github.io/), [Shaohui Liu](http://www.b1ueber2y.me/), [Hengkai Guo](https://guohengkai.github.io/), [Wenping Wang](https://engineering.tamu.edu/cse/profiles/Wang-Wenping.html) and [Yong-Jin Liu](https://cg.cs.tsinghua.edu.cn/people/~Yongjin/Yongjin.htm).\n\n**[Introduction]** ParticleSfM is an offline structure-from-motion system for videos (image sequences). Inspired by [Particle video](http://rvsn.csail.mit.edu/pv/), our method connects pairwise optical flows and optimizes dense point trajectories as long-range video correpondences, which are used in a customized global structure-from-motion framework with similarity averaging and global bundle adjustment. In particular, for dynamic scenes, the acquired dense point trajectories can be fed into a specially designed trajectory-based motion segmentation module to select static point tracks, enabling the system to produce reliable camera trajectories on in-the-wild sequences with complex foreground motion. \n\n![Teaser](misc/media/teaser.png \"teaser\")\n\nContact Wang Zhao (thuzhaowang@163.com), Shaohui Liu (b1ueber2y@gmail.com) and Hengkai Guo (guohengkai@bytedance.com) for questions, comments and reporting bugs.\n\nIf you are interested in potential collaboration or internship at ByteDance, please feel free to contact Hengkai Guo (guohengkai@bytedance.com).\n\n## Update by 2025.02.05\nWe support [GLOMAP](https://github.com/colmap/glomap) in our pipeline, which achieves more accurate results on 13 sequences of the [Sintel dataset](http://sintel.is.tue.mpg.de/):\n\n| Method | ATE (m) | RPE trans (m) | RPE rot (deg) | SfM runtime (min) | #Frames |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| Global SfM - Ours w/ gcolmap(Theia) | 0.104 | 0.054 | 0.414 | **3.35** | 45.6 | \n| Global SfM - Ours w/ GLOMAP | **0.057** | **0.031** | **0.201** | 6.97 | 45.6 | \n\nTest it by simply changing the `sfm_type` to `global_glomap`:\n```\npython run_particlesfm.py --image_dir /path/to/the/image/folder/ \\\n                          --output_dir /path/to/output/workspace/ \\\n                          --sfm_type global_glomap  # \"global_theia\" for the paper version\n```\n\n## Installation\n1. Install dependencies:\n\n* Ceres \u003e= 2.0.0\n\nFor using gcolmap (Theia) as in the original ParticleSfM paper:\n* COLMAP \u003c= 3.8 [[Guide](./misc/doc/colmap.md)]\n* Theia SfM (customized version) [[Guide](./misc/doc/theia.md)]\n\nAlternatively, if you want to use our latest [GLOMAP](https://github.com/colmap/glomap) support:\n* GLOMAP\n\n2. Set up Python environment with Conda:\n```\nconda env create -f particlesfm_env.yaml\nconda activate particlesfm \n```\n\n3. Build our point trajectory optimizer and global structure-from-motion module. \n- The path to your customized python executable should be set [here](https://github.com/bytedance/particle-sfm/blob/main/scripts/build_all.sh#L5). \n- (Optional) Add another gcc search path (e.g. gcc 9) [here](https://github.com/bytedance/particle-sfm/blob/b079361510f5241477d11e11082140b45b5d1c27/sfm/gmapper/cmake/FindOpenMP.cmake#L6) to compile gmapper correctly.\n```\ngit submodule update --init --recursive\nsudo apt-get install libhdf5-dev\nbash scripts/build_all.sh\n```\n\n4. Download pretrained models for [MiDaS](https://github.com/isl-org/MiDaS), [RAFT](https://github.com/princeton-vl/RAFT) and our motion segmentation module ([download script](./scripts/download_model_pretrained.sh)).\n```\nbash scripts/download_all_models.sh\n```\n\n## Quickstart Demo\n1. Download two example in-the-wild sequences [[Google Drive](https://drive.google.com/file/d/1VRBKFUFWJiG0or8VejM3fBjmsy74Aumc/view?usp=sharing)] from [DAVIS](https://davischallenge.org/): \u003cem\u003esnowboard\u003c/em\u003e and \u003cem\u003etrain\u003c/em\u003e:\n```\nbash ./scripts/download_examples.sh\n```\n2. Example command to run the reconstruction (e.g. on \u003cem\u003esnowboard\u003c/em\u003e):\n```\npython run_particlesfm.py --image_dir ./example/snowboard/images --output_dir ./outputs/snowboard/\n```\nYou can also alternatively use the command for the workspace with the `images` folder inside below. This option will write all the output in the same workspace.\n```\npython run_particlesfm.py --workspace_dir ./example/snowboard/\n```\n3. Visualize the outputs with either the COLMAP GUI or your customized visualizer. We also provide a visualization script:\n```\npython -m pip install open3d pycolmap\npython visualize.py --input_dir ./outputs/snowboard/sfm/model --visualize\n```\nThe results below are expected (left: \u003cem\u003esnowboard\u003c/em\u003e; right: \u003cem\u003etrain\u003c/em\u003e):\n![Example](misc/media/example.png \"example\")\n\n## Usage\n1. Given an image sequence, put all the images in the same folder. The sorted ordering of the names should be consistent with its ordering in the sequence.\n\n2. Use the following command to run our whole pipeline:\n    ```\n    python run_particlesfm.py --image_dir /path/to/the/image/folder/ \\\n                              --output_dir /path/to/output/workspace/\n    ```\n    This will sequentially run `optical flow -\u003e point trajectory -\u003e motion seg -\u003e sfm`. The final results will be saved inside the image data folder with COLMAP output format.\n\n    If you have the prior information that the scene to be reconstructed is fully static, you can skip the motion segmentation module with `--assume_static`. Conversely, if you only want to run the motion segmentation, attach `--skip_sfm` to the command.\n\n    \u003e To speed up\n    \u003e - Use \"--skip_path_consistency\" to skip the path consistency optimization of point trajectories\n    \u003e - Try higher down-sampling ratio for optimizing point trajectories: e.g. \"--sample_ratio 4\"\n\n3. Visualize the outputs using COLMAP GUI (Download the COLMAP Binary and import the data sequence directory) or just your customized visualizer.\n\n## Evaluation\n### MPI Sintel dataset\n1. Download the [Sintel dataset](http://sintel.is.tue.mpg.de/). You also need to download the [groundtruth camera motion data](http://sintel.is.tue.mpg.de/depth) and the [generated motion mask](http://files.is.tue.mpg.de/jwulff/mrflow/sintel_rigiditymaps.zip) to evaluate the pose and motion segmentation.\n2. Prepare the sequences:\n```\npython scripts/prepare_sintel.py --src_dir /path/to/your/sintel/training/final/ \\\n                                 --tgt_dir /path/to/the/data/root/dir/want/to/save/\n```\n3. Run ParticleSfM reconstructions:\n```\npython run_particlesfm.py --root_dir /path/to/the/data/root/dir/\n```\n\n4. To evaluate the camera poses:\n```\npython ./evaluation_evo/eval_sintel.py --input_dir /path/to/the/data/root/dir/ \\\n                                       --gt_dir /path/to/the/sintel/training/data/camdata_left/ \\\n                                       --dataset sintel\n```\nThis will output a txt file with detailed error metrics. Also, the camera trajectories are plotted and saved inside each data sequence folder.\n\n5. To evaluate the motion segmentation:\n```\npython ./motion_seg/eval_traj_iou.py --root_dir /path/to/the/data/root/dir/ \\\n                                     --gt_dir /path/to/the/sintel/rigidity/\n```\n\n### ScanNet dataset\n1. Download the test split of [ScanNet dataset](http://www.scan-net.org/), extract the data from .sens data using the [official script](https://github.com/ScanNet/ScanNet/tree/master/SensReader).\n\n2. Prepare the sequences:\n```\npython scripts/prepare_scannet.py --src_dir /path/to/your/scannet/test/scans_test/ \\ \n                                  --tgt_dir /path/to/the/data/root/dir/want/to/save/\n```\nWe use the first 20 sequences of test split and perform downsampling with stride 3, resize the image to 640x480.\n\n3. Run ParticleSfM reconstructions:\n```\npython run_particlesfm.py --root_dir /path/to/the/data/root/dir/ \\\n                          --flow_check_thres 3.0 --assume_static\n```\n\n4. To evaluate the camera poses: \n```\npython ./evaluation_evo/eval_scannet.py --input_dir /path/to/the/data/root/dir/ \\\n                                        --gt_dir /path/to/the/scannet/test/scans_test/ \\\n                                        --dataset scannet\n```\n\nThis will output a txt file with detailed error metrics. Also, the camera trajectories are plotted and saved inside each data sequence folder.\n\n## Training\n1. Download the Flyingthings3D dataset from the [official website](https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html). We need the RGB images (finalpass) and optical flow data.\n\n2. Download the generated binary motion labels from [here](http://thoth.inrialpes.fr/research/mpnet/labels.tar.gz) or [GoogleDrive](https://drive.google.com/file/d/1Bgg7mM6kMtXdIZt11Q-ZF9NWBu4L7qnW/view?usp=sharing), and unpack this archive into the root directory of the FlyingThings3D dataset. We thank the authors of [MPNet](http://thoth.inrialpes.fr/research/mpnet/) for kindly sharing it.\n\n3. Prepare the training data:\n```\npython ./scripts/prepare_flyingthings3d.py --src_dir /path/to/your/flyingthings3d/data/root/\n```\n\n4. To launch the training, configure your config file inside `./motion_seg/configs/` and then run:\n```\ncd ./motion_seg/\npython train_seq.py ./configs/your-config-file\ncd ..\n```\n\n## Applications\n* Motion Control for Video Generation: [MotionCtrl](https://wzhouxiff.github.io/projects/MotionCtrl/), [CamCo](https://ir1d.github.io/CamCo/)\n* Motion Evaluation for Video Generation: [AnimateAnything](https://yu-shaonian.github.io/Animate_Anything/), [AC3D](https://snap-research.github.io/ac3d/)\n* Kinematic Control Annotation: [EgoVid-5M](https://egovid.github.io/)\n\n## Citation\n```\n@inproceedings{zhao2022particlesfm,\n      author    = {Zhao, Wang and Liu, Shaohui and Guo, Hengkai and Wang, Wenping and Liu, Yong-Jin},\n      title     = {ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild},\n      booktitle = {European conference on computer vision (ECCV)},\n      year      = {2022}\n  }\n```\n\n\n## More related projects\n* [DynaSLAM](https://github.com/BertaBescos/DynaSLAM). Bescos et al. DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes. IROS 2018.\n* [TrianFlow](https://github.com/B1ueber2y/TrianFlow). Zhao et al. Towards Better Generalization: Joint Depth-Pose Learning without PoseNet. CVPR 2020.\n* [VOLDOR](https://github.com/htkseason/VOLDOR). Min et al. VOLDOR-SLAM: For the times when feature-based or direct methods are not good enough. ICRA 2021.\n* [DROID-SLAM](https://github.com/princeton-vl/DROID-SLAM). Teed et al. DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. NeurIPS 2021.\n\n## Acknowledgements\nThis project could not be possible without the great open-source works from [COLMAP](https://colmap.github.io/), [Theia](http://theia-sfm.org/), [hloc](https://github.com/cvg/Hierarchical-Localization), [RAFT](https://github.com/princeton-vl/RAFT), [MiDaS](https://github.com/isl-org/MiDaS) and [OANet](https://github.com/zjhthu/OANet). We sincerely thank them all.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytedance%2Fparticle-sfm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbytedance%2Fparticle-sfm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytedance%2Fparticle-sfm/lists"}