{"id":32561416,"url":"https://github.com/bytedance-seed/traceanything","last_synced_at":"2025-10-29T01:50:28.851Z","repository":{"id":318958825,"uuid":"1076623849","full_name":"ByteDance-Seed/TraceAnything","owner":"ByteDance-Seed","description":"Trace Anything: Representing Any Video in 4D via Trajectory Fields","archived":false,"fork":false,"pushed_at":"2025-10-16T02:05:01.000Z","size":32453,"stargazers_count":113,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-16T20:13:59.561Z","etag":null,"topics":["3d-reconstruction","4d-reconstruction","computer-vision"],"latest_commit_sha":null,"homepage":"https://trace-anything.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ByteDance-Seed.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-15T05:59:37.000Z","updated_at":"2025-10-16T20:11:48.000Z","dependencies_parsed_at":"2025-10-17T12:25:49.073Z","dependency_job_id":null,"html_url":"https://github.com/ByteDance-Seed/TraceAnything","commit_stats":null,"previous_names":["bytedance-seed/traceanything"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ByteDance-Seed/TraceAnything","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByteDance-Seed%2FTraceAnything","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByteDance-Seed%2FTraceAnything/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByteDance-Seed%2FTraceAnything/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByteDance-Seed%2FTraceAnything/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ByteDance-Seed","download_url":"https://codeload.github.com/ByteDance-Seed/TraceAnything/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByteDance-Seed%2FTraceAnything/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281544227,"owners_count":26519553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-28T02:00:06.022Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","4d-reconstruction","computer-vision"],"created_at":"2025-10-29T01:50:05.550Z","updated_at":"2025-10-29T01:50:28.842Z","avatar_url":"https://github.com/ByteDance-Seed.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n# Trace Anything: Representing Any Video in 4D via Trajectory Fields\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://trace-anything.github.io/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Project%20Page-222222?logo=googlechrome\u0026logoColor=white\" alt=\"Project Page\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://arxiv.org/abs/2510.13802\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/arXiv-b31b1b?logo=arxiv\u0026logoColor=white\" alt=\"arXiv\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://youtu.be/PFmwkomheSA?si=3WBNSGNIdqlCTl-1\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/YouTube-ea3323?logo=youtube\u0026logoColor=white\" alt=\"YouTube Video\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://trace-anything.github.io/viser-client/interactive.html\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/🖐️ Interactive%20Results-2b7a78?logoColor=white\" alt=\"Interactive Results\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://huggingface.co/depth-anything/trace-anything\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Model-f4b400?logo=huggingface\u0026logoColor=black\" alt=\"Hugging Face Model\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\n\n\u003cdiv align=\"center\" class=\"is-size-5 publication-authors\"\u003e\n  \u003cspan class=\"author-block\"\u003e\n    \u003ca href=\"https://xinhangliu.com/\"\u003eXinhang Liu\u003c/a\u003e\u003csup\u003e1,2\u003c/sup\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003c/span\u003e\n  \u003cspan class=\"author-block\"\u003e\n    \u003ca href=\"https://henry123-boy.github.io/\"\u003eYuxi Xiao\u003c/a\u003e\u003csup\u003e1,3\u003c/sup\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003c/span\u003e\n  \u003cspan class=\"author-block\"\u003e\n    \u003ca href=\"https://donydchen.github.io/\"\u003eDonny Y. Chen\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003c/span\u003e\n  \u003cspan class=\"author-block\"\u003e\n    \u003ca href=\"https://scholar.google.com.sg/citations?user=Q8iay0gAAAAJ\u0026hl=en\"\u003eJiashi Feng\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003c/span\u003e\n  \u003cbr\u003e\n  \u003cspan class=\"author-block\"\u003e\n    \u003ca href=\"https://yuwingtai.github.io/\"\u003eYu-Wing Tai\u003c/a\u003e\u003csup\u003e4\u003c/sup\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003c/span\u003e\n  \u003cspan class=\"author-block\"\u003e\n    \u003ca href=\"https://cse.hkust.edu.hk/~cktang/bio.html\"\u003eChi-Keung Tang\u003c/a\u003e\u003csup\u003e2\u003c/sup\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003c/span\u003e\n  \u003cspan class=\"author-block\"\u003e\n    \u003ca href=\"https://bingykang.github.io/\"\u003eBingyi Kang\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003c/span\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n\u003cdiv align=\"center\" class=\"is-size-5 publication-authors\"\u003e\n  \u003cspan class=\"author-block\"\u003e\u003csup\u003e1\u003c/sup\u003eBytedance Seed\u003c/span\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003cspan class=\"author-block\"\u003e\u003csup\u003e2\u003c/sup\u003eHKUST\u003c/span\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003cspan class=\"author-block\"\u003e\u003csup\u003e3\u003c/sup\u003eZhejiang University\u003c/span\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003cspan class=\"author-block\"\u003e\u003csup\u003e4\u003c/sup\u003eDartmouth College\u003c/span\u003e\n\u003c/div\u003e\n\n\n## Overview\nWe propose a 4D video representation, __trajectory field__, which maps each pixel across frames to a continuous, parametric 3D trajectory. With a single forward pass, the __Trace Anything__ model efficiently estimates such trajectory fields for any video, image pair, or unstructured image set.\nThis repository provides the official PyTorch implementation for running inference with the Trace Anything model and exploring trajectory fields in an interactive 3D viewer.\n\n  ![Teaser](assets/teaser.png)\n\n\n\n\n##  Setup\n\n### Create and activate environment\n\n```bash\n# Clone the repository\ngit clone https://github.com/ByteDance-Seed/TraceAnything.git\ncd TraceAnything\n\n# Create and activate environment\nconda create -n trace_anything python=3.10\nconda activate trace_anything\n```\n\n### Requirements\n\n* **Python** ≥ 3.10\n* **PyTorch** (install according to your CUDA/CPU setup)\n* **Dependencies**:\n\n  ```bash\n  pip install einops omegaconf pillow opencv-python viser imageio matplotlib torchvision\n  ```\n\n**Notes**\n\n- **CUDA:** Tested with **CUDA 12.8**.  \n- **GPU Memory:** The provided examples are tested to run on a **single GPU with ≥ 48 GB VRAM**.\n\n\n### Model weights\n\nDownload the pretrained **[model](https://huggingface.co/depth-anything/trace-anything/resolve/main/trace_anything.pt?download=true)** and place it at:\n\n```text\ncheckpoints/trace_anything.pt\n```\n\n\n\n##  Inference\n\nWe provide example input videos and image pairs under `examples/input`.\nEach subdirectory corresponds to a scene:\n\n```\nexamples/\n  input/\n    scene_name_1/\n      ...\n    scene_name_2/\n      ...\n```\n\nThe inference script loads images from these scene folders and produces outputs.\n\n---\n\n### Notes\n\n* Images must satisfy `W ≥ H`. (Portrait images are automatically transposed.)\n* Images are resized so that the long side = **512**, then cropped to the nearest multiple of 16 (a model requirement).\n* If the number of views exceeds 40, the script automatically downsamples.\n* (Advanced) The script assumes input images are ordered in time (e.g., video frames or paired images). Support for unstructured, unordered inputs will be released in the future.\n\n---\n\n### Running inference\n\nRun the model over all scenes:\n\n```bash\npython scripts/infer.py\n```\n\n#### Default arguments\n\nYou can override these paths with flags:\n\n* `--config     configs/eval.yaml`\n* `--ckpt       checkpoints/trace_anything.pt`\n* `--input_dir  examples/input`\n* `--output_dir examples/output`\n\n#### Example\n\n```bash\npython scripts/infer.py \\\n  --input_dir examples/input \\\n  --output_dir examples/output \\\n  --ckpt checkpoints/trace_anything.pt\n```\n\nResults are saved to:\n\n```text\n\u003coutput_dir\u003e/\u003cscene\u003e/output.pt\n```\n\n---\n\n### What’s inside `output.pt`?\n\n* `preds[i]['ctrl_pts3d']` — 3D control points, shape `[K, H, W, 3]`\n* `preds[i]['ctrl_conf']` — confidence maps, shape `[K, H, W]`\n* `preds[i]['fg_mask']` — binary mask `[H, W]`, computed via Otsu thresholding on control-point variance.\n  (Mask images are also saved under `\u003coutput_dir\u003e/\u003cscene\u003e/masks`.)\n* `preds[i]['time']` — predicted scalar time ∈ `[0, 1)`.\n\n  \u003e Even though the true timestamp is implicit from known sequence order, the network’s timestamp head still estimates it.\n* `views[i]['img']` — normalized input image tensor ∈ `[-1, 1]`\n\n\n\n## Optional: User-Guided Masks with SAM2\n\nIf you prefer **user-guided SAM2 masks** instead of the automatic masks computed from Trace Anything outputs (for visualization), we provide a helper script [`scripts/user_mask.py`](scripts/user_mask.py). This script lets you interactively select points on the first frame of a scene to produce per-frame foreground masks. \n\nInstall [SAM2](https://github.com/facebookresearch/sam2) and download its checkpoint. Then run with:\n\n```bash\npython scripts/user_mask.py --scene \u003coutput_scene_dir\u003e \\\n  --sam2_cfg configs/sam2.1/sam2.1_hiera_l.yaml \\\n  --sam2_ckpt \u003cpath_to_sam2_ckpt\u003e\n```\n\nThis will saves masks to:\n\n  ```\n  \u003cscene\u003e/masks/{i:03d}_user.png\n  ```\nIt also updates `\u003cscene\u003e/output.pt` with:\n\n  ```python\n  preds[i][\"fg_mask_user\"]\n  ```\n\nWhen visualizing, `fg_mask_user` will automatically be preferred over `fg_mask` if available.\n\n\n\n\n##  Interactive Visualization 🚀\nOur visualizer lets you explore the trajectory field interactively:\n\n![Interactive trajectory field demo](./assets/interactive_monkeys.gif)\n\n\nFire up the interactive 3D viewer and dive your trajectory fields:\n\n```bash\npython scripts/view.py --output examples/output/\u003cscene\u003e/output.pt\n```\n\n### Useful flags\n\n* `--port 8020` — set viewer port\n* `--t_step 0.025` — timeline step (smaller = more fine-grained curve evaluation)\n* `--ds 2` — downsample all data by `::2` for extra speed\n\n\n### Remote use (SSH port-forwarding)\n\n  ```bash\n  ssh -N -L 8020:localhost:8020 \u003cuser\u003e@\u003cserver\u003e\n  # Then open http://localhost:8020 locally\n  ```\n\n### Trajectory panel\n  Input a frame number, or simply type `\"mid\"` / `\"last\"`.\n  Then hit **Build / Refresh** to construct trajectories, and toggle **Show trajectories** to view them.\n\n  ![Trajectories Panel](assets/panel_screenshot.png)\n\n### Play around! 🎉\n\n  * Pump up or shrink point size\n  * Filter out noisy background / foreground points by confidence\n  * Drag to swivel the viewpoint\n  * Slide through time and watch the trajectories evolve\n\n## Acknowledgements\nWe sincerely thank the authors of the open-source repositories [DUSt3R](https://github.com/naver/dust3r), [Fast3R](https://github.com/facebookresearch/fast3r), [VGGT](https://github.com/facebookresearch/vggt), [MonST3R](https://github.com/Junyi42/monst3r), [Easi3R](https://github.com/Inception3D/Easi3R), [St4RTrack](https://github.com/HavenFeng/St4RTrack?tab=readme-ov-file), [POMATO](https://github.com/wyddmw/POMATO?tab=readme-ov-file), [SpaTrackerV2](https://github.com/henry123-boy/SpaTrackerV2) and [Viser](https://github.com/nerfstudio-project/viser) for their inspiring and high-quality work that greatly contributed to this project.\n\n\n## License\n\n- **Code**: Licensed under the [Apache 2.0 License](http://www.apache.org/licenses/LICENSE-2.0).  \n- **Model weights**: Licensed under the [CC BY-NC 4.0 License](https://creativecommons.org/licenses/by-nc/4.0/). These weights are provided for research and non-commercial use only.\n\n## Citation\n\nIf you find our repository useful, please consider giving it a star ⭐ and citing our paper in your work:\n\n```bibtex\n@misc{liu2025traceanythingrepresentingvideo,\n      title={Trace Anything: Representing Any Video in 4D via Trajectory Fields}, \n      author={Xinhang Liu and Yuxi Xiao and Donny Y. Chen and Jiashi Feng and Yu-Wing Tai and Chi-Keung Tang and Bingyi Kang},\n      year={2025},\n      eprint={2510.13802},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2510.13802}, \n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytedance-seed%2Ftraceanything","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbytedance-seed%2Ftraceanything","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytedance-seed%2Ftraceanything/lists"}