https://github.com/calledit/metric_depth_video_toolbox
Python tools for rendering, viewing and generating metric 3D depth videos. Tools for recovering and exporting camera pose and 3D geometry to popular formats as well as tools for projecting depthvideo in to side by side stereo(sbs) 3D stereo video.
https://github.com/calledit/metric_depth_video_toolbox
3d-reconstruction camera-pose-estimation depth-estimation monocular-depth-estimation video-depth
Last synced: 3 months ago
JSON representation
Python tools for rendering, viewing and generating metric 3D depth videos. Tools for recovering and exporting camera pose and 3D geometry to popular formats as well as tools for projecting depthvideo in to side by side stereo(sbs) 3D stereo video.
- Host: GitHub
- URL: https://github.com/calledit/metric_depth_video_toolbox
- Owner: calledit
- License: mit
- Created: 2025-01-27T08:43:53.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2026-01-24T15:39:10.000Z (3 months ago)
- Last Synced: 2026-01-24T23:51:11.747Z (3 months ago)
- Topics: 3d-reconstruction, camera-pose-estimation, depth-estimation, monocular-depth-estimation, video-depth
- Language: Python
- Homepage:
- Size: 648 KB
- Stars: 126
- Watchers: 9
- Forks: 11
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Metric depth video toolbox (MDVToolbox)
Tools for Generating and working with metric 3D depth videos.
_Banner created with 3d_view_depthfile.py_
## 📽️ Demos
**Showcase video:** https://youtu.be/nEiUloZ591Q
**Movie → 3D conversion demo**: https://www.youtube.com/watch?v=PLFjoNgkZDY
**Sample stereo clips:**
https://github.com/calledit/metric_depth_video_toolbox/releases/tag/Showcase
---
## ✨ Features
### Metric Depth Video Generation
Convert rgb videos into metric depth videos using multiple SOTA models:
- Depth-Anything series (`video_metric_convert.py`) and (`videoanythingmetric_video.py`)
- MoGe (`moge_video.py`)
- UniDepth (`unidepth_video.py`)
- UniK3D (`unik3d_video.py`)
- DepthPro (`depthpro_video.py`)
- DepthCrafter (`depthcrafter_video.py`)
- MVSAnywhere (`video_mvsa.py`)
### Stereo / 3D Conversion
- 2D movies → 3D (`movie_2_3D.py`)
- Stereo rendering from depth and rgb video (`stereo_rerender.py`)
- Parallax infill & diffusion-based stereo inpainting (`stereo_crafter_infill.py`)
### Visualization
- Real-time 3D viewer for metric depth videos (`3d_view_depthfile.py`)
- Novel-view rendering from depth video (`3d_view_depthfile.py`)
### Camera Tracking & 3D Reconstruction
- Depth-assisted camera tracking (`sam_track_video.py`) and (`align_3d_points.py`)
- Long-term point tracking (CoTracker3) (`track_points_in_video.py`)
- Pose extraction & alignment tools
### Export Tools
Export metric depth video to standard formats for use in other tools: (`convert_metric_depth_video_to_other_format.py`)
- `.ply` point clouds
- `.obj` meshes
- Blender `.blend` & Alembic `.abc` camera tracks
- 8-bit / 16-bit depth maps
- Depth rescaling based on triangulation
### Masking & Cleanup
- Automatic main subject masking (`generate_video_mask.py`)
- Subtitle/logo inpainting (`apply_inpainting.sh`)
---
## 📚 Documentation
| Topic | Link |
|---|---|
Beginner guide | [`HOWTO.md`](docs/HOWTO.md)
Movie → 3D guide | [`HOWTO_movie2_3d.md`](docs/HOWTO_movie2_3d.md)
Full tool reference | [`USAGE.md`](docs/USAGE.md)
GUI tutorial video | https://youtu.be/BE_aJCI7DHI
---
## 🧠 Depth Video Format
MDVT uses **RGB-encoded 16-bit metric depth**:
- **Red + Green** = upper 8 bits (duplicated for visibility)
- **Blue** = lower 8 bits
- Default range: **100 meters**
- Resolution: **~1.5mm depth precision**
Future upgrade for more precision will be done with: **24-bit depth** or **log-encoded depth** for long-range accuracy.
---
## ⚙️ Installation
### Windows
1. install git https://git-scm.com/downloads/win
2. Install miniconda https://docs.conda.io/en/latest/
3. Open the Anacoda prompt(miniconda) from the start menu
4. run
```batch
git clone https://github.com/calledit/metric_depth_video_toolbox
cd metric_depth_video_toolbox
windows_installer.bat
```
5. use `conda activate mdvt` to activate conda and use the tools. Some ML models are not supported on windows but the most esential ones are supported like Sterecrafter and depth anything.
### Ubuntu/Debian and OSX
```bash
git clone https://github.com/calledit/metric_depth_video_toolbox
cd metric_depth_video_toolbox
# on linux
sudo apt-get install -y libgl1
./install_mdvtoolbox.sh
#Optional (only required for some tools)
./install_mdvtoolbox.sh -megasam
./install_mdvtoolbox.sh -geometrycrafter
./install_mdvtoolbox.sh -unik3d
./install_mdvtoolbox.sh -depthpro
./install_mdvtoolbox.sh -stereocrafter
./install_mdvtoolbox.sh -madpose
./install_mdvtoolbox.sh -unidepth
./install_mdvtoolbox.sh -moge
./install_mdvtoolbox.sh -promptda
# if using headless linux you need to start a virtual x11 server
apt-get install xvfb
Xvfb :2 &
export DISPLAY=:2
# OSX (OSX only supports post processing of depth videos not generation of them. As the ML models need CUDA)
# (open3d requires python3.11 on OSX (as of 2025)))
pip3.11 install open3d numpy opencv-python
```
### Requirements
The tools that reuire ML models have been tested on machines with nvida 3090 cards that support Cuda 12.4 and Torch 2.5.1 on [vast.ai](https://cloud.vast.ai/?ref_id=148636) using "template PyTorch (cuDNN Devel)"
### Next steps
- Currently waiting for new stable and accurate depth models.
### Contributing
Is appreciated. Even for simple things like spelling.