{"id":16522704,"url":"https://github.com/patriciogonzalezvivo/prisma","last_synced_at":"2025-04-09T22:16:14.573Z","repository":{"id":218219155,"uuid":"708736422","full_name":"patriciogonzalezvivo/prisma","owner":"patriciogonzalezvivo","description":"Computational photography pipeline that performs multiple inferences from any image or video.","archived":false,"fork":false,"pushed_at":"2024-02-04T19:17:24.000Z","size":1628,"stargazers_count":245,"open_issues_count":6,"forks_count":8,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-09T22:16:07.670Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/patriciogonzalezvivo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-23T09:34:16.000Z","updated_at":"2025-03-29T14:01:59.000Z","dependencies_parsed_at":"2025-01-17T12:12:44.670Z","dependency_job_id":"1bf4238d-7515-4451-a786-0ccd5899e85f","html_url":"https://github.com/patriciogonzalezvivo/prisma","commit_stats":{"total_commits":104,"total_committers":2,"mean_commits":52.0,"dds":0.009615384615384581,"last_synced_commit":"e00192dd9a93b469c73b4a025617f234e972f9de"},"previous_names":["patriciogonzalezvivo/prisma"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patriciogonzalezvivo%2Fprisma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patriciogonzalezvivo%2Fprisma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patriciogonzalezvivo%2Fprisma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patriciogonzalezvivo%2Fprisma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/patriciogonzalezvivo","download_url":"https://codeload.github.com/patriciogonzalezvivo/prisma/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248119287,"owners_count":21050755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T17:00:46.318Z","updated_at":"2025-04-09T22:16:14.550Z","avatar_url":"https://github.com/patriciogonzalezvivo.png","language":"Python","funding_links":[],"categories":["Digital Content Creation Software (DCCs)","Tools, Pipeline \u0026 Utilities"],"sub_categories":["Tools","Scene \u0026 Pipeline Tools"],"readme":"![_prisma_](https://github.com/patriciogonzalezvivo/prisma/assets/346914/0a468415-5a19-4993-a9ff-e8ee867fc629)\n\n# PRISMA\n\nPRISMA is a computational photography pipeline that performs multiple inferences (refere as \"bands\") from any image or video. Like light pasing through a prism that bends it into different wavelengths, this pipeline expands images into data that can be use for 3D reconstruction or realtime post-processing operations.\n\nIt's a combination of different algorithms and open sourced pre-train models such as:\n\n* Monocular `depth` ([MiDAS v3.1, ZoeDepth, Marigold, PatchFusion, Depth_Anything](https://medium.com/@patriciogv/the-state-of-the-art-of-depth-estimation-from-single-images-9e245d51a315))\n* Optical `flow` (RAFT, GMFlow)\n* Segmentation `mask` (mmdet)\n* `camera pose` (COLMAP)\n\n![2024-01-20 09-31-13](https://github.com/patriciogonzalezvivo/prisma/assets/346914/9f8960e5-bf52-479f-bbc3-d816ab9644ea)\n\nThe resulting bands are stored in a folder with the same name as the input file. Each band is stored as a single `.png` or `.mp4` file. And can be imported on:\n\n* [Estimated depth can be importer to Blender projects using this blender project](https://github.com/patriciogonzalezvivo/prisma_blender), also [COLMAP scenes can be imported using this addon](https://github.com/SBCV/Blender-Addon-Photogrammetry-Importer)\n* [GlslViewer for applying real-time shaders](https://github.com/patriciogonzalezvivo/prisma_glslViewer)\n* Videos can be use for both NeRFs (like [NVidia's Instant-ngp](https://github.com/NVlabs/instant-ngp)) or [Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting) training. \n\n**Notes**:\n* Infered depth is exported by default as a heatmap that can be decoded realtime using [LYGIA's heatmap GLSL/HLSL sampling](https://lygia.xyz/sample/heatmap). \n* optical flow is encoded as HUE (angle) and saturation which also can be decoded realtime using [LYGIA opticalFlow GLSL/HLSL sampler](https://lygia.xyz/sample/opticalFlow).\n\n## Install\n\nMain dependencies:\n\n* [Conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html)\n* [COLMAP](https://colmap.github.io/install.html)\n\n```Shell\ngit clone git@github.com:patriciogonzalezvivo/prisma.git\ncd prisma\n\nconda env create -f environment.yml\nconda activate prisma\nsh download_models.sh\n\n# Install mmcv (for mmdetection)\npip install -U openmim\nmim install mmengine\nmim install \"mmcv-full==1.7.1\"\n```\n\n## How it works?\n\n### a. Process\n\nWe start by processing an image or video. Let's start by processing an image:\n\n```bash\npython process.py -i data/gog.jpg\n```\n\nWith out providing an `--output` this will create a folder with the same filename which will contain all the derived bands (`rgba`, `flow`, `mask` and `depth_*`).\n\n```\ngog.jpg\ngog/\n├── depth_patchfusion.png\n├── mask.png\n├── metadata.json\n└── rgba.png\n```\n\nIn the forlder you will find a `metadata.json` file that contains all the metadata associated with the original image or video.\n\n```json\n{\n    \"bands\": {\n        \"rgba\": {\n            \"url\": \"rgba.png\"\n        },\n        \"depth_patchfusion\": {\n            \"url\": \"depth_patchfusion.png\",\n            \"values\": {\n                \"min\": {\n                    \"value\": 1.6147574186325073,\n                    \"type\": \"float\"\n                },\n                \"max\": {\n                    \"value\": 11.678544044494629,\n                    \"type\": \"float\"\n                }\n            }\n        },\n        \"mask\": {\n            \"url\": \"mask.png\",\n            \"ids\": [\n                \"person\",\n                \"bird\",\n                \"cat\",\n                \"dog\",\n                \"horse\",\n                \"sheep\",\n                \"cow\",\n                \"elephant\",\n                \"bear\",\n                \"zebra\",\n                \"giraffe\"\n            ]\n        }\n    },\n    \"width\": 934,\n    \"height\": 440,\n    \"principal_point\": [\n        467.0,\n        220.0\n    ],\n    \"focal_length\": 641.0616195031489,\n    \"field_of_view\": 37.88246641919117\n}\n```\n\nCurrently PRISMA supports multiple depth estimation algorithms. You can select which one to use by providing the `--depth`|`-d` argument: `depth_midas`, `depth_zoedepth`, `depth_patchfusion`, `depth_marigold` or `all`. By defualt images will be processed using `depth_patchfusion`, while videos will use `depth_anything`.\n\nWhen processing videos, by default PRISMA creates the least ammount of data by creating a single `.png` or `.mp4` for each band. In the case of videos data like min/max values will be stored on `.cvs`.\n\nit's possible to save extra data by setting the `--extra`|`-e` level number.\n\n0. store bands as a single `.png` and `.mp4` (video have usually an associated `.csv` file)\n1. store images as `.ply` point clouds, for videos it extracts the reslting frames as `.png`\n2. store optical flow from videos as `.flo` files.\n3. store inferenced depth as `.npy` files.\n\nLet's try now extracting all depth models and individual frames from a video:\n\n```bash\npython process.py -i data/rocky.mp4 -d all -e 1\n```\n\nWhich produce the folowing folder structure:\n\n```\nrocky.mp4\nrocky/\n├── depth_anything/\n│   ├── 000000.png\n│   ├── 000001.png\n│   ├── ...\n│   └── 000110.png\n├── depth_anything_max.csv\n├── depth_anything_min.csv\n├── depth_anything.mp4\n├── depth_marigold/\n│   ├── 000000.png\n│   ├── 000001.png\n│   ├── ...\n│   └── 000110.png\n├── depth_marigold_max.csv\n├── depth_marigold_min.csv\n├── depth_marigold.mp4\n├── depth_midas/\n│   ├── 000000.png\n│   ├── 000001.png\n│   ├── ...\n│   └── 000110.png\n├── depth_midas_max.csv\n├── depth_midas_min.csv\n├── depth_midas.mp4\n├── depth_patchfusion/\n│   ├── 000000.png\n│   ├── 000001.png\n│   ├── ...\n│   └── 000110.png\n├── depth_patchfusion_max.csv\n├── depth_patchfusion_min.csv\n├── depth_patchfusion.mp4\n├── depth_zoedepth/\n│   ├── 000000.png\n│   ├── 000001.png\n│   ├── ...\n│   └── 000110.png\n├── depth_zoedepth_max.csv\n├── depth_zoedepth_min.csv\n├── depth_zoedepth.mp4\n├── flow_raft/\n│   ├── 000000.png\n│   ├── 000001.png\n│   ├── ...\n│   └── 000110.png\n├── flow_raft.csv\n├── flow_raft.mp4\n├── flow_gmflow/\n│   ├── 000000.png\n│   ├── 000001.png\n│   ├── ...\n│   └── 000110.png\n├── flow_gmflow.csv\n├── flow_gmflow.mp4\n├── images/\n│   ├── 000000.png\n│   ├── 000001.png\n│   ├── ...\n│   └── 000110.png\n├── mask/\n|   ├── 000000.png\n|   ├── 000001.png\n|   ├── ...\n|   └── 000110.png\n├── mask.mp4\n|── sparse/\n|   └── 0/\n|       ├── cameras.bin\n|       ├── images.bin\n|       ├── points3D.bin\n|       └── points3D.txt\n|── camera_pose.csv\n|── colmap.db\n├── metadata.json\n└── rgba.mp4\n```\n\n### b. Visualize\n\nView the resulting bands from the processed image/video using [ReRun](https://www.rerun.io/):\n\n```bash\n\n```bash\npython view.py -i data/rocky\n```\n\n![2024-01-20 06-35-33](https://github.com/patriciogonzalezvivo/prisma/assets/346914/7abff827-631a-45cd-8aba-819172f59877)\n\n### c. Concatenate bands\n\nIn order to export the bands as a single image or video you can use the `concat.py` script:\n\n```bash\npython concat.py -i data/gog -o test.png\n```\n\n![test](https://github.com/patriciogonzalezvivo/prisma/assets/346914/763d3ada-736c-4676-ad4f-55eafe9dcf40)\n\n## Licenses and Credits\n\nThis pipeline is Copyright (c) 2024, Patricio Gonzalez Vivo and Licensed under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en) please reach out to patriciogonzalezvivo at gmail dot com, for getting a comercial license.\n\nAll the models and software used by it are commercial ready licenses like MIT, Apache and BSD.\n\n### Depth estimation (MiDAS 3.1)\n\n**Paper:** [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://arxiv.org/abs/1907.01341v3)\n\n**License:** [MIT](bands/midas/LICENSE)\n\n**Code Repo:** [isl-org/MiDaS](https://github.com/isl-org/MiDaS)\n\n**Use:**\n\n```Shell\ndepth_midas.py --input \u003cIMAGE/VIDEO\u003e --output \u003cIMAGE/VIDEO\u003e\n```\n\nCitation:\n```\n@ARTICLE {Ranftl2022,\n    author  = \"Ren\\'{e} Ranftl and Katrin Lasinger and David Hafner and Konrad Schindler and Vladlen Koltun\",\n    title   = \"Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer\",\n    journal = \"IEEE Transactions on Pattern Analysis and Machine Intelligence\",\n    year    = \"2022\",\n    volume  = \"44\",\n    number  = \"3\"\n}\n```\n\nCitation for DPT-based model:\n```\n@article{Ranftl2021,\n    author    = {Ren\\'{e} Ranftl and Alexey Bochkovskiy and Vladlen Koltun},\n    title     = {Vision Transformers for Dense Prediction},\n    journal   = {ArXiv preprint},\n    year      = {2021},\n}\n```\n\n\n### Depth Estimation (ZoeDepth)\n\n**Paper:** [Zero-shot Transfer by Combining Relative and Metric Depth](https://arxiv.org/abs/2302.12288)\n\n**License:** [MIT](bands/patchfusion/zoedepth/LICENSE)\n\n**Code Repo:** [isl-org/ZoeDepth](https://github.com/isl-org/ZoeDepth)\n\n**Use:**\n\n```Shell\ndepth_zoedepth.py --input \u003cIMAGE/VIDEO\u003e --output \u003cIMAGE/VIDEO\u003e\n```\n\nCitation\n```\n@misc{https://doi.org/10.48550/arxiv.2302.12288,\n    doi = {10.48550/ARXIV.2302.12288},\n    url = {https://arxiv.org/abs/2302.12288},\n    author = {Bhat, Shariq Farooq and Birkl, Reiner and Wofk, Diana and Wonka, Peter and Müller, Matthias},  \n    keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},\n    title = {ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth},  \n    publisher = {arXiv},\n    year = {2023},\n    copyright = {arXiv.org perpetual, non-exclusive license}\n}\n```\n\n\n### Depth Estimation (PatchFusion)\n\n**Paper:** [PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation](https://zhyever.github.io/patchfusion/images/paper.pdf)\n\n**License:** [MIT](bands/patchfusion/LICENSE)\n\n**Code Repo:** [zhyever/PatchFusion](https://github.com/zhyever/PatchFusion)\n\n**Use:**\n\n```Shell\ndepth_patchfusion.py --input \u003cIMAGE/VIDEO\u003e --output \u003cIMAGE/VIDEO\u003e\n```\n\n**Note:** [This pretrained model](https://huggingface.co/zhyever/PatchFusion/resolve/main/patchfusion_u4k.pt?download=true) needs to be downloaded and placed in the `models/` folder.\n\n\nCitation\n\n```\n@article{li2023patchfusion,\n    title={PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation}, \n    author={Zhenyu Li and Shariq Farooq Bhat and Peter Wonka},\n    year={2023},\n    eprint={2312.02284},\n    archivePrefix={arXiv},\n    primaryClass={cs.CV}}\n```\n\n\n### Depth Estimation (Marigold)\n\n**Paper:** [Marigold: Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://arxiv.org/abs/2312.02145)\n\n**License:** [Apache](bands/marigold/LICENSE)\n\n**Code Repo:** [prs-eth/Marigold](https://github.com/prs-eth/Marigold)\n\n**Use:**\n\n```Shell\ndepth_marigold.py --input \u003cIMAGE/VIDEO\u003e --output \u003cIMAGE/VIDEO\u003e\n```\n\nCitation\n\n```bibtex\n@misc{ke2023repurposing,\n      title={Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation}, \n      author={Bingxin Ke and Anton Obukhov and Shengyu Huang and Nando Metzger and Rodrigo Caye Daudt and Konrad Schindler},\n      year={2023},\n      eprint={2312.02145},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\n\n### Depth Estimation (Depth Anything)\n\n**Paper:** [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://github.com/LiheYoung/Depth-Anything/blob/main/assets/paper.pdf)\n\n**License:** [Apache](bands/d_anything/LICENSE)\n\n**Code Repo:** [LiheYoung/Depth-Anything](https://github.com/LiheYoung/Depth-Anything)\n\n**Use:**\n\n```Shell\ndepth_anything.py --input \u003cIMAGE/VIDEO\u003e --output \u003cIMAGE/VIDEO\u003e\n```\n\nCitation\n\n```bibtex\n@article{depthanything,\n      title={Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data}, \n      author={Yang, Lihe and Kang, Bingyi and Huang, Zilong and Xu, Xiaogang and Feng, Jiashi and Zhao, Hengshuang},\n      journal={arXiv:2401.10891},\n      year={2024}\n}\n```\n\n\n### Optical Flow (RAFT)\n\nBased on https://github.com/SharifElfouly/opical-flow-estimation-with-RAFT\n\nSeems to be very good: [Optical Flow Estimation Benchmark](https://paperswithcode.com/sota/optical-flow-estimation-on-sintel-clean)\n\n**Paper:** [RAFT: Recurrent All Pairs Field Transforms for Optical Flow](https://arxiv.org/pdf/2003.12039)\n\n**License:** [BSD](bands/raft/LICENSE)\n\n**Code Repo:** [princeton-vl/RAFT](https://github.com/princeton-vl/RAFT)\n\n**Use:**\n\n```Shell\nflow_raft.py --input \u003cIMAGE/VIDEO\u003e --output \u003cIMAGE/VIDEO\u003e\n```\n\n\n### Optical Flow (GMFlow)\n\n**Paper:** [GMFlow: Learning Optical Flow via Global Matching](https://arxiv.org/abs/2111.13680)\n\n**License:** [Apache](bands/gmflow/LICENSE)\n\n**Code Repo:** [haofeixu/gmflow](https://github.com/haofeixu/gmflow)\n\n**Use:**\n\n```Shell\nflow_gmflow.py --input \u003cIMAGE/VIDEO\u003e --output \u003cIMAGE/VIDEO\u003e\n```\n\n\n### Segmentation (MMDetection)\n\n**Code Repo:** [MMDetection](https://github.com/open-mmlab/mmdetection)\n\n**License:** [Apache](bands/mmdet/LICENSE)\n\n**Use:**\n\n```Shell\nmask_mmdet.py --input \u003cIMAGE/VIDEO\u003e --output \u003cIMAGE/VIDEO\u003e\n```\n\nCitation:\n```\n@article{mmdetection,\n    title   = {{MMDetection}: Open MMLab Detection Toolbox and Benchmark},\n    author  = {Chen, Kai and Wang, Jiaqi and Pang, Jiangmiao and Cao, Yuhang and\n              Xiong, Yu and Li, Xiaoxiao and Sun, Shuyang and Feng, Wansen and\n              Liu, Ziwei and Xu, Jiarui and Zhang, Zheng and Cheng, Dazhi and\n              Zhu, Chenchen and Cheng, Tianheng and Zhao, Qijie and Li, Buyu and\n              Lu, Xin and Zhu, Rui and Wu, Yue and Dai, Jifeng and Wang, Jingdong\n              and Shi, Jianping and Ouyang, Wanli and Loy, Chen Change and Lin, Dahua},\n    journal= {arXiv preprint arXiv:1906.07155},\n    year={2019}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatriciogonzalezvivo%2Fprisma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpatriciogonzalezvivo%2Fprisma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatriciogonzalezvivo%2Fprisma/lists"}