{"id":20388180,"url":"https://github.com/ivy-llc/vision","last_synced_at":"2025-04-12T10:32:11.361Z","repository":{"id":40237933,"uuid":"330975829","full_name":"ivy-llc/vision","owner":"ivy-llc","description":"3D Vision functions with end-to-end support for deep learning developers, written in Ivy.","archived":false,"fork":false,"pushed_at":"2023-08-28T06:44:00.000Z","size":26588,"stargazers_count":71,"open_issues_count":1,"forks_count":17,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-03-24T17:17:21.296Z","etag":null,"topics":["3d","deep-learning","depth","ivy","jax","machine-learning","mxnet","numpy","point-cloud","python","pytorch","rendering","sdf","tensorflow","vision","voxel"],"latest_commit_sha":null,"homepage":"https://unify.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivy-llc.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-01-19T12:41:06.000Z","updated_at":"2024-10-26T06:03:50.000Z","dependencies_parsed_at":"2023-09-24T06:31:03.481Z","dependency_job_id":null,"html_url":"https://github.com/ivy-llc/vision","commit_stats":{"total_commits":276,"total_committers":17,"mean_commits":"16.235294117647058","dds":0.4782608695652174,"last_synced_commit":"e04cdfc63694608b07c0167e267193a81e2de432"},"previous_names":["ivy-dl/vision","unifyai/vision","ivy-llc/vision","transpile-ai/vision"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivy-llc%2Fvision","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivy-llc%2Fvision/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivy-llc%2Fvision/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivy-llc%2Fvision/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivy-llc","download_url":"https://codeload.github.com/ivy-llc/vision/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248553154,"owners_count":21123387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","deep-learning","depth","ivy","jax","machine-learning","mxnet","numpy","point-cloud","python","pytorch","rendering","sdf","tensorflow","vision","voxel"],"created_at":"2024-11-15T03:07:46.399Z","updated_at":"2025-04-12T10:32:11.001Z","avatar_url":"https://github.com/ivy-llc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":".. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/logo.png?raw=true#gh-light-mode-only\n   :width: 100%\n   :class: only-light\n\n.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/logo_dark.png?raw=true#gh-dark-mode-only\n   :width: 100%\n   :class: only-dark\n\n.. raw:: html\n\n    \u003cbr/\u003e\n    \u003ca href=\"https://pypi.org/project/ivy-vision/0.0.1.post0/\"\u003e\n        \u003cimg class=\"dark-light\" style=\"float: left; padding-right: 4px; padding-bottom: 4px;\" src=\"https://badge.fury.io/py/ivy-vision.svg\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/unifyai/vision/actions?query=workflow%3Adocs\"\u003e\n        \u003cimg class=\"dark-light\" style=\"float: left; padding-right: 4px; padding-bottom: 4px;\" src=\"https://github.com/unifyai/vision/actions/workflows/docs.yml/badge.svg\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/unifyai/vision/actions?query=workflow%3Anightly-tests\"\u003e\n        \u003cimg class=\"dark-light\" style=\"float: left; padding-right: 4px; padding-bottom: 4px;\" src=\"https://github.com/unifyai/vision/actions/workflows/nightly-tests.yml/badge.svg\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://discord.gg/G4aR9Q7DTN\"\u003e\n        \u003cimg class=\"dark-light\" style=\"float: left; padding-right: 4px; padding-bottom: 4px;\" src=\"https://img.shields.io/discord/799879767196958751?color=blue\u0026label=%20\u0026logo=discord\u0026logoColor=white\"\u003e\n    \u003c/a\u003e\n    \u003cbr clear=\"all\" /\u003e\n\n**3D Vision functions with end-to-end support for machine learning developers, written in Ivy.**\n\n.. raw:: html\n\n    \u003cdiv style=\"display: block;\" align=\"center\"\u003e\n        \u003cimg class=\"dark-light\" width=\"6%\" style=\"float: left;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/supported/empty.png\"\u003e\n        \u003ca href=\"https://jax.readthedocs.io\"\u003e\n            \u003cimg class=\"dark-light\" width=\"13%\" style=\"float: left;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/supported/jax_logo.png\"\u003e\n        \u003c/a\u003e\n        \u003cimg class=\"dark-light\" width=\"12%\" style=\"float: left;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/supported/empty.png\"\u003e\n        \u003ca href=\"https://www.tensorflow.org\"\u003e\n            \u003cimg class=\"dark-light\" width=\"13%\" style=\"float: left;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/supported/tensorflow_logo.png\"\u003e\n        \u003c/a\u003e\n        \u003cimg class=\"dark-light\" width=\"12%\" style=\"float: left;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/supported/empty.png\"\u003e\n        \u003ca href=\"https://pytorch.org\"\u003e\n            \u003cimg class=\"dark-light\" width=\"13%\" style=\"float: left;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/supported/pytorch_logo.png\"\u003e\n        \u003c/a\u003e\n        \u003cimg class=\"dark-light\" width=\"12%\" style=\"float: left;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/supported/empty.png\"\u003e\n        \u003ca href=\"https://numpy.org\"\u003e\n            \u003cimg class=\"dark-light\" width=\"13%\" style=\"float: left;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/supported/numpy_logo.png\"\u003e\n        \u003c/a\u003e\n        \u003cimg class=\"dark-light\" width=\"6%\" style=\"float: left;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/supported/empty.png\"\u003e\n    \u003c/div\u003e\n\nContents\n--------\n\n* `Overview`_\n* `Run Through`_\n* `Interactive Demos`_\n* `Get Involved`_\n\nOverview\n--------\n\n.. _docs: https://unify.ai/docs/vision/\n\n**What is Ivy Vision?**\n\nIvy vision focuses predominantly on 3D vision, with functions for camera geometry, image projections,\nco-ordinate frame transformations, forward warping, inverse warping, optical flow, depth triangulation, voxel grids,\npoint clouds, signed distance functions, and others.  Check out the docs_ for more info!\n\nThe library is built on top of the Ivy machine learning framework.\nThis means all functions simultaneously support:\nJax, Tensorflow, PyTorch, MXNet, and Numpy.\n\n**Ivy Libraries**\n\nThere are a host of derived libraries written in Ivy, in the areas of mechanics, 3D vision, robotics, gym environments,\nneural memory, pre-trained models + implementations, and builder tools with trainers, data loaders and more. Click on the icons below to learn more!\n\n.. raw:: html\n\n    \u003cdiv style=\"display: block;\"\u003e\n        \u003ca href=\"https://github.com/unifyai/mech\"\u003e\n            \u003cpicture\u003e\n                \u003csource class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_mech_dark.png\"\u003e\n                \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_mech.png\"\u003e\n            \u003c/picture\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/vision\"\u003e\n            \u003cpicture\u003e\n                \u003csource class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_vision_dark.png\"\u003e\n                \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_vision.png\"\u003e\n            \u003c/picture\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/robot\"\u003e\n            \u003cpicture\u003e\n                \u003csource class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_robot_dark.png\"\u003e\n                \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_robot.png\"\u003e\n            \u003c/picture\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/gym\"\u003e\n            \u003cpicture\u003e\n                \u003csource class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_gym_dark.png\"\u003e\n                \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_gym.png\"\u003e\n            \u003c/picture\u003e\n        \u003c/a\u003e\n\n        \u003cbr clear=\"all\" /\u003e\n\n        \u003ca href=\"https://pypi.org/project/ivy-mech/0.0.1.post0/\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://badge.fury.io/py/ivy-mech.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://pypi.org/project/ivy-vision/0.0.1.post0/\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://badge.fury.io/py/ivy-vision.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://pypi.org/project/ivy-robot/0.0.1.post0/\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://badge.fury.io/py/ivy-robot.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://pypi.org/project/ivy-gym\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\"width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://badge.fury.io/py/ivy-gym.svg\"\u003e\n        \u003c/a\u003e\n\n        \u003cbr clear=\"all\" /\u003e\n\n        \u003ca href=\"https://github.com/unifyai/mech/actions?query=workflow%3Anightly-tests\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\"src=\"https://github.com/unifyai/mech/actions/workflows/nightly-tests.yml/badge.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/vision/actions?query=workflow%3Anightly-tests\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://github.com/unifyai/vision/actions/workflows/nightly-tests.yml/badge.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/robot/actions?query=workflow%3Anightly-tests\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://github.com/unifyai/robot/actions/workflows/nightly-tests.yml/badge.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/gym/actions?query=workflow%3Anightly-tests\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://github.com/unifyai/gym/actions/workflows/nightly-tests.yml/badge.svg\"\u003e\n        \u003c/a\u003e\n\n        \u003cbr clear=\"all\" /\u003e\n\n        \u003ca href=\"https://github.com/unifyai/memory\"\u003e\n            \u003cpicture\u003e\n                \u003csource class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_memory_dark.png\"\u003e\n                \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_memory.png\"\u003e\n            \u003c/picture\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/builder\"\u003e\n            \u003cpicture\u003e\n                \u003csource class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_builder_dark.png\"\u003e\n                \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_builder.png\"\u003e\n            \u003c/picture\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/models\"\u003e\n            \u003cpicture\u003e\n                \u003csource class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_models_dark.png\"\u003e\n                \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_models.png\"\u003e\n            \u003c/picture\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/ecosystem\"\u003e\n            \u003cpicture\u003e\n                \u003csource class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_ecosystem_dark.png\"\u003e\n                \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/logos/ivy_ecosystem.png\"\u003e\n            \u003c/picture\u003e\n        \u003c/a\u003e\n\n        \u003cbr clear=\"all\" /\u003e\n\n        \u003ca href=\"https://pypi.org/project/ivy-memory/0.0.1.post0/\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://badge.fury.io/py/ivy-memory.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://pypi.org/project/ivy-builder/0.0.1.post0/\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://badge.fury.io/py/ivy-builder.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://pypi.org/project/ivy-models\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://badge.fury.io/py/ivy-models.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/ecosystem/actions?query=workflow%3Adocs\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://github.com/unifyai/ecosystem/actions/workflows/docs.yml/badge.svg\"\u003e\n        \u003c/a\u003e\n\n        \u003cbr clear=\"all\" /\u003e\n\n        \u003ca href=\"https://github.com/unifyai/memory/actions?query=workflow%3Anightly-tests\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://github.com/unifyai/memory/actions/workflows/nightly-tests.yml/badge.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/builder/actions?query=workflow%3Anightly-tests\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://github.com/unifyai/builder/actions/workflows/nightly-tests.yml/badge.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/unifyai/models/actions?query=workflow%3Anightly-tests\"\u003e\n            \u003cimg class=\"dark-light\" width=\"15%\" style=\"float: left; margin: 0% 5%;\" src=\"https://github.com/unifyai/models/actions/workflows/nightly-tests.yml/badge.svg\"\u003e\n        \u003c/a\u003e\n\n        \u003cbr clear=\"all\" /\u003e\n\n    \u003c/div\u003e\n    \u003cbr clear=\"all\" /\u003e\n\n**Quick Start**\n\nIvy vision can be installed like so: ``pip install ivy-vision==0.0.1.post0``\n\n.. _demos: https://github.com/unifyai/vision/tree/main/ivy_vision_demos\n.. _interactive: https://github.com/unifyai/vision/tree/main/ivy_vision_demos/interactive\n\nTo quickly see the different aspects of the library, we suggest you check out the demos_!\nwe suggest you start by running the script ``run_through.py``,\nand read the \"Run Through\" section below which explains this script.\n\nFor more interactive demos, we suggest you run either\n``coords_to_voxel_grid.py`` or ``render_image.py`` in the interactive_ demos folder.\n\nRun Through\n-----------\n\nWe run through some of the different parts of the library via a simple ongoing example script.\nThe full script is available in the demos_ folder, as file ``run_through.py``.\nFirst, we select a random backend framework to use for the examples, from the options\n``ivy.jax``, ``ivy.tensorflow``, ``ivy.torch``, ``ivy.mxnet`` or ``ivy.numpy``,\nand use this to set the ivy backend framework.\n\n.. code-block:: python\n\n    import  ivy\n    ivy.set_backend(ivy.choose_random_backend())\n\n**Camera Geometry**\n\nTo get to grips with some of the basics, we next show how to construct ivy containers which represent camera geometry.\nThe camera intrinsic matrix, extrinsic matrix, full matrix, and all of their inverses are central to most of the\nfunctions in this library.\n\nAll of these matrices are contained within the Ivy camera geometry class.\n\n.. code-block:: python\n\n    # intrinsics\n\n    # common intrinsic params\n    img_dims = [512, 512]\n    pp_offsets = ivy.array([dim / 2 - 0.5 for dim in img_dims], 'float32')\n    cam_persp_angles = ivy.array([60 * np.pi / 180] * 2, 'float32')\n\n    # ivy cam intrinsics container\n    intrinsics = ivy_vision.persp_angles_and_pp_offsets_to_intrinsics_object(\n        cam_persp_angles, pp_offsets, img_dims)\n\n    # extrinsics\n\n    # 3 x 4\n    cam1_inv_ext_mat = ivy.array(np.load(data_dir + '/cam1_inv_ext_mat.npy'), 'float32')\n    cam2_inv_ext_mat = ivy.array(np.load(data_dir + '/cam2_inv_ext_mat.npy'), 'float32')\n\n    # full geometry\n\n    # ivy cam geometry container\n    cam1_geom = ivy_vision.inv_ext_mat_and_intrinsics_to_cam_geometry_object(\n        cam1_inv_ext_mat, intrinsics)\n    cam2_geom = ivy_vision.inv_ext_mat_and_intrinsics_to_cam_geometry_object(\n        cam2_inv_ext_mat, intrinsics)\n    cam_geoms = [cam1_geom, cam2_geom]\n\nThe geometries used in this quick start demo are based upon the scene presented below.\n\n.. image:: https://github.com/unifyai/vision/blob/main/docs/images/scene.png?raw=true\n   :width: 100%\n\nThe code sample below demonstrates all of the attributes contained within the Ivy camera geometry class.\n\n.. code-block:: python\n\n    for cam_geom in cam_geoms:\n\n        assert cam_geom.intrinsics.focal_lengths.shape == (2,)\n        assert cam_geom.intrinsics.persp_angles.shape == (2,)\n        assert cam_geom.intrinsics.pp_offsets.shape == (2,)\n        assert cam_geom.intrinsics.calib_mats.shape == (3, 3)\n        assert cam_geom.intrinsics.inv_calib_mats.shape == (3, 3)\n\n        assert cam_geom.extrinsics.cam_centers.shape == (3, 1)\n        assert cam_geom.extrinsics.Rs.shape == (3, 3)\n        assert cam_geom.extrinsics.inv_Rs.shape == (3, 3)\n        assert cam_geom.extrinsics.ext_mats_homo.shape == (4, 4)\n        assert cam_geom.extrinsics.inv_ext_mats_homo.shape == (4, 4)\n\n        assert cam_geom.full_mats_homo.shape == (4, 4)\n        assert cam_geom.inv_full_mats_homo.shape == (4, 4)\n\n**Load Images**\n\nWe next load the color and depth images corresponding to the two camera frames.\nWe also construct the depth-scaled homogeneous pixel co-ordinates for each image,\nwhich is a central representation for the ivy_vision functions.\nThis representation simplifies projections between frames.\n\n.. code-block:: python\n\n    # load images\n\n    # h x w x 3\n    color1 = ivy.array(cv2.imread(data_dir + '/rgb1.png').astype(np.float32) / 255)\n    color2 = ivy.array(cv2.imread(data_dir + '/rgb2.png').astype(np.float32) / 255)\n\n    # h x w x 1\n    depth1 = ivy.array(np.reshape(np.frombuffer(cv2.imread(\n        data_dir + '/depth1.png', -1).tobytes(), np.float32), img_dims + [1]))\n    depth2 = ivy.array(np.reshape(np.frombuffer(cv2.imread(\n        data_dir + '/depth2.png', -1).tobytes(), np.float32), img_dims + [1]))\n\n    # depth scaled pixel coords\n\n    # h x w x 3\n    u_pix_coords = ivy_vision.create_uniform_pixel_coords_image(img_dims)\n    ds_pixel_coords1 = u_pix_coords * depth1\n    ds_pixel_coords2 = u_pix_coords * depth2\n\nThe rgb and depth images are presented below.\n\n.. image:: https://github.com/unifyai/vision/blob/main/docs/images/rgb_and_depth.png?raw=true\n   :width: 100%\n\n**Optical Flow and Depth Triangulation**\n\nNow that we have two cameras, their geometries, and their images fully defined,\nwe can start to apply some of the more interesting vision functions.\nWe start with some optical flow and depth triangulation functions.\n\n.. code-block:: python\n\n    # required mat formats\n    cam1to2_full_mat_homo = ivy.matmul(cam2_geom.full_mats_homo, cam1_geom.inv_full_mats_homo)\n    cam1to2_full_mat = cam1to2_full_mat_homo[..., 0:3, :]\n    full_mats_homo = ivy.concat((ivy.expand_dims(cam1_geom.full_mats_homo, axis=0),\n                                      ivy.expand_dims(cam2_geom.full_mats_homo, axis=0)), axis=0)\n    full_mats = full_mats_homo[..., 0:3, :]\n\n    # flow\n    flow1to2 = ivy_vision.flow_from_depth_and_cam_mats(ds_pixel_coords1, cam1to2_full_mat)\n\n    # depth again\n    depth1_from_flow = ivy_vision.depth_from_flow_and_cam_mats(flow1to2, full_mats)\n\nVisualizations of these images are given below.\n\n.. image:: https://github.com/unifyai/vision/blob/main/docs/images/flow_and_depth.png?raw=true\n   :width: 100%\n\n**Inverse and Forward Warping**\n\nMost of the vision functions, including the flow and depth functions above,\nmake use of image projections,\nwhereby an image of depth-scaled homogeneous pixel-coordinates is transformed into\ncartesian co-ordinates relative to the acquiring camera, the world, another camera,\nor transformed directly to pixel co-ordinates in another camera frame.\nThese projections also allow warping of the color values from one camera to another.\n\nFor inverse warping, we assume depth to be known for the target frame.\nWe can then determine the pixel projections into the source frame,\nand bilinearly interpolate these color values at the pixel projections,\nto infer the color image in the target frame.\n\nTreating frame 1 as our target frame,\nwe can use the previously calculated optical flow from frame 1 to 2, in order\nto inverse warp the color data from frame 2 to frame 1, as shown below.\n\n\n.. code-block:: python\n\n    # inverse warp rendering\n    warp = u_pix_coords[..., 0:2] + flow1to2\n    color2_warp_to_f1 = ivy_vision.image.bilinear_resample(color2, warp)\n\n    # projected depth scaled pixel coords 2\n    ds_pixel_coords1_wrt_f2 = ivy_vision.ds_pixel_to_ds_pixel_coords(ds_pixel_coords1, cam1to2_full_mat)\n\n    # projected depth 2\n    depth1_wrt_f2 = ds_pixel_coords1_wrt_f2[..., -1:]\n\n    # inverse warp depth\n    depth2_warp_to_f1 = ivy_vision.image.bilinear_resample(depth2, warp)\n\n    # depth validity\n    depth_validity = ivy.abs(depth1_wrt_f2 - depth2_warp_to_f1) \u003c 0.01\n\n    # inverse warp rendering with mask\n    color2_warp_to_f1_masked = ivy.where(depth_validity, color2_warp_to_f1, ivy.zeros_like(color2_warp_to_f1))\n\nAgain, visualizations of these images are given below.\nThe images represent intermediate steps for the inverse warping of color from frame 2 to frame 1,\nwhich is shown in the bottom right corner.\n\n.. image:: https://github.com/unifyai/vision/blob/main/docs/images/inverse_warped.png?raw=true\n   :width: 100%\n\nFor forward warping, we instead assume depth to be known in the source frame.\nA common approach is to construct a mesh, and then perform rasterization of the mesh.\n\nThe ivy method ``ivy_vision.render_pixel_coords`` instead takes a simpler approach,\nby determining the pixel projections into the target frame,\nquantizing these to integer pixel co-ordinates,\nand scattering the corresponding color values directly into these integer pixel co-ordinates.\n\nThis process in general leads to holes and duplicates in the resultant image,\nbut when compared to inverse warping,\nit has the beneft that the target frame does not need to correspond to a real camera with known depth.\nOnly the target camera geometry is required, which can be for any hypothetical camera.\n\nWe now consider the case of forward warping the color data from camera frame 2 to camera frame 1,\nand again render the new color image in target frame 1.\n\n.. code-block:: python\n\n    # forward warp rendering\n    ds_pixel_coords1_proj = ivy_vision.ds_pixel_to_ds_pixel_coords(\n        ds_pixel_coords2, ivy.inv(cam1to2_full_mat_homo)[..., 0:3, :])\n    depth1_proj = ds_pixel_coords1_proj[..., -1:]\n    ds_pixel_coords1_proj = ds_pixel_coords1_proj[..., 0:2] / depth1_proj\n    features_to_render = ivy.concat((depth1_proj, color2), axis=-1)\n\n    # without depth buffer\n    f1_forward_warp_no_db, _, _ = ivy_vision.quantize_to_image(\n        ivy.reshape(ds_pixel_coords1_proj, (-1, 2)), img_dims, ivy.reshape(features_to_render, (-1, 4)),\n        ivy.zeros_like(features_to_render), with_db=False)\n\n    # with depth buffer\n    f1_forward_warp_w_db, _, _ = ivy_vision.quantize_to_image(\n        ivy.reshape(ds_pixel_coords1_proj, (-1, 2)), img_dims, ivy.reshape(features_to_render, (-1, 4)),\n        ivy.zeros_like(features_to_render), with_db=False if ivy.get_framework() == 'mxnet' else True)\n\nAgain, visualizations of these images are given below.\nThe images show the forward warping of both depth and color from frame 2 to frame 1,\nwhich are shown with and without depth buffers in the right-hand and central columns respectively.\n\n.. image:: https://github.com/unifyai/vision/blob/main/docs/images/forward_warped.png?raw=true\n   :width: 100%\n\nInteractive Demos\n-----------------\n\nIn addition to the examples above, we provide two further demo scripts,\nwhich are more visual and interactive, and are each built around a particular function.\n\nRather than presenting the code here, we show visualizations of the demos.\nThe scripts for these demos can be found in the interactive_ demos folder.\n\n**Neural Rendering**\n\nThe first demo uses method ``ivy_vision.render_implicit_features_and_depth``\nto train a Neural Radiance Field (NeRF) model to encode a lego digger. The NeRF model can then be queried at new camera\nposes to render new images from poses unseen during training.\n\n.. raw:: html\n\n    \u003cp align=\"center\"\u003e\n        \u003cimg width=\"50%\" style=\"display: block;\" src='https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/ivy_vision/nerf_demo.gif?raw=true'\u003e\n    \u003c/p\u003e\n\n**Co-ordinates to Voxel Grid**\n\nThe second demo captures depth and color images from a set of cameras,\nconverts the depth to world-centric co-ordinartes,\nand uses the method ``ivy_vision.coords_to_voxel_grid`` to\nvoxelize the depth and color values into a grid, as shown below:\n\n.. raw:: html\n\n    \u003cp align=\"center\"\u003e\n        \u003cimg width=\"75%\" style=\"display: block;\" src='https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/ivy_vision/voxel_grid_demo.gif?raw=true'\u003e\n    \u003c/p\u003e\n\n**Point Rendering**\n\nThe final demo again captures depth and color images from a set of cameras,\nbut this time uses the method ``ivy_vision.quantize_to_image`` to\ndynamically forward warp and point render the images into a new target frame, as shown below.\nThe acquiring cameras all remain static, while the target frame for point rendering moves freely.\n\n.. raw:: html\n\n    \u003cp align=\"center\"\u003e\n        \u003cimg width=\"75%\" style=\"display: block;\" src='https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/ivy_vision/point_render_demo.gif?raw=true'\u003e\n    \u003c/p\u003e\n\nGet Involved\n------------\n\nWe hope the functions in this library are useful to a wide range of machine learning developers.\nHowever, there are many more areas of 3D vision which could be covered by this library.\n\nIf there are any particular vision functions you feel are missing,\nand your needs are not met by the functions currently on offer,\nthen we are very happy to accept pull requests!\n\nWe look forward to working with the community on expanding and improving the Ivy vision library.\n\nCitation\n--------\n\n::\n\n    @article{lenton2021ivy,\n      title={Ivy: Templated deep learning for inter-framework portability},\n      author={Lenton, Daniel and Pardo, Fabio and Falck, Fabian and James, Stephen and Clark, Ronald},\n      journal={arXiv preprint arXiv:2102.02886},\n      year={2021}\n    }\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivy-llc%2Fvision","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivy-llc%2Fvision","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivy-llc%2Fvision/lists"}