{"id":19138013,"url":"https://github.com/lemonpi/multidim_indexing","last_synced_at":"2025-07-21T16:03:47.788Z","repository":{"id":38031636,"uuid":"477920523","full_name":"LemonPi/multidim_indexing","owner":"LemonPi","description":"Batch multidimensional indexing for pytorch","archived":false,"fork":false,"pushed_at":"2024-07-29T20:49:30.000Z","size":88,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-10T19:01:06.346Z","etag":null,"topics":["arrays","batch","batch-processing","indexing","numpy","python","pytorch","tensor","voxel-grid","voxels"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LemonPi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-05T00:27:49.000Z","updated_at":"2024-07-29T20:49:33.000Z","dependencies_parsed_at":"2024-07-30T02:19:24.099Z","dependency_job_id":null,"html_url":"https://github.com/LemonPi/multidim_indexing","commit_stats":{"total_commits":31,"total_committers":1,"mean_commits":31.0,"dds":0.0,"last_synced_commit":"464b5f78d16a6bc0573ffceacdcc5ec77015c038"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/LemonPi/multidim_indexing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LemonPi%2Fmultidim_indexing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LemonPi%2Fmultidim_indexing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LemonPi%2Fmultidim_indexing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LemonPi%2Fmultidim_indexing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LemonPi","download_url":"https://codeload.github.com/LemonPi/multidim_indexing/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LemonPi%2Fmultidim_indexing/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266332369,"owners_count":23912660,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-21T11:47:31.412Z","response_time":64,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arrays","batch","batch-processing","indexing","numpy","python","pytorch","tensor","voxel-grid","voxels"],"created_at":"2024-11-09T06:41:26.251Z","updated_at":"2025-07-21T16:03:47.764Z","avatar_url":"https://github.com/LemonPi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"This repository documents the syntax for multidimensional indexing for Pytorch and Numpy, and offers classes that\nencapsulates the process and provides additional features on top for data that represents a coordinate grid.\nYou can follow along the code blocks here with the included Jupyter notebook.\n\n## Multidimensional Indexing\n\nSuppose we have a multidimensional tensor, which could be a cached voxel grid, or a batch of images\n(the values are ordered to make clear how the indexing works):\n\n```python\nimport torch\n\nB = 256  # batch size (optional)\nshape = (B, 64, 64)\nhigh = torch.prod(torch.tensor(shape)).to(dtype=torch.long)\ndata = torch.arange(0, high).reshape(shape)\n```\n\nA key operation on this tensor is to index it for querying and assignment. It is straightforward to index a single\nvalue, and particular groupings of dimensions:\n\n```python\n# index a single element\nprint(data[124, 5, 52])\n\n# index all dimensions given the first is index 0 (the following are equivalent)\nprint(data[0])\nprint(data[0, :, :])\nprint(data[0, ...])  # pytorch only syntax\n\n# index all dimensions given the last is index 5 (the following are equivalent)\nprint(data[..., 5])\nprint(data[:, :, 5])\n```\n\nIt is also straightforward to batch index along a single dimension:\n\n```python\nidx = [4, 8, 15, 16, 23, 42]\n\n# index all dimensions given the first follows idx\nprint(data[idx].shape)  # (len(idx), 64, 64)\nprint(data[idx, ...].shape)\nprint(data[idx, :, :].shape)\n\n# index all dimensions given the second follows idx\nprint(data[:, idx].shape)\nprint(data[:, idx, :].shape)\n```\n\nIt is also reasonable to batch index along multiple dimensions. Note that it does not make sense for `idx` and `idx2` to\nhave different lengths since that would lead to combinations where one is missing a value.\n\n```python\nidx = [4, 8, 15, 16, 23, 42]\nidx2 = [5, 2, 7, 1, 32, 4]\n\n# index the last dimension when the first two are (4,5), (8,2), (15,7), (16,1), (23,32), and (42,4)\nprint(data[idx, idx2].shape)  # (len(idx), 64)\n```\n\nIt is also common to have a list of entries by their indices that we'd like to batch query.\n\n```python\n# indices of 5 entries\nidx3 = [[0, 5, 3],\n        [2, 7, 5],\n        [100, 23, 45],\n        [3, 6, 4],\n        [4, 2, 1]]\n```\n\nDirectly indexing the tensor with a multidimensional index does not do what you want:\n\n```python\nprint(data[idx3])  # results in an error\n```\n\nInstead, **split up the indices by their dimension** either manually, or with `torch.unbind`\n\n```python\n# easier to convert it to something that allows column indexing first\nidx4 = torch.tensor(idx3)\nprint(data[idx4[:, 0], idx4[:, 1], idx4[:, 2]])  # returns the 5 entries as desired\nprint(data[torch.unbind(idx4, -1)])              # can also use unbind\n```\n\n## How can it be improved?\n\nMost importantly, it may not be clear why simply doing `data[idx3]` does not work, and what the correct syntax is. So\nreading up to here should resolve most questions about indexing with a batch of indices on a multidimensional tensor.\nThis library provides `MultidimView` variants (torch and numpy) that provide a view for these tensors with features\nspecialized to multidimensional tensor that represent coordinate gridded values:\n\n- direct indexing so `data[idx3]` does what you want\n- optional indexing on values if you specify value ranges\n    - value resolution implicitly defined by size of source and value range\n- optional safety checking for out of bound values or indices\n    - provide default value for out of bound queries instead of throwing an exception\n\n## Installation\nnumpy only\n```shell\npip install multidim-indexing[numpy]\n```\npytorch only\n```shell\npip install multidim-indexing[torch]\n```\nall\n```shell\npip install multidim-indexing[all]\n```\n## Usage\n\nContinuing with `data` and the indices described before,\n\n```python\nfrom multidim_indexing import torch_view as view\n\n# for numpy, import numpy_view and use NumpyMultidimView\n\n# simple wrapper with bounds checking\ndata_multi = view.TorchMultidimView(data)\n# another view into the data, treating it as a batch of 2 dimensional grid data with X in [-5, 5] and Y in [0, 10]\n# can specify value to assign a query if it's out of bounds (defaults to -1)\n# note that the invalid value needs to be of the same type as the source, so we can't for example use float('inf') here\ndata_batch = view.TorchMultidimView(data, value_ranges=[[0, B], [-5, 5], [0, 10]], invalid_value=-1)\n# another view into the data, treating it as a 3D grid data with X in [-2.5, 5], Y in [0, 4], and Z in [0, 10]\ndata_3d = view.TorchMultidimView(data, value_ranges=[[-2.5, 5], [0, 4], [0, 10]])\n```\nBy default, the nearest grid value is returned. You can instead use linear interpolation like scipy's interpn by setting\n`method='linear'` in the constructor.\n```python\ndata_3d = view.TorchMultidimView(data, value_ranges=[[-2.5, 5], [0, 4], [0, 10]], method='linear')\n```\n\nWe can then use them like:\n\n```python\n# convert index to the corresponding type (pytorch vs numpy)\nkey = torch.tensor(idx3, dtype=torch.long)\nprint(data_multi[key])  # returns the 5 entries as desired\n```\n\n```python\n# query the other views using grid values\n# first, let's try keying the same 2D values across all batches\nvalue_key_per_batch = torch.tensor([[-3.5, 0.2],\n                                    [-4, 0.1],\n                                    [-7, 0.5],  # this is out of bounds\n                                    [3, 2]])\n# number of entries to query\nN = value_key_per_batch.shape[0]\nprint(torch.arange(B, dtype=value_key_per_batch.dtype).reshape(B, 1, 1).repeat(1, N, 1).shape)\n# make the indices for all batches\nvalue_key_batch = torch.cat(\n    (torch.arange(B, dtype=value_key_per_batch.dtype).reshape(B, 1, 1).repeat(1, N, 1),\n     value_key_per_batch.repeat(B, 1, 1)), dim=-1)\n# keys can have an additional batch indices at the front\nprint(value_key_batch.shape)  # (B, N, 3)\n# these 2 should be the same apart from the first batch index\nprint(value_key_batch[0:N])\nprint(value_key_batch[12*N:13*N])\n\n# should see some -1 to indicate invalid value\nprint(data_batch[value_key_batch]) \n\n# also there is a shorthand for directly using the per batch indices\nprint(data_batch[value_key_per_batch.repeat(B,1,1)]) # should be the same as above\n```\n\n```python\nvalue_key_3d = torch.tensor([[-2.5, 0., 0.],  # right on the boundary of validity\n                             [-2.51, 0.5, 0],  # out of bounds\n                             [5, 4, 10]  # right on the boundary\n                             ]\n                            )\nprint(data_3d[value_key_3d])  # (0, -1 for invalid, high - 1)\nprint(torch.prod(torch.tensor(data.shape)) - 1)\nprint(high - 1)\n```\n\nThe indexing naturally allows setting in addition to querying. Out of bound indices will be\nignored.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemonpi%2Fmultidim_indexing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flemonpi%2Fmultidim_indexing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemonpi%2Fmultidim_indexing/lists"}