{"id":20956548,"url":"https://github.com/bluebrain/atlas-interpolation","last_synced_at":"2025-05-14T05:32:02.013Z","repository":{"id":39801788,"uuid":"417129668","full_name":"BlueBrain/atlas-interpolation","owner":"BlueBrain","description":"Interpolate missing section images in gene expression volumes","archived":false,"fork":false,"pushed_at":"2023-12-21T09:40:09.000Z","size":904,"stargazers_count":2,"open_issues_count":6,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-18T12:10:04.109Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://atlas-interpolation.rtfd.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BlueBrain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.txt"}},"created_at":"2021-10-14T12:54:07.000Z","updated_at":"2024-02-29T03:49:18.000Z","dependencies_parsed_at":"2023-12-21T11:54:52.448Z","dependency_job_id":null,"html_url":"https://github.com/BlueBrain/atlas-interpolation","commit_stats":{"total_commits":41,"total_committers":5,"mean_commits":8.2,"dds":0.7317073170731707,"last_synced_commit":"5458bdc376d7c825ada4f1563e917d75e30081d5"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlueBrain%2Fatlas-interpolation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlueBrain%2Fatlas-interpolation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlueBrain%2Fatlas-interpolation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlueBrain%2Fatlas-interpolation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BlueBrain","download_url":"https://codeload.github.com/BlueBrain/atlas-interpolation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254076977,"owners_count":22010630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T01:26:38.564Z","updated_at":"2025-05-14T05:32:01.187Z","avatar_url":"https://github.com/BlueBrain.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Atlas Interpolation\n\n\u003cimg src=\"docs/_images/Blue-Brain-Atlas-Interpolation-Github-banner.jpg\"/\u003e\n\nThe Allen Brain Institute hosts a rich database of mouse brain imagery. It\ncontains a large number of gene expression datasets obtained\nthrough the in situ hybridization (ISH) staining. While for a given gene\na number of datasets corresponding to different specimen can be found, each of\nthese datasets only contains sparse section images that do not form a\ncontinuous volume. This package explores techniques that allow to interpolate\nthe missing slices and thus reconstruct whole gene expression volumes.\n\n* [Installation](#installation)\n    * [Python Version and Environment](#python-version-and-environment)\n    * [Install \"Atlas Interpolation\"](#install-atlas-interpolation)\n* [Data](#data)\n    * [Remote Storage Access](#remote-storage-access) \n    * [Model Checkpoints](#model-checkpoints)\n    * [Section Images and Datasets](#section-images-and-datasets)\n    * [New ISH datasets (advanced, optional)](#new-ish-datasets-advanced-optional)\n* [Examples](#examples)\n    * [Pair Interpolation](#pair-interpolation)\n    * [Optical Flow Models](#optical-flow-models)\n    * [Predict an Entire Gene Volume (longer runtime)](#predict-an-entire-gene-volume-longer-runtime)\n* [Vendors](#vendors)\n* [Funding \u0026 Acknowledgment](#funding--acknowledgment)\n\n## Installation\n### Python Version and Environment\nNote that due to some of our dependencies we're currently limited to python\nversion `3.7`. Please make sure you set up a virtual environment with that\nversion before trying to install this library. If you're unsure how to do that\nplease have a look at [conda](https://docs.conda.io) or\n[pyenv](https://github.com/pyenv/pyenv).\n\nIf you are part of the Blue Brain Project and are working on the BB5 you can\nfind the correct python version in the archive modules between `archive/2020-02`\nand `archive/2020-12` (inclusive). Here's an example of a set of commands\nthat will set up your environment on the BB5:\n```shell\nmodule purge\nmodule load archive/2020-12\nmodule load python\npython -m venv venv\n. ./venv/bin/activate\npython --version\n```\n\nWe also recommend that you make sure that `pip` is up-to-date and that the\npackages `wheel` and `setuptools` are installed:\n```shell\npip install --upgrade pip wheel setuptools\n```\n\n### Install \"Atlas Interpolation\"\nIn order to access the data and the example scripts a local clone of this\nrepository is required. Run these commands to get it:\n```shell\ngit clone https://github.com/BlueBrain/atlas-interpolation\ncd atlas-interpolation\n```\n\nThe \"Atlas Interpolation\" package can now be installed directly from the clone\nwe just created:\n```shell\npip install '.[data, optical]'\n```\n\n## Data\nThe data for this project is managed by the [DVC tool](https://dvc.org/) and all\nrelated files are located in the `data` directory. The DVC tool has already been\ninstalled together with the \"Atlas Interpolation\" package. Every time you need\nto run a DVC command (`dvc ...`) make sure to change to the `data` directory\nfirst (`cd data`).\n\n### Remote Storage Access\nWe have already prepared all the data, but it is located on a remote storage\nthat is only accessible to people within the Blue Brain Project who have\naccess permissions to project `proj101`. If you're unsure you can test your\npermissions with the following command:\n```shell\nssh bbpv1.bbp.epfl.ch \\\n\"ls /gpfs/bbp.cscs.ch/data/project/proj101/dvc_remotes\"\n```\nPossible outcomes:\n```shell\n# Access OK\natlas_annotation\natlas_interpolation\n\n# Access denied\nls: cannot open directory [...]: Permission denied\n```\nDepending on whether you have access to the remote storage in the following\nsections you will either pull the data from the remote (`dvc pull`) or download\nthe input data manually and re-run the data processing pipelines to reproduce\nthe output data (`dvc repro`).\n\nIf you work on the BB5 and have access to the remote storage then run the\nfollowing command to short-circuit the remote access (because the remote is\nlocated on the BB5 itself):\n```shell\ncd data\ndvc remote add --local gpfs_proj101 \\\n  /gpfs/bbp.cscs.ch/data/project/proj101/dvc_remotes/atlas_interpolation\ncd ..\n```\n\n### Model Checkpoints\nMuch of the functionality of \"Atlas Interpolation\" relies on pre-trained deep\nlearning models. The model checkpoints that need to be loaded are part of the\ndata.\n\nIf you have access to the remote storage (see above) you can pull all model\ncheckpoints from the remote:\n```shell\ncd data\ndvc pull checkpoints/rife.dvc\ndvc pull checkpoints/cain.dvc\ndvc pull checkpoints/maskflownet.params.dvc\ndvc pull checkpoints/RAFT.dvc\ncd ..\n```\n\nIf you don't have access to the remote you need to download the checkpoint files\nby hand and put the downloaded data into the `data/checkpoints` folder. You\nmay not need all the checkpoints depending on the examples you want to run. Here\nare the instructions for the four models we use: RIFE, CAIN, MaskFlowNet, and\nRAFT:\n* **RIFE**: download the checkpoint from a shared Google Drive folder by following\n  [this link](https://drive.google.com/file/d/11l8zknO1V5hapv2-Ke4DG9mHyBomS0Fc/view?usp=sharing).\n  Unzip the contents of the downloaded file into `data/checkpoints/rife`.\n  [[ref]](https://github.com/hzwer/arXiv2020-RIFE/tree/2a1eafe27d5ff12eb31df96e47352fe30c18ac46#usage)\n* **CAIN**: download the checkpoint from a shared Dropbox folder by following\n  [this link](https://www.dropbox.com/s/y1xf46m2cbwk7yf/pretrained_cain.pth?dl=0).\n  Move the downloaded file to `data/checkpoints/cain`.\n  [[ref]](https://github.com/myungsub/CAIN/tree/2e727d2a07d3f1061f17e2edaa47a7fb3f7e62c5#interpolating-with-custom-video)\n* **MaskFlowNet**: download the checkpoint directly from GitHub by following\n  [this link](https://github.com/microsoft/MaskFlownet/raw/5cba12772e2201f0d1c1e27161d224e585334571/weights/8caNov12-1532_300000.params).\n  Rename the file to `maskflownet.params` and move it to `data/checkpoints`.\n  [[ref]](https://github.com/microsoft/MaskFlownet/raw/5cba12772e2201f0d1c1e27161d224e585334571/weights)\n* **RAFT**: download the checkpoint files from a shared Dropbox folder by following\n  [this link](https://drive.google.com/drive/folders/1sWDsfuZ3Up38EUQt7-JDTT1HcGHuJgvT?usp=sharing).\n  Move all downloaded `.pth` files to the `data/checkpoints/RAFT/models` folder.\n  [[ref]](https://github.com/princeton-vl/RAFT/tree/224320502d66c356d88e6c712f38129e60661e80#demos)\n\nIf you downloaded all checkpoints or pulled them from the remote you should\nhave the following files:\n```text\ndata\n└── checkpoints\n    ├── RAFT\n    │   ├── models\n    │   │   ├── raft-chairs.pth\n    │   │   ├── raft-kitti.pth\n    │   │   ├── raft-sintel.pth\n    │   │   ├── raft-small.pth\n    │   │   └── raft-things.pth\n    ├── cain\n    │   └── pretrained_cain.pth\n    ├── maskflownet.params\n    └── rife\n        ├── contextnet.pkl\n        ├── flownet.pkl\n        └── unet.pkl\n```\n\n### Section Images and Datasets\nThe purpose of the \"Atlas Interpolation\" package is to interpolate missing\nsection images within section image datasets. This section explains how to\nobtain these data.\n\nRemember that if you don't have access to the remote storage (see above) you'll\nneed to use the `dvc repro` commands that download/process the data live. If\nyou do have access, you'll use `dvc pull` instead, which is faster.\n\nNormally it's not necessary to get all data. Due to its size it may take a lot\nof disk space as well as time to download and pre-process. If you still decide\nto do so you can by running `dvc repro` or `dvc pull` without any parameters.\n\nSpecific examples only require specific data. You can use DVC to list all data\npipeline stages to find out which stage produces the data you're interested in.\nTo list all data pipeline stages run:\n```shell\ncd data\ndvc stage list\n```\nIf, for example, you need data located in `data/aligned/coronal/Gad1`, then\naccording to the output of command above the relevant stage is named\n`align@Gad1`. Therefore, you only need to run this stage to get the necessary\ndata (replace `repro` by `pull` if you can access the remote storage):\n```shell\ndvc repro align@Gad1\n```\n\n### New ISH datasets (advanced, optional)\nIf you're familiar with the AIBS data that we're using and would like to add\nnew ISH gene expressions that are not yet available as one of our pipeline\nstages (check the output of `dvc stage list`) then follow the following\ninstructions.\n\n1. Edit the file `data/dvc.yaml` and add the new gene name to the lists in the\n   `stages:download_dataset:foreach` and `stages:align:foreach` sections.\n2. Run the data downloading and processing pipelines (replace `NEW_GENE` by the\n   real gene name that you used in `data/dvc.yaml`):\n   ```shell\n   dvc repro download_dataset@NEW_GENE\n   dvc repro align@NEW_GENE\n   ```\n\n## Examples\nIn this section we showcase several typical use-cases of \"Atlas Interpolation\":\n- Use pair interpolation to predict an intermediate image between two given\n  images\n- Predict optical flow between any pair of images and use it to morph a third\n  image\n- In a gene expression volume predict missing slices and reconstruct the whole\n  volume\n\nNote that all models accept both RGB images (`shape=(height, width, 3)`)\nand grayscale images (`shape=(height, width)`).\n\n### Pair Interpolation\nThe only data you need for this example is the RIFE model checkpoint. Follow\nthe instructions in the corresponding section above to get it. If you have\naccess to the remote data storage it's enough to run the following commands:\n```shell\ncd data\ndvc pull checkpoints/rife.dvc\ncd ..\n```\n\nIn this example we start with a pair of images `img1` and `img2` (randomly\ngenerated for example's sake). First use the RIFE model to interpolate between\nthem in a manual way and find the image in-between (`img_middle`). Then we\ndemonstrate the use of the `PairInterpolate` class that streamlines the\ninterpolation procedure. Starting with the same pair of images we iterate the\ninterpolation three times to produce a stack of seven interpolated images\n(`interpolated_imgs`).\n```python\nimport numpy as np\n\nfrom atlinter.vendor.rife.RIFE_HD import Model as RifeModel\nfrom atlinter.vendor.rife.RIFE_HD import device as rife_device\nfrom atlinter.pair_interpolation import PairInterpolate, RIFEPairInterpolationModel\n\n# Get the input images\nimg1 = np.random.rand(100, 200, 3) # replace by real section image\nimg2 = np.random.rand(100, 200, 3) # replace by real section image\n\n# Get the RIFE interpolation model\ncheckpoint_path = \"data/checkpoints/rife/\" # Please change, if needed\nrife_model = RifeModel()\nrife_model.load_model(checkpoint_path, -1)\nrife_model.eval()\ninterpolation_model = RIFEPairInterpolationModel(rife_model, rife_device)\n\n# Manually predict middle image between img1 and img2\npreimg1, preimg2 = interpolation_model.before_interpolation(img1=img1, img2=img2)\nimg_middle = interpolation_model.interpolate(img1=preimg1, img2=preimg2)\nimg_middle = interpolation_model.after_interpolation(img_middle)\nprint(img_middle.shape)\n\n# Streamline the interpolation using PairInterpolate and predict a stack\n# of 7 intermediate images\ninterpolator = PairInterpolate(n_repeat=3)\ninterpolated_imgs = interpolator(img1, img2, interpolation_model)\nprint(interpolated_imgs.shape)\n``` \n\n### Optical Flow Models\nThe only data you need for this example is the MaskFlowNet model checkpoint.\nFollow the instructions in the corresponding section above to get it. If you\nhave access to the remote data storage it's enough to run the following\ncommands:\n```shell\ncd data\ndvc pull checkpoints/maskflownet.params.dvc\ncd ..\n```\n\nThis example demonstrates how an optical flow model can be used to compute the\noptical flow between a pair of images. It can then be used to warp a third\nimage. The images in this example are randomly generated. In a realistic setting\nthey should be replaced by real images.\n```python\nimport numpy as np\n\nfrom atlinter.optical_flow import MaskFlowNet\n\n# Instantiate an optical flow model (in this case: MaskFlowNet)\ncheckpoint_path = \"data/checkpoints/maskflownet.params\"\nnet = MaskFlowNet(checkpoint_path)\n\n# Prepare random images. Should be replaced by real section images\nimg1 = np.random.rand(100, 200, 3)\nimg2 = np.random.rand(100, 200, 3)\nimg3 = np.random.rand(100, 200, 3)\n\n# Predict the optical flow between img1 and img2\nimg1, img2 = net.preprocess_images(img1=img1, img2=img2)\npredicted_flow = net.predict_flow(img1=img1, img2=img2)\n\n# Warp a third image using the optical flow\npredicted_img = net.warp_image(predicted_flow, img3)\nprint(predicted_img.shape)\n``` \n\n### Predict an Entire Gene Volume (Longer Runtime)\nThe data you need for this example are the RIFE model checkpoint and the Vip\ngene expression dataset. To get the RIFE checkpoint follow the instruction in\nthe corresponding section above. If you have access to the remote data storage\nit's enough to run the following commands:\n```shell\ncd data\ndvc pull checkpoints/rife.dvc\ncd ..\n```\nAs described in the data section above, there are two ways of getting the Vip\ngene expression dataset. If you have access to the remote data storage you can\npull it from there:\n```shell\ncd data\ndvc pull download_dataset@Vip\ncd ..\n```\nIf you don't have access then you can re-download it. This should always work,\nbut may take several minutes:\n```shell\ncd data\ndvc repro download_dataset@Vip\ncd ..\n```\n\nIn this example with start with a gene expression volume that has missing\nsection images. First we load the image data and the metadata from disk and\nwrap it into a `GeneDataset` class. Then we instantiate the RIFE deep learning\nmodel that will be used for interpolation. We use this model to first predict a\nsingle slice in the volume, then we reconstruct the whole volume by predicting\nall intermediate slices. Note that this last step is computation-heavy and might\ntherefore take some time.\n```python\nimport json\n\nimport numpy as np\n\nfrom atlinter.data import GeneDataset\nfrom atlinter.pair_interpolation import GeneInterpolate, RIFEPairInterpolationModel\nfrom atlinter.vendor.rife.RIFE_HD import Model as RifeModel\nfrom atlinter.vendor.rife.RIFE_HD import device as rife_device\n\n# Load the gene expression dataset from disk\ndata_path = \"data/sagittal/Vip/1102.npy\"  # Change the path if needed\ndata_json = \"data/sagittal/Vip/1102.json\" # Change the path if needed\nsection_images = np.load(data_path)\nwith open(data_json) as fh:\n    metadata = json.load(fh)\n\nsection_numbers = [int(s) for s in metadata[\"section_numbers\"]]\naxis = metadata[\"axis\"]\n\n# Wrap the data into a GeneDataset class\ngene_dataset = GeneDataset(\n  section_images,\n  section_numbers,\n  volume_shape=(528, 320, 456, 3),\n  axis=axis,\n)\n\n# Load the RIFE deep learning model that will be used for interpolation\ncheckpoint_path = \"data/checkpoints/rife\"\nrife_model = RifeModel()\nrife_model.load_model(checkpoint_path, -1)\nrife_model.eval()\nrife_interpolation_model = RIFEPairInterpolationModel(rife_model, rife_device)\n\n# Create a gene interpolator\ngene_interpolate = GeneInterpolate(gene_dataset, rife_interpolation_model)\n\n# Predict a single section image\npredicted_slice = gene_interpolate.predict_slice(10)\nprint(predicted_slice.shape)\n\n# Reconstruct the whole volume. This might take some time.\npredicted_volume = gene_interpolate.predict_volume()\nprint(predicted_volume.shape)\n```\n\n## Vendors\nSome dependencies are not available as packages and therefore had to be\nvendored. The vendoring is done using the\n[`py-vendor`](https://pypi.org/project/py-vendor/) utility. It's installed\nautomatically together with the `dev` extras. You can also install it by hand\nvia `pip install py-vendor==0.1.2`.\n\nThe vendoring is then done using the following command (add `--force` to\noverwrite existing folders):\n```shell\npy-vendor run --config py-vendor.yaml\n```\nSee the `py-vendor.yaml` file for details on the vendor sources and files.\n\n## Funding \u0026 Acknowledgment\n\nThe development of this software was supported by funding to the Blue Brain Project,\na research center of the École polytechnique fédérale de Lausanne (EPFL),\nfrom the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.\n\nCopyright (c) 2021-2022 Blue Brain Project/EPFL\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluebrain%2Fatlas-interpolation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbluebrain%2Fatlas-interpolation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluebrain%2Fatlas-interpolation/lists"}