{"id":22632197,"url":"https://github.com/alleninstitute/deepinterpolation","last_synced_at":"2025-04-05T21:07:50.719Z","repository":{"id":43792725,"uuid":"304488783","full_name":"AllenInstitute/deepinterpolation","owner":"AllenInstitute","description":null,"archived":false,"fork":false,"pushed_at":"2024-02-15T01:03:35.000Z","size":190971,"stargazers_count":203,"open_issues_count":22,"forks_count":49,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-04-27T23:35:39.790Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AllenInstitute.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-16T01:33:40.000Z","updated_at":"2024-04-27T23:35:43.849Z","dependencies_parsed_at":"2024-04-27T23:45:46.796Z","dependency_job_id":null,"html_url":"https://github.com/AllenInstitute/deepinterpolation","commit_stats":{"total_commits":364,"total_committers":5,"mean_commits":72.8,"dds":0.08241758241758246,"last_synced_commit":"25333467409a9359b076f392d01a61bffcb8ed4d"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AllenInstitute%2Fdeepinterpolation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AllenInstitute%2Fdeepinterpolation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AllenInstitute%2Fdeepinterpolation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AllenInstitute%2Fdeepinterpolation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AllenInstitute","download_url":"https://codeload.github.com/AllenInstitute/deepinterpolation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247399877,"owners_count":20932876,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-09T02:16:47.286Z","updated_at":"2025-04-05T21:07:50.695Z","avatar_url":"https://github.com/AllenInstitute.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":".. image:: https://circleci.com/gh/AllenInstitute/deepinterpolation.svg?style=svg\n    :target: https://circleci.com/gh/AllenInstitute/deepinterpolation\n\n\n# **Deep Interpolation**\n\n\n*deepinterpolation* is a Python library to denoise data by removing independent noise. Importantly training does **NOT** require ground truth. This repository is currently meant to support the publication : https://www.nature.com/articles/s41592-021-01285-2\n\n\n# **Principle of Deep Interpolation**\n\n\n.. image:: /docs/principle.png\n    :alt: principle of deep interpolation\n    :width: 700 px\n\t\n**Figure 1** - Schematic introducing the principles of deep interpolation.  **A**. An interpolation model is trained to predict a noisy block from other blocks with independent noise. The loss is the difference between the predicted data and a new noisy block. **B**. The interpolation model is used to create a noiseless version of the input data. \n\nFor more information, consult the associated publication : https://www.nature.com/articles/s41592-021-01285-2\n\n\n# **Support**\n\nFor bug and issues, please submit issue tickets on this repository. For installation and running support, we are trying to move to using the more public discussion forum on this repository (https://github.com/AllenInstitute/deepinterpolation/discussions). Alternatively you can join the slack channel where the past support history was saved (if invitation has expired: email to Jerome): https://join.slack.com/t/deepinterpolation/shared_invite/zt-rkmcw7h1-v8y0Grwe3fZg4m~DiAQVMg\n\n# **Installation**\n\nIn all cases, unless you only want to work from CPU, you will have to consider installing tensorflow GPU dependencies (ie. cuda drivers). To that end, you might have to consult tensorflow documentation to enable your GPU. \n\nTo install the package, you have 2 options. \n\n1. Install from pypi using: \n\nCreate new conda environment called 'local_env'\n\n\tconda create -n local_env python=3.7\n    \nOur integration tests on the CI server are currently running with python 3.7. While it is likely working with other versions, we cannot guarantee it. \n\n\tpip install deepinterpolation\n\nThis will install the latest deployed stable version and only the core components of the library. You will NOT have access to sample datasets present on this repository. \n\n2. Install from a clone of this repository. \n\nThis will give you access to the latest developments as well as the provided sample data. Our step by step example assume this installation mode as it depends on the sample datasets. \n\nThe small training examples below works on both CPU and GPU architecture (ie. even a small macbook). If you are not familiar with using deep learning, we recommend to play with smaller datasets first, such as the example Neuropixel data provided. \n\n* Clone the repository locally on a directory 'local_dir'\n\n\tgit clone https://github.com/AllenInstitute/deepinterpolation.git\n\n* Go to that directory\n\n\tcd 'local_dir'\n\n* Create new conda environment called 'local_env'\n\n\tconda create -n local_env python=3.7\n\nOur integration tests on the CI server are currently running with python 3.7. While it is likely working with other versions, we cannot guarantee it. \n\n* activate environment\n\n\tconda activate local_env\n\n* install necessary packages\n\n\tmake init\n\n* install deepinterpolation package\n\n\tpython setup.py install\n\n# **General package description**\n\nThe files in the deepinterpolation folder contain the core classes for training, inference, loss calculation and network generations. Those are called 'Collection'. Each collection is essentially a local list of functions that are used to create different type of objects and can be extended on one another. \nFor instance, the network_collection.py contains a list of networks that can be generated for training. This allows for quick iteration and modification of an architecture while keeping the code organized. \n\n# **FAQ**\n\nSee here : https://github.com/AllenInstitute/deepinterpolation/tree/master/faq\n\n# **Example training**\n\nTo try out training your own DeepInterpolation network, we recommend to start with this file: https://github.com/AllenInstitute/deepinterpolation/blob/master/examples/example_tiny_ephys_training.py\n\nIn this file, you will need to edit the paths to a local folder appropriate to save your models. \n\nThen, activate your conda env called 'local_env'\n\n\tconda activate local_env\n\t\nthen run\n\n\tpython example_tiny_ephys_training.py\n\nIf everything runs correctly, you should see the following in just a few minutes :\n\n\t2020-10-19 18:01:03.735098: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n\tTo enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n\tsh: sysctl: command not found\n\t2020-10-19 18:01:03.749184: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f9b1f115860 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n\t2020-10-19 18:01:03.749202: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n\tWARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of batches seen.\n\tEpoch 1/5\n\t10/10 [==============================] - 19s 2s/step - loss: 0.4597 - val_loss: 0.3987\n\tEpoch 2/5\n\t10/10 [==============================] - 20s 2s/step - loss: 0.3796 - val_loss: 0.3785\n\tEpoch 3/5\n\t10/10 [==============================] - 22s 2s/step - loss: 0.3646 - val_loss: 0.3709\n\tEpoch 4/5\n\t10/10 [==============================] - 21s 2s/step - loss: 0.3797 - val_loss: 0.3698\n\tEpoch 5/5\n\t10/10 [==============================] - 21s 2s/step - loss: 0.3835 - val_loss: 0.3675\n\tSaved model to disk\n\nThis is a toy example but you can increase the number of training frames to increase the quality of the model. \nAll parameters are commented in the file. To adjust to a larger dataset, change the path parameters, the start_frame and end_frame parameters.\nPlease consult the annotated example for more details of each parameter.\n\n# **Example inference**\n\nRaw pre-trained models are available as separate h5 file to download. \n\nThe following models are currently available : \n\n*Two-photon Ai93 excitatory line DeepInterpolation network:*\n\nKey recording parameters: \n\n- 30Hz sampling rate, 400x400 μm2 field of view, 512x512 pixels.\n- 0.8 NA objective.\n- 910 nm excitation wavelength.\n- Gcamp6f calcium indicator.\n- Ai93 reporter line expressed in excitatory neurons.\n- Docker hub id : 245412653747/deep_interpolation:allen_400um_512pix_30hz_ai93\n- Download link : https://alleninstitute.sharepoint.com/:u:/s/ExternalSharing/EdFnzUm4lV5LkG28Hp27cdwB4CFrCh37hEC-TxvsSURwwA?e=6BcyQl\n- Training data : https://github.com/AllenInstitute/deepinterpolation/blob/cfb42bc2c5bd8cc08ee870a163230e8c89c676ba/examples/paper_generation_code/json_data/2019-09-05-train-very-large-single-plane-Ai93-norm.json\n*Two-photon Ai148 excitatory line DeepInterpolation network:*\n\nKey recording parameters: \n\n- 30 Hz sampling rate, 400x400 μm2 field of view, 512x512 pixels.\n- 0.8 NA objective.\n- 910 nm excitation wavelength.\n- Gcamp6f calcium indicator.\n- Ai148 reporter line expressed in excitatory neurons.\n- Pre-processing: Individual movies were motion corrected. Each movie recording was mean-centered and normalized with a single pair of value for all pixels \n- Docker hub id : 245412653747/deep_interpolation:allen_400um_512pix_30hz_ai148\n- Download link : https://alleninstitute.sharepoint.com/:u:/s/ExternalSharing/EdhA7EyIr5hFjf2QyvTZIuMB-0dmbzCFTtKrnHPh9wgADQ?e=nrIMKW\n- Training data : https://github.com/AllenInstitute/deepinterpolation/blob/cfb42bc2c5bd8cc08ee870a163230e8c89c676ba/examples/paper_generation_code/json_data/2019-09-05-train-very-large-single-plane-Ai148-norm.json\n\n*Neuropixel DeepInterpolation network:*\n\nKey recording parameters: \n\n- Neuropixels Phase 3a probes\n- 374 simultaneous recording sites across 3.84 mm, 10 reference channels\n- Four-column checkerboard site layout with 20 µm spacing between rows\n- 30 kHz sampling rate\n- 500x hardware gain setting\n- 500 Hz high pass filter in hardware, 150 Hz high-pass filter applied offline. \n- Pre-processing: Median subtraction was applied to individual probes to remove signals that were common across all recording sites. Each probe recording was mean-centered and normalized with a single pair of value for all nodes on the probe. \n- Docker hub id : 245412653747/deep_interpolation:allen_neuropixel\n- Download link : https://alleninstitute.sharepoint.com/:u:/s/ExternalSharing/EQgayR4bgk5Oj5j2RdWZywMBHdMhueENyqXyce0F_jM6og?e=wBr71e\n\n*fMRI DeepInterpolation network:*\n\nKey recording parameters: \n\n- TR, 3000 ms; TE, 30 ms; flip angle, 80°; voxel size, 3 × 3 × 3 mm; FOV, 192 × 192 mm; number of slices, 50, slice gap, 0 mm\n- Pre-processing: N/A\n- Docker hub id : 245412653747/deep_interpolation:allen_3_3_3_tr_3000_fmri\n- Download link : https://alleninstitute.sharepoint.com/:u:/s/ExternalSharing/EZLfzUtd3WRMjbhk0g_rIr8BGPV-kHsOQ4j8Fmhc0guikw?e=Iua1TO\n\nTo start inference, we recommend to start with this file: https://github.com/AllenInstitute/deepinterpolation/blob/master/examples/example_tiny_ephys_inference.py\n\nIn this file, you will need to edit the paths strings to fit your local paths. \n\nThen, activate your conda env called 'local_env'\n\n\tconda activate local_env\n\t\nthen run:\n\n\tpython example_tiny_ephys_inference.py\n\nIf everything runs correctly, you should see the following in just a few minutes:\n\n\t2020-10-20 14:10:37.549061: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n\tTo enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n\tsh: sysctl: command not found\n\t2020-10-20 14:10:37.564133: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f82ada8a520 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n\t2020-10-20 14:10:37.564156: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n\nThis is a toy example but you can increase the start_frame and end_frame variable for larger data. \n\nIt is important to keep in mind that this process is easily parallelizable. In practice, we wrapped this code with additional routines to leverage 20 to 100 cluster CPU nodes to accelerate this process. You could also use GPU nodes as well, we just had access to a much larger number of CPU machines quickly.  \n\n# **Adapting the module to a newer data structure**\n\nTo adapt DeepInterpolation to a new dataset, you will need to use or recreate a generator in 'generator_collection.py'. Those are all constructed from core classes called *DeepGenerator* and *SequentialGenerator*. \n\nThe *CollectorGenerator* class allows to group generators if your dataset is distributed across many files/folder/sources. \nThis system was designed to allow to train very large DeepInterpolation models from TB of data distributed on a network infrastructure. \n\n# **License**\n\nAllen Institute Software License – This software license is the 2-clause BSD \nlicense plus clause a third clause that prohibits redistribution and use for \ncommercial purposes without further permission. \n\nCopyright © 2019. Allen Institute.  All rights reserved.\n\nRedistribution and use in source and binary forms, with or without \nmodification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this \nlist of conditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice, \nthis list of conditions and the following disclaimer in the documentation \nand/or other materials provided with the distribution.\n\n3. Redistributions and use for commercial purposes are not permitted without \nthe Allen Institute’s written permission. For purposes of this license, \ncommercial purposes are the incorporation of the Allen Institute's software \ninto anything for which you will charge fees or other compensation or use of \nthe software to perform a commercial service for a third party. Contact \nterms@alleninstitute.org for commercial licensing opportunities.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND \nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED \nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE \nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE \nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL \nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR \nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER \nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, \nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE \nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falleninstitute%2Fdeepinterpolation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falleninstitute%2Fdeepinterpolation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falleninstitute%2Fdeepinterpolation/lists"}