{"id":13745456,"url":"https://github.com/hsokooti/RegNet","last_synced_at":"2025-05-09T06:30:43.401Z","repository":{"id":201253960,"uuid":"116586770","full_name":"hsokooti/RegNet","owner":"hsokooti","description":"Nonrigid image registration using multi-scale 3D convolutional neural networks","archived":false,"fork":false,"pushed_at":"2024-06-21T13:47:23.000Z","size":2811,"stargazers_count":192,"open_issues_count":11,"forks_count":50,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-11-15T18:38:04.987Z","etag":null,"topics":["convolutional-neural-networks","image-registration","machine-learning","medical-image-processing","tensorflow"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hsokooti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-ApacheV2","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-07T17:52:19.000Z","updated_at":"2024-09-14T08:03:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"70830e7e-14f5-46f4-869c-7c11c793edfd","html_url":"https://github.com/hsokooti/RegNet","commit_stats":{"total_commits":88,"total_committers":2,"mean_commits":44.0,"dds":"0.022727272727272707","last_synced_commit":"28a8b6132677bb58e9fc811c0dd15d78913c7e86"},"previous_names":["hsokooti/regnet"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsokooti%2FRegNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsokooti%2FRegNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsokooti%2FRegNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsokooti%2FRegNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hsokooti","download_url":"https://codeload.github.com/hsokooti/RegNet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253205850,"owners_count":21871158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convolutional-neural-networks","image-registration","machine-learning","medical-image-processing","tensorflow"],"created_at":"2024-08-03T06:00:18.950Z","updated_at":"2025-05-09T06:30:43.017Z","avatar_url":"https://github.com/hsokooti.png","language":"Python","funding_links":[],"categories":["Toolbox","Registration"],"sub_categories":["Medical Image"],"readme":"\n\nRegNet\n==========\n\n## Introduction\nIn this work we propose a method to solve nonrigid image registration through a learning approach, instead of via iterative optimization of a predefined dissimilarity metric. We design a Convolutional Neural Network (CNN) architecture that, in contrast to all other work, directly estimates the displacement vector field (DVF) from a pair of input images. The proposed RegNet is trained using a large set of artificially generated DVFs, does not explicitly define a dissimilarity metric, and integrates image content at multiple scales to equip the network with contextual information. At testing time nonrigid registration is performed in a single shot, in contrast to current iterative methods.\n\n### Citation\n[1] [Hessam Sokooti](https://scholar.google.com/citations?user=cTizuxcAAAAJ), [Bob de Vos](https://scholar.google.nl/citations?user=Q0KDwp8AAAAJ), [Floris Berendsen](https://scholar.google.nl/citations?user=x-_btzsAAAAJ), [Mohsen Ghafoorian](https://scholar.google.nl/citations?user=989WL-wAAAAJ), [Sahar Yousefi](https://scholar.google.nl/citations?user=gOuDXuIAAAAJ),  [Boudewijn P.F. Lelieveldt](https://scholar.google.nl/citations?user=J20kK1oAAAAJ), [Ivana Išgum](https://scholar.google.nl/citations?user=xelGbyIAAAAJ) and [Marius Staring](https://scholar.google.nl/citations?user=pKFkfq4AAAAJ), 2019. [3D Convolutional Neural Networks Image Registration Based on Efficient Supervised Learning from Artificial Deformations](https://arxiv.org/abs/1908.10235). arXiv preprint arXiv:1908.10235.\n\n[2] [Hessam Sokooti](https://scholar.google.com/citations?user=cTizuxcAAAAJ), [Bob de Vos](https://scholar.google.nl/citations?user=Q0KDwp8AAAAJ), [Floris Berendsen](https://scholar.google.nl/citations?user=x-_btzsAAAAJ), [Boudewijn P.F. Lelieveldt](https://scholar.google.nl/citations?user=J20kK1oAAAAJ), [Ivana Išgum](https://scholar.google.nl/citations?user=xelGbyIAAAAJ), and [Marius Staring](https://scholar.google.nl/citations?user=pKFkfq4AAAAJ), 2017, September. [Nonrigid image registration using multi-scale 3D convolutional neural networks](https://elastix.lumc.nl/marius/downloads/2017_c_MICCAIa.pdf). In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 232-239). Springer, Cham.\n\n\t\n## 1. Dependencies\n- [Joblib](http://github.com/joblib/joblib) : Running Python functions as pipeline jobs.\n- [Matplotlib](https://matplotlib.org/) A plotting library for the Python programming language and its numerical mathematics extension NumPy.\n- [NumPy](http://www.numpy.org/) : General purpose array-processing package.\n- [SimpleITK](http://www.simpleitk.org/) : Simplified interface to the Insight Toolkit for image registration and segmentation.\n- [SciPy](https://www.scipy.org/) : A Python-based ecosystem of open-source software for mathematics, science, and engineering.\n- [TensorFlow v1.x](https://www.tensorflow.org/) : TensorFlow helps the tensors flow.\n- [xmltodict](https://github.com/martinblech/xmltodict) : Python module that makes working with XML feel like you are working with JSON.\n\t\n\n## 2. Running RegNet\nRun`RegNet3D.py`. Please note that current RegNet only works with 3D images.\n\n### 2.1 Data\nAll images are read and written by [SimpleITK](http://www.simpleitk.org/). The images are already resampled to an isotropic voxel size of [1, 1, 1] mm.\n\nThe images in the training and validation set can be defined in a list of dictionaries: \n```python\n# simple example how to load the data:\nimport functions.setting.setting_utils as su\n\nsetting = su.initialize_setting(current_experiment='MyCurrentExperiment', where_to_run='Root')\ndata_exp_dict = [\n    {'data': 'DIR-Lab_4D',           # Data to load. The image addresses can be modified in setting_utils.py\n     'deform_exp': '3D_max7_D14_K',  # Synthetic deformation experiment\n     'TrainingCNList': [1, 2, 3],    # Case number of images to load (The patient number)\n     'TrainingTypeImList': [i for i in range(8)],    # Types images for each case number, for example [baseline, follow-up]\n     'TrainingDSmoothList': [i for i in range(14)],  # The synthetic type to load. For instance, ['single_frequency', 'mixed_frequency']\n     'ValidationCNList': [1, 2],\n     'ValidationTypeImList': [8, 9],\n     'ValidationDSmoothList': [0, 5, 10],\n     },\n    {'data': 'SPREAD',\n     'deform_exp': '3D_max7_D14_K',\n     'TrainingCNList': [i for i in range(1, 11)],\n     'TrainingTypeImList': [0, 1],\n     'TrainingDSmoothList': [i for i in range(14)],\n     'ValidationCNList': [11, 12],\n     'ValidationTypeImList': [0, 1],\n     'ValidationDSmoothList': [0, 5, 10],\n     },\n\n]\n\nsetting = su.load_setting_from_data_dict(setting, data_exp_dict)\noriginal_image_address = su.address_generator(setting, 'Im', data='DIR-Lab_4D', cn=1, type_im=0, stage=1)\nprint(original_image_address)\n\nim_info_list_training = su.get_im_info_list_from_train_mode(setting, 'Training', load_mode='Single', read_pair_mode='Synthetic', stage=1)\nim_info_list_training = im_info_list_training[0:4]\nprint('\\n The first four elements are: ')\nprint(*im_info_list_training, sep=\"\\n\")\n\nfor im_info in im_info_list_training:\n    im_info_su = {'data': im_info['data'], 'deform_exp': im_info['deform_exp'], 'type_im': im_info['type_im'],\n                  'cn': im_info['cn'], 'dsmooth': im_info['dsmooth'], 'stage': im_info['stage'], }\n    print(su.address_generator(setting, 'Im', **im_info_su))\n    print(su.address_generator(setting, 'DeformedIm', **im_info_su))\n\n\n```\n\n```\n./Data/DIR-Lab/4DCT/mha/case1/case1_T00_RS1.mha\n\n The first four elements are: \n{'data': 'DIR-Lab_4D', 'type_im': 0, 'cn': 1, 'deform_exp': '3D_max7_D14_K', 'dsmooth': 0, 'deform_method': 'respiratory_motion', 'deform_number': 0, 'stage': 1}\n{'data': 'DIR-Lab_4D', 'type_im': 0, 'cn': 1, 'deform_exp': '3D_max7_D14_K', 'dsmooth': 1, 'deform_method': 'respiratory_motion', 'deform_number': 1, 'stage': 1}\n{'data': 'DIR-Lab_4D', 'type_im': 0, 'cn': 1, 'deform_exp': '3D_max7_D14_K', 'dsmooth': 2, 'deform_method': 'respiratory_motion', 'deform_number': 2, 'stage': 1}\n{'data': 'DIR-Lab_4D', 'type_im': 0, 'cn': 1, 'deform_exp': '3D_max7_D14_K', 'dsmooth': 3, 'deform_method': 'respiratory_motion', 'deform_number': 3, 'stage': 1}\n./Data/DIR-Lab/4DCT/mha/case1/case1_T00_RS1.mha\n./Elastix/Artificial_Generation/3D_max7_D14_K/DIR-Lab_4D/T00/case1/Dsmooth0/respiratory_motion_D0/DeformedImage.mha\n./Elastix/Artificial_Generation/3D_max7_D14_K/DIR-Lab_4D/T00/case1/Dsmooth0/DNext1/NextIm.mha\n./Elastix/Artificial_Generation/3D_max7_D14_K/DIR-Lab_4D/T00/case1/Dsmooth1/respiratory_motion_D1/DeformedImage.mha\n./Elastix/Artificial_Generation/3D_max7_D14_K/DIR-Lab_4D/T00/case1/Dsmooth0/DNext2/NextIm.mha\n./Elastix/Artificial_Generation/3D_max7_D14_K/DIR-Lab_4D/T00/case1/Dsmooth2/respiratory_motion_D2/DeformedImage.mha\n./Elastix/Artificial_Generation/3D_max7_D14_K/DIR-Lab_4D/T00/case1/Dsmooth0/DNext3/NextIm.mha\n./Elastix/Artificial_Generation/3D_max7_D14_K/DIR-Lab_4D/T00/case1/Dsmooth3/respiratory_motion_D3/DeformedImage.mha\n```\n\n#### `'data'`: \nThe details of `'data'` should be written in the `setting.setting_utils.py`. The general setting of each `'data'` should be defined in \n`load_data_setting(selected_data)` like the extension, total number of types and default pixel value. The global data folder (`setting['DataFolder']`) can be defined in `root_address_generator(where_to_run='Auto')`. \n\nThe details of the image address can be defined in `setting.setting_utils.address_generator()` after the line `if data == 'YourOwnData':`. For example you can take a look at the line 370: `if data == 'DIR-Lab_4D':`. The orginal images are defined with `requested_address= 'OriginalIm'`. To test the reading function, you can run the above script and check the `original_image_address`.\n\n\n#### `'deform_exp', 'TrainingDSmoothList'`: \ncheck section 2.2.4 Setting of generating synthetic DVFs\n\n#### `'TrainingCNList', 'TrainingTypeImList'`: \n`'TrainingCNList'` indicates the Case Numbers (CN) that you want to use for training. Usually each cn refers to a specific patient. `'TrainingTypeImList'` indicates which types of the available images for each patient you want to load. For example in the SPREAD data, two types are available: baseline and follow-up. In the DIR-Lab_4D data, for each patient 10 images are available from the maximum inhale to maximum exhale phase.\n\n### 2.2 Setting of generating synthetic DVFs\nFour categories of synthetic DVF are available in the software: zero, single frequency, mixed frequency, respiratory motion\n#### 2.2.1 Zero `'zero'`\n#### 2.2.2 Single frequency `'single_frequency'`\nFor generating single-frequency DVF, we proposed the following algorithm:\n1. Initialize a B-spline grid points with a grid spacing of `deform_exp_setting['SingleFrequency_BSplineGridSpacing']`.\n2. Perturb the gird points in a smooth and random fashion.\n3. Interpolate to get the DVF.\n4. Normalize the DVF linearly, if it is out of the range `[-deform_exp_setting['MaxDeform'], +deform_exp_setting['MaxDeform']]`.\nBy varying the spacing, different spatial frequencies are generated.\n![alt text](Documentation/SyntheticDVF_SingleFreq.png \"Single Frequency\")\n\u003cp align=\"center\"\u003eFigure 1: Single Frequency: B-spline grid spacing are 40, 30 and 20 mm from left to right.\u003c/p\u003e\n\n#### 2.2.3 Mixed frequency `'mixed_frequency'`\n\nThe steps for the mixed-frequency category is as follows:\n1. Extract edges with Canny edge detection method.\n2. Copy the binary image three times to get a vector of 3D image with the length of three.\n3. Set some voxels to be zero randomly for each image. \n4. Dilate the binary image for `deform_exp_setting['MixedFrequency_Np']` iteration by using a random structure element for each image.\n5. Fill the binary dilated image with a DVF generated from the single-frequency method.\n6. Smooth the DVF with a Gaussian kernel with standard deviation of `deform_exp_setting['MixedFrequency_SigmaRange']`. The sigma is relatively small which leads to a higher spatial frequency in comparison with the filled DVF.\nBy varying the sigma value and `deform_exp_setting['MixedFrequency_BSplineGridSpacing']` in the filled DVF, different spatial frequencies will be mixed together.\n\n![alt text](Documentation/SyntheticDVF_MixedFreq.png \"Mixed Frequency\")\n\u003cp align=\"center\"\u003eFigure 2: Mixed Frequency.\u003c/p\u003e\n\n#### 2.2.4 `'deform_exp', 'TrainingDSmoothList'`\n`'deform_exp'` is defined in the `setting.artificial_generation_setting.py`. For example you can use multiple types of single frequency and mixed frequency:\n```python\ndef_setting['DeformMethods'] = ['respiratory_motion', 'respiratory_motion', 'respiratory_motion', 'respiratory_motion',\n                                 'single_frequency', 'single_frequency', 'single_frequency', 'single_frequency', 'single_frequency',\n                                 'mixed_frequency', 'mixed_frequency', 'mixed_frequency', 'mixed_frequency',\n                                 'zero']\n```\nThe above setting is at the generation time. However, you might not want to load all of them at the reading time.\n\n`'ValidationDSmoothList': [2, 4, 8]`: This means that you want to load translation type2, smoothBspline type1 and dilatedEdgeSmooth type 2.\n\n\n### 2.3 Network\nThe proposed networks are given in Figure 3, 4, 5.\n![alt text](Documentation/UNET1.png)\n\u003cp align=\"center\"\u003eFigure 3: unet1 (U-Net).\u003c/p\u003e\n\n![alt text](Documentation/Decimation4.png \"decimation4 (Multi-view)\")\n\u003cp align=\"center\"\u003eFigure 4: decimation4 (Multi-view).\u003c/p\u003e\n\n![alt text](Documentation/Crop4.png \"crop4 (U-Net advanced)\")\n\u003cp align=\"center\"\u003eFigure 5: crop4 (U-Net advanced).\u003c/p\u003e\n\n\n\n### 2.4 Software Architecture\n![alt text](Documentation/Software_Architecture2.PNG \"Software Architecture\")\n\u003cp align=\"center\"\u003eFigure 6: Software Architecture.\u003c/p\u003e\n\n#### 2.4.1 Memory efficiency\nIt is not efficient (or possible)  to load all images with their DVFs to the memory. A DVF is three times bigger than its corresponding image with the type of float32. Alternatively, this software loads a chunk of images.  The number of images per chunk can be chosen by the parameter: `setting['NetworkTraining']['NumberOfImagesPerChunk']`\n```python\nsetting['NetworkTraining']['NumberOfImagesPerChunk'] = 16  # Number of images that I would like to load in RAM\nsetting['NetworkTraining']['SamplesPerImage'] = 50\nsetting['NetworkTraining']['BatchSize'] = 15\nsetting['NetworkTraining']['MaxQueueSize'] = 20\n```\n\n#### 2.4.2 Parallel Computing\nWe used `threading` in order to read patches in parallel with training the network. The class `functions.reading.chunk_image.Images` is defined to read images with threading.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhsokooti%2FRegNet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhsokooti%2FRegNet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhsokooti%2FRegNet/lists"}