Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dbouget/validation_metrics_computation
Validation and metrics computation over 3D medical volumes (backend for Raidionics)
https://github.com/dbouget/validation_metrics_computation
cross-validation metrics raidionics segmentation
Last synced: 2 months ago
JSON representation
Validation and metrics computation over 3D medical volumes (backend for Raidionics)
- Host: GitHub
- URL: https://github.com/dbouget/validation_metrics_computation
- Owner: dbouget
- Created: 2021-09-24T09:13:56.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2024-02-09T12:44:43.000Z (12 months ago)
- Last Synced: 2024-05-21T06:14:22.354Z (8 months ago)
- Topics: cross-validation, metrics, raidionics, segmentation
- Language: Jupyter Notebook
- Homepage: https://raidionics.github.io
- Size: 502 KB
- Stars: 6
- Watchers: 2
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Segmentation validation and metrics computation backend for Raidionics related publications
[](https://opensource.org/licenses/BSD-2-Clause)
[](https://www.python.org/downloads/)
[](https://www.frontiersin.org/articles/10.3389/fneur.2022.932219/full)
[](https://colab.research.google.com/gist/dbouget/7560fe410db03e384a45ddc77bbe9a57/compute_validation_example.ipynb)
The code corresponds to the Raidionics backend for running the k-fold cross-validation and metrics computation.
The module can either be used as a Python library, as CLI, or as Docker container.
## [Installation](https://github.com/dbouget/validation_metrics_computation#installation)
```
pip install git+https://github.com/dbouget/validation_metrics_computation.git
```
## [Continuous integration](https://github.com/dbouget/validation_metrics_computation#continuous-integration)
| Operating System | Status |
|------------------|--------------------------------------------------------------------------------------------------------------------|
| **Windows** | [](https://github.com/dbouget/validation_metrics_computation/actions/workflows/build_windows.yml) |
| **Ubuntu** | [](https://github.com/dbouget/validation_metrics_computation/actions/workflows/build_ubuntu.yml) |
| **macOS** | [](https://github.com/dbouget/validation_metrics_computation/actions/workflows/build_macos.yml) |
| **macOS ARM** | [](https://github.com/dbouget/validation_metrics_computation/actions/workflows/build_macos_arm.yml) |
## [Getting started](https://github.com/dbouget/validation_metrics_computation#getting-started)
### [Notebooks](https://github.com/dbouget/validation_metrics_computation#notebooks)
Below are two Jupyter Notebooks which include simple examples on how to get started.
| Notebook | Colab | GitHub |
|----------------|-------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------|
| **Validation** | 
| [](https://github.com/dbouget/validation_metrics_computation/blob/master/notebooks/compute_validation_example.ipynb) |
| **Study** | | [](https://github.com/dbouget/validation_metrics_computation/blob/master/notebooks/compute_study_example.ipynb) |
### [Usage](https://github.com/dbouget/validation_metrics_computation#usage)
In the following, a description of how the data should be organized on disk is provided, and a test dataset can
be downloaded [here](https://github.com/raidionics/Raidionics-models/releases/download/1.2.0/Samples-RaidionicsValLib_UnitTest1.zip).
### [1. Folder and data structures and naming conventions](https://github.com/dbouget/validation_metrics_computation#1-folder-and-data-structures-and-naming-conventions)
Two main structure types are supported, without or without following an index-based naming convention.
Assuming in the following example that the data indexes are based on their origin, but
anything should work. The folders named _index0_ and _index1_ could be renamed to any sets of strings.
The metrics and overall validation can be computed for multiple segmentation classes at the same time, granted that
unique and name-matching sets of files (i.e., ground truth and prediction files) are provided.
#### [1.1 Original data folder structure](https://github.com/dbouget/validation_metrics_computation#11-original-data-folder-structure)
The main data directory containing the original 3D volumes and corresponding manual annotations is expected
to resemble the following structure using an index-based naming convention:
└── path/to/data/root/
└── index0/
├── Pat001/
│ ├── volumes/
│ │ └── Pat001_MRI.nii.gz
│ └── segmentations/
│ │ ├── Pat001_MRI_label_tumor.nii.gz
│ │ └── Pat001_MRI_label_other.nii.gz
├── Pat025/
└── Pat050/
└── index1/
├── Pat100/
└── Pat150/
The main data directory containing the original 3D volumes and corresponding manual annotations is expected
to resemble the following structure when **not** using an index-based naming convention:
└── path/to/data/root/
└── Pat001/
│ ├── Pat001_MRI.nii.gz
│ ├── Pat001_MRI_label_tumor.nii.gz
│ └── Pat001_MRI_label_other.nii.gz
└── Pat010/
│ ├── Pat010_MRI.nii.gz
│ ├── Pat010_MRI_label_tumor.nii.gz
│ └── Pat010_MRI_label_other.nii.gz
[...]
└── Pat100/
│ ├── Pat100_MRI.nii.gz
│ ├── Pat100_MRI_label_tumor.nii.gz
│ └── Pat100_MRI_label_other.nii.gz
#### [1.2 Inference results folder structure](https://github.com/dbouget/validation_metrics_computation#12-inference-results-folder-structure)
Predictions results are expected to be stored inside a _predictions/_ sub-folder, the outer-most sub-folder
naming convention inside the folder are the fold numbers.
The inference results should be grouped inside what will become the validation folder, resembling the following
structure when using an index-based naming convention.
└── path/to/validation/study/
└── predictions/
├── 0/
│ ├── index0_Pat001/
│ │ ├── Pat001_MRI-pred_tumor.nii.gz
│ │ └── Pat001_MRI-pred_other.nii.gz
│ ├── index0_Pat002/
│ │ ├── Pat002_MRI-pred_tumor.nii.gz
│ │ └── Pat002_MRI-pred_other.nii.gz
└── 1/
│ ├── index1_Pat100/
│ │ ├── Pat100_MRI-pred_tumor.nii.gz
│ │ └── Pat100_MRI-pred_other.nii.gz
│ └── index1_Pat150/
│ │ ├── Pat150_MRI-pred_tumor.nii.gz
│ │ └── Pat150_MRI-pred_other.nii.gz
The inference results should be grouped inside what will become the validation folder, resembling the following
structure when **not** using an index-based naming convention.
└── path/to/validation/study/
└── predictions/
├── 0/
│ ├── Pat001/
│ │ ├── Pat001_MRI-pred_tumor.nii.gz
│ │ └── Pat001_MRI-pred_other.nii.gz
│ ├── Pat002/
│ │ ├── Pat002_MRI-pred_tumor.nii.gz
│ │ └── Pat002_MRI-pred_other.nii.gz
└── 1/
│ ├── Pat100/
│ │ ├── Pat100_MRI-pred_tumor.nii.gz
│ │ └── Pat100_MRI-pred_other.nii.gz
│ └── Pat150/
│ │ ├── Pat150_MRI-pred_tumor.nii.gz
│ │ └── Pat150_MRI-pred_other.nii.gz
#### [1.3 Folds file](https://github.com/dbouget/validation_metrics_computation#13-folds-file)
The file with patients' distribution within each fold used for training should list
the content of the validation and test sets iteratively.
The file should be called __cross\_validation\_folds.txt__ and placed in the validation
study folder side-by-side with the _predictions_ sub-folder.
An example of its content is given below when using an index-based naming convention:
```
index0_Pat1000_MRI_sample index1_Pat1250_MRI_sample\n
index0_Pat001_MRI_sample index1_Pat025_MRI_sample\n
index0_Pat001_MRI_sample index1_Pat025_MRI_sample\n
index0_Pat100_MRI_sample index1_Pat150_MRI_sample\n
```
An example of its content is given below when **not** using an index-based naming convention:
```
Pat001_MRI Pat002_MRI\n
Pat100_MRI Pat150_MRI\n
Pat100_MRI Pat150_MRI\n
Pat200_MRI Pat250_MRI\n
```
### [2. Installation](https://github.com/dbouget/validation_metrics_computation#2-installation)
Create a virtual environment using at least Python 3.8, and install all dependencies from
the requirements.txt file.
```
cd /path/to/validation_metrics_computation
virtualenv -p python3 venv
source venv/bin/activate
TMPDIR=$PWD/venv pip install --cache-dir=$PWD/venv -r requirements.txt (--no-deps)
```
Then the final step is to do the following in a terminal.
```
cd /path/to/validation_metrics_computation
cp blank_main_config.ini main_config.ini
```
You can now edit your __main\_config.ini__ file for running the different processes.
An additional explanation of all parameters specified in the configuration file can be
found in _/Utils/resources.py_.
### [3. Process](https://github.com/dbouget/validation_metrics_computation#3-process)
After filling in the configuration file specifying all runtime parameters,
according to the pattern from [**blank_main_config.ini**](https://github.com/dbouget/validation_metrics_computation/blob/master/blank_main_config.ini),
you should run first the __validation__ task and then the __study__ task.
#### [CLI](https://github.com/dbouget/validation_metrics_computation#cli)
```
raidionicsval -c CONFIG (-v debug)
```
CONFIG should point to a configuration file (*.ini).
#### [Python module](https://github.com/dbouget/validation_metrics_computation#python-module)
```
from raidionicsval import compute
compute(config_filename="/path/to/main_config.ini")
```
"/path/to/main_config.ini" should point to a valid configuration file.
#### [Docker](https://github.com/dbouget/validation_metrics_computation#docker)
When calling Docker images, the --user flag must be properly used in order for the folders and files created inside
the container to inherit the proper read/write permissions. The user ID is retrieved on-the-fly in the following
examples, but it can be given in a more hard-coded fashion if known by the user.
```
docker pull dbouget/raidionics-val:v1.0-py38-cpu
```
For opening the Docker image and interacting with it, run:
```
docker run --entrypoint /bin/bash -v /home//:/workspace/resources -t -i --network=host --ipc=host --user $(id -u) dbouget/raidionics-val:v1.0-py38-cpu
```
The `/home//` before the column sign has to be changed to match a directory on your local
machine containing the data to expose to the docker image. Namely, it must contain folder(s) with data to use as input
for the validation studies, and it will contain the destination folder where the results will be saved.
For launching the Docker image as a CLI, run:
```
docker run -v /home//:/workspace/resources -t -i --network=host --ipc=host --user $(id -u) dbouget/raidionics-val:v1.0-py38-cpu -c /workspace/resources///main_config.ini -v
```
The `//main_config.ini` must point to a valid configuration file on your machine, as a relative path to the `/home//` described above.
For example, if the file is located on my machine under `/home/myuser/Data/Validation/main_config.ini`,
and that `/home/myuser/Data` is the mounted resources partition mounted on the Docker image, the new relative path will be `Validation/main_config.ini`.
The `` level can be selected from [debug, info, warning, error].
## [How to cite](https://github.com/dbouget/validation_metrics_computation#how-to-cite)
If you are using Raidionics in your research, please cite the following references.
For segmentation validation and metrics computation:
```
@article{bouget2022preoptumorseg,
title={Preoperative Brain Tumor Imaging: Models and Software for Segmentation and Standardized Reporting},
author={Bouget, David and Pedersen, André and Jakola, Asgeir S. and Kavouridis, Vasileios and Emblem, Kyrre E. and Eijgelaar, Roelant S. and Kommers, Ivar and Ardon, Hilko and Barkhof, Frederik and Bello, Lorenzo and Berger, Mitchel S. and Conti Nibali, Marco and Furtner, Julia and Hervey-Jumper, Shawn and Idema, Albert J. S. and Kiesel, Barbara and Kloet, Alfred and Mandonnet, Emmanuel and Müller, Domenique M. J. and Robe, Pierre A. and Rossi, Marco and Sciortino, Tommaso and Van den Brink, Wimar A. and Wagemakers, Michiel and Widhalm, Georg and Witte, Marnix G. and Zwinderman, Aeilko H. and De Witt Hamer, Philip C. and Solheim, Ole and Reinertsen, Ingerid},
journal={Frontiers in Neurology},
volume={13},
year={2022},
url={https://www.frontiersin.org/articles/10.3389/fneur.2022.932219},
doi={10.3389/fneur.2022.932219},
issn={1664-2295}
}
```
The final software including updated performance metrics for preoperative tumors and introducing postoperative tumor segmentation:
```
@article{bouget2023raidionics,
author = {Bouget, David and Alsinan, Demah and Gaitan, Valeria and Holden Helland, Ragnhild and Pedersen, André and Solheim, Ole and Reinertsen, Ingerid},
year = {2023},
month = {09},
pages = {},
title = {Raidionics: an open software for pre-and postoperative central nervous system tumor segmentation and standardized reporting},
volume = {13},
journal = {Scientific Reports},
doi = {10.1038/s41598-023-42048-7},
}
```