{"id":18785252,"url":"https://github.com/dbouget/validation_metrics_computation","last_synced_at":"2025-10-24T01:57:26.607Z","repository":{"id":96785302,"uuid":"409902884","full_name":"dbouget/validation_metrics_computation","owner":"dbouget","description":"Validation and metrics computation over 3D medical volumes (backend for Raidionics)","archived":false,"fork":false,"pushed_at":"2025-08-06T12:59:06.000Z","size":636,"stargazers_count":6,"open_issues_count":1,"forks_count":5,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-08T11:53:08.227Z","etag":null,"topics":["cross-validation","metrics","raidionics","segmentation"],"latest_commit_sha":null,"homepage":"https://raidionics.github.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dbouget.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-09-24T09:13:56.000Z","updated_at":"2025-08-06T12:59:09.000Z","dependencies_parsed_at":"2024-11-07T20:46:59.891Z","dependency_job_id":"c8103d2a-d702-43fe-a8b0-87aad020f28e","html_url":"https://github.com/dbouget/validation_metrics_computation","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/dbouget/validation_metrics_computation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbouget%2Fvalidation_metrics_computation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbouget%2Fvalidation_metrics_computation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbouget%2Fvalidation_metrics_computation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbouget%2Fvalidation_metrics_computation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dbouget","download_url":"https://codeload.github.com/dbouget/validation_metrics_computation/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbouget%2Fvalidation_metrics_computation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280723928,"owners_count":26380108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-23T02:00:06.710Z","response_time":142,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cross-validation","metrics","raidionics","segmentation"],"created_at":"2024-11-07T20:46:04.745Z","updated_at":"2025-10-24T01:57:26.596Z","avatar_url":"https://github.com/dbouget.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Segmentation validation and metrics computation backend for Raidionics related publications\n\n[![License](https://img.shields.io/badge/License-BSD%202--Clause-orange.svg)](https://opensource.org/licenses/BSD-2-Clause)\n[![](https://img.shields.io/badge/python-3.9|3.10|3.11|3.12|3.13-blue.svg)](https://www.python.org/downloads/)\n[![Paper](https://zenodo.org/badge/DOI/10.3389/fneur.2022.932219.svg)](https://www.frontiersin.org/articles/10.3389/fneur.2022.932219/full)\n[![codecov](https://codecov.io/gh/dbouget/validation_metrics_computation/branch/master/graph/badge.svg?token=ZSPQVR7RKX)](https://codecov.io/gh/dbouget/validation_metrics_computation)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/dbouget/491b0d34e3df00e730cd7fe7a8989202/compute_validation_example.ipynb)\n[![PyPI version](https://img.shields.io/pypi/v/raidionicsval.svg)](https://pypi.org/project/raidionicsval/)\n\nThe code corresponds to the Raidionics backend for running the k-fold cross-validation and metrics computation.\nThe module can either be used as a Python library, as CLI, or as Docker container. It supports both 2D and 3D inputs,\nthe only hard requirement is the expected folder structure to use as input.  \n:warning: For using custom structures, modifying the code [here](https://github.com/dbouget/validation_metrics_computation/blob/master/raidionicsval/Validation/kfold_model_validation.py#L155) is a good place to start.\n\n## [Installation](https://github.com/dbouget/validation_metrics_computation#installation)\n\n```\npip install raidionicsval\nor \npip install git+https://github.com/dbouget/validation_metrics_computation.git\n```\n\n## [Getting started](https://github.com/dbouget/validation_metrics_computation#getting-started)\n\n### [Notebooks](https://github.com/dbouget/validation_metrics_computation#notebooks)\n\nBelow are two Jupyter Notebooks which include simple examples on how to get started.\n\n\u003cdiv style=\"display: flex;\"\u003e\n  \u003cdiv style=\"flex: 1; margin-right: 20px;\"\u003e\n\n| Notebook       | Colab                                                | GitHub                                                                                                                   |\n|----------------|-------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------|\n| **Validation** | \u003ca href=\"https://colab.research.google.com/gist/dbouget/491b0d34e3df00e730cd7fe7a8989202/compute_validation_example.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e | [![View on GitHub](https://img.shields.io/badge/View%20on%20GitHub-blue?logo=github)](https://github.com/dbouget/validation_metrics_computation/blob/master/notebooks/compute_validation_example.ipynb) |\n| **Study**      | \u003ca href=\"https://colab.research.google.com/gist/dbouget/ccf77f31ac4ef58bb61d0808eaa9f454/compute_study_example.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e   | [![View on GitHub](https://img.shields.io/badge/View%20on%20GitHub-blue?logo=github)](https://github.com/dbouget/validation_metrics_computation/blob/master/notebooks/compute_study_example.ipynb)      |\n\n  \u003c/div\u003e\n\u003c/div\u003e\n\n### [Usage](https://github.com/dbouget/validation_metrics_computation#usage)\n\nIn the following, a description of how the data should be organized on disk is provided, and a test dataset can\nbe downloaded [here](https://github.com/raidionics/Raidionics-models/releases/download/v1.3.0-rc/Samples-RaidionicsValLib_UnitTest1-v1.1.zip).\n\n\u003cdetails\u003e\n\u003csummary\u003e\n\n### [1. Folder and data structures and naming conventions](https://github.com/dbouget/validation_metrics_computation#1-folder-and-data-structures-and-naming-conventions)\n\u003c/summary\u003e\n\nTwo main structure types are supported, without or without following an index-based naming convention.\nAssuming in the following example that the data indexes are based on their origin, but\nanything should work. The folders named _index0_ and _index1_ could be renamed to any sets of strings.\n\nThe metrics and overall validation can be computed for multiple segmentation classes at the same time, granted that\nunique and name-matching sets of files (i.e., ground truth and prediction files) are provided.\n\n#### [1.1 Original data folder structure](https://github.com/dbouget/validation_metrics_computation#11-original-data-folder-structure)\nThe main data directory containing the original 3D volumes and corresponding manual annotations is expected\nto resemble the following structure using an index-based naming convention:\n\n    └── path/to/data/root/\n        └── index0/\n            ├── Pat001/\n            │   ├── volumes/\n            │   │   └── Pat001_MRI.nii.gz\n            │   └── segmentations/\n            │   │   ├── Pat001_MRI_label_tumor.nii.gz\n            │   │   └── Pat001_MRI_label_other.nii.gz\n            ├── Pat025/\n            └── Pat050/\n        └── index1/\n            ├── Pat100/\n            └── Pat150/\n\nThe main data directory containing the original 3D volumes and corresponding manual annotations is expected\nto resemble the following structure when **not** using an index-based naming convention:\n\n    └── path/to/data/root/\n        └── Pat001/\n        │   ├── Pat001_MRI.nii.gz\n        │   ├── Pat001_MRI_label_tumor.nii.gz\n        │   └── Pat001_MRI_label_other.nii.gz\n        └── Pat010/\n        │   ├── Pat010_MRI.nii.gz\n        │   ├── Pat010_MRI_label_tumor.nii.gz\n        │   └── Pat010_MRI_label_other.nii.gz\n        [...]\n        └── Pat100/\n        │   ├── Pat100_MRI.nii.gz\n        │   ├── Pat100_MRI_label_tumor.nii.gz\n        │   └── Pat100_MRI_label_other.nii.gz\n\n#### [1.2 Inference results folder structure](https://github.com/dbouget/validation_metrics_computation#12-inference-results-folder-structure)\nPredictions results are expected to be stored inside a _predictions/_ sub-folder, the outer-most sub-folder \nnaming convention inside the folder are the fold numbers.\nThe inference results should be grouped inside what will become the validation folder, resembling the following\nstructure when using an index-based naming convention.\n\n    └── path/to/validation/study/\n        └── predictions/\n            ├── 0/\n            │   ├── index0_Pat001/\n            │   │   ├── Pat001_MRI-pred_tumor.nii.gz\n            │   │   └── Pat001_MRI-pred_other.nii.gz  \n            │   ├── index0_Pat002/\n            │   │   ├── Pat002_MRI-pred_tumor.nii.gz\n            │   │   └── Pat002_MRI-pred_other.nii.gz  \n            └── 1/\n            │   ├── index1_Pat100/\n            │   │   ├── Pat100_MRI-pred_tumor.nii.gz\n            │   │   └── Pat100_MRI-pred_other.nii.gz  \n            │   └── index1_Pat150/ \n            │   │   ├── Pat150_MRI-pred_tumor.nii.gz\n            │   │   └── Pat150_MRI-pred_other.nii.gz  \n\nThe inference results should be grouped inside what will become the validation folder, resembling the following\nstructure when **not** using an index-based naming convention.\n\n    └── path/to/validation/study/\n        └── predictions/\n            ├── 0/\n            │   ├── Pat001/\n            │   │   ├── Pat001_MRI-pred_tumor.nii.gz\n            │   │   └── Pat001_MRI-pred_other.nii.gz  \n            │   ├── Pat002/\n            │   │   ├── Pat002_MRI-pred_tumor.nii.gz\n            │   │   └── Pat002_MRI-pred_other.nii.gz  \n            └── 1/\n            │   ├── Pat100/\n            │   │   ├── Pat100_MRI-pred_tumor.nii.gz\n            │   │   └── Pat100_MRI-pred_other.nii.gz  \n            │   └── Pat150/ \n            │   │   ├── Pat150_MRI-pred_tumor.nii.gz\n            │   │   └── Pat150_MRI-pred_other.nii.gz  \n\n#### [1.3 Folds file](https://github.com/dbouget/validation_metrics_computation#13-folds-file)\nThe file with patients' distribution within each fold used for training should list\nthe content of the validation and test sets iteratively.  \nThe file should be called __cross\\_validation\\_folds.txt__ and placed in the validation\nstudy folder side-by-side with the _predictions_ sub-folder.  \n\nAn example of its content is given below when using an index-based naming convention:\n```\n  index0_Pat1000_MRI_sample index1_Pat1250_MRI_sample\\n    \n  index0_Pat001_MRI_sample index1_Pat025_MRI_sample\\n  \n  index0_Pat001_MRI_sample index1_Pat025_MRI_sample\\n    \n  index0_Pat100_MRI_sample index1_Pat150_MRI_sample\\n  \n```\n\nAn example of its content is given below when **not** using an index-based naming convention:\n```\n  Pat001_MRI Pat002_MRI\\n    \n  Pat100_MRI Pat150_MRI\\n  \n  Pat100_MRI Pat150_MRI\\n    \n  Pat200_MRI Pat250_MRI\\n  \n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\n\n### [2. Installation](https://github.com/dbouget/validation_metrics_computation#2-installation)\n\u003c/summary\u003e\nCreate a virtual environment using at least Python 3.8, and install the library.\n\n```\n  cd /path/to/validation_metrics_computation  \n  virtualenv -p python3 venv  \n  source venv/bin/activate  \n  TMPDIR=$PWD/venv pip install --cache-dir=$PWD/venv -e .\n```\n\nThen the final step is to do the following in a terminal.\n```\n  cd /path/to/validation_metrics_computation  \n  cp blank_main_config.ini main_config.ini \n```\n\nYou can now edit your __main\\_config.ini__ file for running the different processes.  \nAn additional explanation of all parameters specified in the configuration file can be\nfound in _/Utils/resources.py_. \n\n\u003c/details\u003e\n \n\u003cdetails\u003e\n\u003csummary\u003e\n\n### [3. Process](https://github.com/dbouget/validation_metrics_computation#3-process)\n\u003c/summary\u003e\n\nAfter filling in the configuration file specifying all runtime parameters,\naccording to the pattern from [**blank_main_config.ini**](https://github.com/dbouget/validation_metrics_computation/blob/master/blank_main_config.ini),\nyou should run first the __validation__ task and then the __study__ task.  \n\n\n#### [CLI](https://github.com/dbouget/validation_metrics_computation#cli)\n```\nraidionicsval -c CONFIG (-v debug)\n```\n\nCONFIG should point to a configuration file (*.ini).\n\n#### [Python module](https://github.com/dbouget/validation_metrics_computation#python-module)\n```\nfrom raidionicsval import compute\ncompute(config_filename=\"/path/to/main_config.ini\")\n```\n\n\"/path/to/main_config.ini\" should point to a valid configuration file.\n\n#### [Docker](https://github.com/dbouget/validation_metrics_computation#docker)\nWhen calling Docker images, the --user flag must be properly used in order for the folders and files created inside\nthe container to inherit the proper read/write permissions. The user ID is retrieved on-the-fly in the following\nexamples, but it can be given in a more hard-coded fashion if known by the user.\n\n```\ndocker pull dbouget/raidionics-val:v1.1.1-py39-cpu\n```\n\nFor opening the Docker image and interacting with it, run:  \n```\ndocker run --entrypoint /bin/bash -v /home/\u003cusername\u003e/\u003cresources_path\u003e:/workspace/resources -t -i --network=host --ipc=host --user $(id -u) dbouget/raidionics-val:v1.1.1-py39-cpu\n```\n\nThe `/home/\u003cusername\u003e/\u003cresources_path\u003e` before the column sign has to be changed to match a directory on your local \nmachine containing the data to expose to the docker image. Namely, it must contain folder(s) with data to use as input\nfor the validation studies, and it will contain the destination folder where the results will be saved.\n\nFor launching the Docker image as a CLI, run:  \n```\ndocker run -v /home/\u003cusername\u003e/\u003cresources_path\u003e:/workspace/resources -t -i --network=host --ipc=host --user $(id -u) dbouget/raidionics-val:v1.1.1-py39-cpu -c /workspace/resources/\u003cpath\u003e/\u003cto\u003e/main_config.ini -v \u003cverbose\u003e\n```\n\nThe `\u003cpath\u003e/\u003cto\u003e/main_config.ini` must point to a valid configuration file on your machine, as a relative path to the `/home/\u003cusername\u003e/\u003cresources_path\u003e` described above.\nFor example, if the file is located on my machine under `/home/myuser/Data/Validation/main_config.ini`, \nand that `/home/myuser/Data` is the mounted resources partition mounted on the Docker image, the new relative path will be `Validation/main_config.ini`.  \nThe `\u003cverbose\u003e` level can be selected from [debug, info, warning, error].\n\n\u003c/details\u003e\n\n## [How to cite](https://github.com/dbouget/validation_metrics_computation#how-to-cite)\n\nIf you are using Raidionics in your research, please cite the following references.\n\nFor segmentation validation and metrics computation:\n```\n@article{bouget2022preoptumorseg,\n    title={Preoperative Brain Tumor Imaging: Models and Software for Segmentation and Standardized Reporting},\n    author={Bouget, David and Pedersen, André and Jakola, Asgeir S. and Kavouridis, Vasileios and Emblem, Kyrre E. and Eijgelaar, Roelant S. and Kommers, Ivar and Ardon, Hilko and Barkhof, Frederik and Bello, Lorenzo and Berger, Mitchel S. and Conti Nibali, Marco and Furtner, Julia and Hervey-Jumper, Shawn and Idema, Albert J. S. and Kiesel, Barbara and Kloet, Alfred and Mandonnet, Emmanuel and Müller, Domenique M. J. and Robe, Pierre A. and Rossi, Marco and Sciortino, Tommaso and Van den Brink, Wimar A. and Wagemakers, Michiel and Widhalm, Georg and Witte, Marnix G. and Zwinderman, Aeilko H. and De Witt Hamer, Philip C. and Solheim, Ole and Reinertsen, Ingerid},\n    journal={Frontiers in Neurology},\n    volume={13},\n    year={2022},\n    url={https://www.frontiersin.org/articles/10.3389/fneur.2022.932219},\n    doi={10.3389/fneur.2022.932219},\n    issn={1664-2295}\n}\n```\n\nThe final software including updated performance metrics for preoperative tumors and introducing postoperative tumor segmentation:\n```\n@article{bouget2023raidionics,\n    author = {Bouget, David and Alsinan, Demah and Gaitan, Valeria and Holden Helland, Ragnhild and Pedersen, André and Solheim, Ole and Reinertsen, Ingerid},\n    year = {2023},\n    month = {09},\n    pages = {},\n    title = {Raidionics: an open software for pre-and postoperative central nervous system tumor segmentation and standardized reporting},\n    volume = {13},\n    journal = {Scientific Reports},\n    doi = {10.1038/s41598-023-42048-7},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdbouget%2Fvalidation_metrics_computation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdbouget%2Fvalidation_metrics_computation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdbouget%2Fvalidation_metrics_computation/lists"}