https://github.com/lren-chuv/airflow-imaging-plugins
Set of plugins helping to work with imaging data in Airflow.
https://github.com/lren-chuv/airflow-imaging-plugins
airflow data-factory mri neuroimaging neuroscience spm
Last synced: about 1 month ago
JSON representation
Set of plugins helping to work with imaging data in Airflow.
- Host: GitHub
- URL: https://github.com/lren-chuv/airflow-imaging-plugins
- Owner: LREN-CHUV
- License: apache-2.0
- Created: 2016-11-28T17:46:03.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2024-07-10T04:58:37.000Z (11 months ago)
- Last Synced: 2025-04-19T15:40:36.773Z (about 2 months ago)
- Topics: airflow, data-factory, mri, neuroimaging, neuroscience, spm
- Language: Python
- Size: 257 KB
- Stars: 15
- Watchers: 4
- Forks: 2
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://www.unil.ch/lren/en/home.html) [](https://github.com/LREN-CHUV/airflow-imaging-plugins/blob/master/LICENSE) [](https://www.codacy.com/app/hbp-mip/airflow-imaging-plugins?utm_source=github.com&utm_medium=referral&utm_content=LREN-CHUV/airflow-imaging-plugins&utm_campaign=Badge_Grade) [](https://pypi.python.org/pypi/airflow-imaging-plugins/) [](https://circleci.com/gh/HBPMedical/airflow-imaging-plugins)
# Airflow Imaging plugins
Airflow plugins providing support for preprocessing of neuroimaging data.
The following packages are provided:
* airflow_freespace: Sensors that check the amount of free space on the disk and waits until enough free space is available
* airflow_pipeline: Operators and helpers to build generic processing pipelines
* airflow_scan_folder: Operators used to scan folders for new work
* airflow_spm: Operators adapting Matlab and [SPM 12](http://www.fil.ion.ucl.ac.uk/spm) to work inside Airflow
## Usage
Imaging data is organised by folders, where each fist-level folder represents a scanning session.
A 'pipeline' represents the steps to process a folder containing one scanning session.
In Airflow, we use the XCOM mechanism to transmit data from one step of the pipeline to the next step.
This is why each processing pipelines need to start with airflow_pipeline.operators.PreparePipelineOperator as it injects into XCOM the necessary information that is required for the other \*PipelineOperator:
All Python callback functions provided to those operators can use as arguments the following variables coming from XCOM:
* folder
* session_id
* participant_id (optional)
* scan_date (optional)
* output
* error
* dataset
* matlab_version
* spm_version
* spm_revision
* provenance_details
* provenance_previous_step_id
* relative_context_path
See airflow_pipeline.pipelines.PIPELINE_XCOMS for an up-to-date list
## List of plugins
* airflow_spm.operators.SpmOperator: Executes SPM or just Matlab
* airflow_spm.operators.SpmPipelineOperator: Executes a pipeline on SPM, where a 'pipeline' is a function implemented in SPM
* airflow_freespace.operators.FreeSpaceSensor: Waits for enough free disk space on the disk.
* airflow_scan_folder.operators.ScanFlatFolderOperator: Triggers a DAG run for a specified ``dag_id`` for each scan folder discovered in a folder.
* airflow_scan_folder.operators.ScanDailyFolderOperator: Triggers a DAG run for a specified ``dag_id`` for each scan folder discovered in a daily folder.
* airflow_scan_folder.operators.ScanFlatFolderPipelineOperator: Triggers a DAG run for a specified ``dag_id`` for each folder discovered in a parent folder, where the parent folder location is provided by the pipeline XCOMs.
* airflow_pipeline.operators.PreparePipelineOperator: An operator that prepares the pipeline
* airflow_pipeline.operators.PythonPipelineOperator: A PythonOperator that moves XCOM data used by the pipeline
* airflow_pipeline.operators.BashPipelineOperator: A BashOperator that registers provenance information in the pipeline
* airflow_pipeline.operators.DockerPipelineOperator: A DockerOperator that registers provenance information in the pipeline
Python version: 3
## Installation
```
pip3 install from git: pip3 install git+https://github.com/LREN-CHUV/airflow-imaging-plugins.git@master#egg=airflow_imaging_plugins
```
## Setup and configuration
### Airflow setup for MRI scans pipeline:
* In Airflow config file, add the [spm] section with the following entries:
* SPM_DIR: root path to the installation of SPM
## build
Run `./build.sh`.
## Push on PyPi
Run `./publish.sh`.
(This builds the project prior to pushing it).
# Acknowledgements
This work has been funded by the European Union Seventh Framework Program (FP7/20072013) under grant agreement no. 604102 (HBP)
This work is part of SP8 of the Human Brain Project (SGA1).