{"id":19733979,"url":"https://github.com/scaleoutsystems/digital-pathology-tutorial","last_synced_at":"2025-08-23T06:03:53.224Z","repository":{"id":44992148,"uuid":"240691132","full_name":"scaleoutsystems/digital-pathology-tutorial","owner":"scaleoutsystems","description":"Federated digital pathology:  classification of Acute Myeloid Leukemia (AML) cells with a CNN. ","archived":false,"fork":false,"pushed_at":"2023-05-10T07:07:10.000Z","size":64180,"stargazers_count":6,"open_issues_count":1,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-30T03:37:06.242Z","etag":null,"topics":["acute-myeloid-leukemia","cnn","computer-vision","convolutional-neural-networks","deep-learning","digitalpathology","fedn","keras-tensoflow","studio"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scaleoutsystems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-02-15T10:56:20.000Z","updated_at":"2023-12-12T09:55:33.000Z","dependencies_parsed_at":"2024-04-19T20:58:30.083Z","dependency_job_id":"069defe6-d776-45e6-851a-5c22a7358220","html_url":"https://github.com/scaleoutsystems/digital-pathology-tutorial","commit_stats":null,"previous_names":["scaleoutsystems/digital-pathology-tutorial"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/scaleoutsystems/digital-pathology-tutorial","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scaleoutsystems%2Fdigital-pathology-tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scaleoutsystems%2Fdigital-pathology-tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scaleoutsystems%2Fdigital-pathology-tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scaleoutsystems%2Fdigital-pathology-tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scaleoutsystems","download_url":"https://codeload.github.com/scaleoutsystems/digital-pathology-tutorial/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scaleoutsystems%2Fdigital-pathology-tutorial/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271745659,"owners_count":24813515,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acute-myeloid-leukemia","cnn","computer-vision","convolutional-neural-networks","deep-learning","digitalpathology","fedn","keras-tensoflow","studio"],"created_at":"2024-11-12T00:34:50.566Z","updated_at":"2025-08-23T06:03:53.196Z","avatar_url":"https://github.com/scaleoutsystems.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Acute Myeloid Leukemia classification using a federated Convolutional Neural Network \n\nThe purpose of this workshop is to explore the possibilities of federate learning (FL) through a realistic use-case in digital pathology. We will work with a scenario where a public research dataset is partitioned into a number of subsets, where each subset would corresponds to private data on one hostipital/clinic. We will then use Studio and FEDn to: \n\n1. Train a model on data from one clinic (the model one client can obtain in isolation).  \n2. Export the weights in the pre-trained network for use as a seed model in a FEDn federation.\n3. Deploy a FEDn network, configure local clients, and try to improve the model using federated learning.\n4. (Optionally) Deploy the model in production.\n5. (Optionally) Deploy a simplistic app that lets a pathologist use it to classify images from a browser interface. \n\nThe local datasets used for each site/clinic/client are created by partitioning a publicly available dataset. If you are doing the workshop led by an instructor, you will take the role of a client setting up a local data node on your own local hardware and join a federation that is deployed by the instructor. You will receive additional instructions for how to obtain your data partition, as well as the configuration file needed to attach to the federation. \n\nIf you are doing the tutorial on your own, you need access to a working deployment of [STACKn](https://github.com/scaleoutsystems/stackn) to do the model and app serving part of the tutorial (Steps 4,5). The federated learning part (1,2,3) can be completed with a deployed [FEDn network](https://github.com/scaleoutsystems/fedn). You will also need to download the raw data and prepare your own data partitions (see instructions below). The data download can take up to a few hours depending on your network connection.    \n\n## The model \n\nThis model we will work with is a slimmed down version of the model developed for the Acute Myeloid Leukemia (AML) classification problem in [[1]](#1). Compared to the original work, here the Convolutional Neural Network (CNN) is slightly simplified, and the images are downssampled in order to reduce the computation time and resources, so that the tutorial can be completed without access to GPU resources (a normal Linux or OSX laptop should be enough in most circumstances).    \n\nThe purpose of the model is, as described by the original authors: \n\n\"Reliable recognition of malignant white blood cells is a key step in the diagnosis of hematologic malignancies such as Acute Myeloid Leukemia. Microscopic morphological examination of blood cells is usually performed by trained human examiners, making the process tedious, time-consuming and hard to standardise.\n\nWe compile an annotated image dataset of over 18,000 white blood cells, use it to train a convolutional neural network for leukocyte classification, and evaluate the network’s performance. The network classifies the most important cell types with high accuracy. \n\nOur approach holds the potential to be used as a classification aid for examining much larger numbers of cells in a smear than can usually be done by a human expert. This will allow clinicians to recognize malignant cell populations with lower prevalence at an earlier stage of the disease.\" [[2]](#2).\n\n![Cell image](image.png)\n\n## Preparing the data partitions \n\nThe following instructions are for those preparing their own data partitions from the raw dataset. This is necessary if your are doing the tutorial on your own. If you take part in an instructor-led workshop, partitions will already be available for download.\n\nFirst, clone this repostitory and install the dependencies (requirements.txt). \n\n### Download the raw data\nDownload the dataset from:\nhttps://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=61080958\n\n*Note that the download requires a third-party plugin, and that it can take up to a few hours depending on internet connection.*  \n\n### Partion the dataset\n\nplace the downloaded folder 'AML-Cytomorphology' in the folder 'dataset/raw' in your local clone of this repository, then: \n\n```bash\npython prepare_dataset.py NR_OF_PARTITIONS\n```\nwhere NR_OF_PARTITIONS are the desired number of equal sized splits of the dataset. The script will also downsample the images. To modify this behavior, simply edit prepare_dataset.py. \n\n### Training a centralized model for a given data partition (single clinic)\nThe next step is to gain some experience training the model using the dataset from a single clinic (one partition). To do this, follow the instructions in the notebook 'single_clinic.ipynb'. \n\n### Prepare an initial model for use in FEDn\nIn the final cell of 'single_clinic.ipynb' you will save the weights from the pre-trained single-clinic model as an initial model for use to seed the FEDn federation.  \n\n## Set up a FEDn network \nAgain, if you are working in Studio as part of a workshop, you will set up the FEDn network in collaboration with the instructor. If you are working on your own, follow the instructions [here](https://github.com/scaleoutsystems/fedn) to deploy a FEDn network on your own hardware.  \n\nConfigure FEDn from the UI by uploading the compute package in 'package/package.tar.gz' and the seed file created in the previous step. \n\n## Participating as a data client in the FEDn network \n\nThe below instructions are to set up a local data provider (client in federated learning terminology) to join the federation (FEDn network).   \n\nAttaching a data client involves three main steps:\n\n1. Setting up the local compute environment / install dependencies. \n2. Stage local training and validation data.\n3. Start the *fedn client* pointing to a Reducer endpoint. \n\nDuring step 3, a compute package will be downloaded and staged locally, readying the client for executing training and validation requests.\n\n### 1. Set up the client structure and local environment\n\nOn you local computer/device, create a folder with the following structure \n```yaml\naml-client\n   requirements.txt \n   data/\n```\nrequirements.txt should have the same content as the corresponding file in this repostitory. \n```yaml\ntensorflow==2.7.2\npandas\nscikit-learn\n```\n\n### 2. Obtain a data partition\n\nObtain a data partition: \n\n1. If you are doing this tuturial as part of a workshop you will obtain a download link from the instructor. Unpack the downloaded file\n2. If not, pick a data partition from the set you prepared following the instructions above. \n\nCopy the content of the partition folder to the 'data' folder.\n\n```yaml\naml-client\n   requirements.txt \n   data/\n      --\u003e data_singlets\n      --\u003e labels.npy\n```\n\n### 3. Start a client \nYou can either start a client natively on Linux/OSX, or use the provided Dockerfile in this repository. \n\n#### Native client Linux/OSX\n\nStanding in your created folder: \n\n1. Create a virtual environment and activate it\n```bash\n$ python3 -m venv env\n$ source env/bin/activate\n```\n\n2. Install the fedn client\n```bash\n$ pip install fedn\n```\n\n3. Install dependencies:\n```bash\n$ pip install -r requirements.txt\n``` \n\n4. Access the FEDn UI and download the network configuration file, then place it in the folder (replacing any potential existing client.yaml)\n\n5. Start the client!\n```bash\n$ fedn run client -in client.yaml --name YOUR_CLIENT_NAME\n```\n\n#### Docker\n\n##### Using a prebuilt image\n\n```bash\ndocker pull scaleoutsystems/fedn-client-aml:latest\n```\n\n##### Alternative - download the Dockerfile (or clone this repository), then:\n\nBuild the docker image:\n```bash\ndocker build . -t aml-client:latest\n```\n\nStart a client (edit the path of the volume mounts to provide the absolute path to your local folder)\n```\ndocker run -v /absolute-path-to-this-folder/data/:/app/data:ro -v /absolute-path-to-this-folder/client.yaml:/app/client.yaml scaleoutsystems/fedn-client-aml:latest fedn run client -in client.yaml --name YOUR_CLIENT_NAME --secure=True --force-ssl \n```\n\n### Evaulating a given model version in the FEDn model trail \nFollow the instructions in the notebook 'use_fedn_model.ipynb' (Replace the UUID in the notebook with the desired version from the FEDn model trail. Here we assume that the model trail is accessible on the default path in the Minio instance in your Studio project, if this is not the case,  modify the notebook as needed.)\n \n### Serving the model (optional)\nIf you are working in STACKn, you can easily deploy and serve the single clinic model, or any version of the global model using Tensorflow Serving by following the intructions in the notebook 'single_clinic.ipynb'. You can also edit this file to instead/in addition serve any version of the global federated model. \n\n### Deploying the prediction app\n1. Serve the model you want to use with Tensorflow serving (see previous step).\n2. Using the STACKn UI, start a notebook mounting the \"project-volume\". Then, from a terminal, clone this repository onto \"project-volume\". Edit \"app.py\" to provide your serving endpoint (make sure that the enpoint is public). From the \"Serving\" menu, then create a \"Dash App\", with \"Persistent Volume\" set to \"project-volume\" and \"Path to folder\" set to \"aml-example-project/app\". \n\n## References\n\u003ca id=\"1\"\u003e[1]\u003c/a\u003e \nMatek, C., Schwarz, S., Marr, C., \u0026 Spiekermann, K. (2019). A Single-cell Morphological Dataset of Leukocytes from AML Patients and Non-malignant Controls [Data set]. The Cancer Imaging Archive.\" https://doi.org/10.7937/tcia.2019.36f5o9ld\n\n\u003ca id=\"1\"\u003e[2]\u003c/a\u003e \nMatek, C., Schwarz, S., Spiekermann, K.  et al.  Human-level recognition of blast cells in acute myeloid leukaemia with convolutional neural networks.  Nat Mach Intell   1,  538–544 (2019). https://doi.org/10.1038/s42256-019-0101-9\n\n\u003ca id=\"1\"\u003e[3]\u003c/a\u003e \nClark K, Vendt B, Smith K, Freymann J, Kirby J, Koppel P, Moore S, Phillips S, Maffitt D, Pringle M, Tarbox L, Prior F. The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository, Journal of Digital Imaging, Volume 26, Number 6, December, 2013, pp 1045-1057. DOI: 10.1007/s10278-013-9622-7\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscaleoutsystems%2Fdigital-pathology-tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscaleoutsystems%2Fdigital-pathology-tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscaleoutsystems%2Fdigital-pathology-tutorial/lists"}