{"id":20653355,"url":"https://github.com/ad12/alcv-pipeline","last_synced_at":"2025-03-09T21:53:08.351Z","repository":{"id":97079459,"uuid":"144431915","full_name":"ad12/ALCV-pipeline","owner":"ad12","description":null,"archived":false,"fork":false,"pushed_at":"2018-11-08T05:58:19.000Z","size":4921,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-17T10:44:30.751Z","etag":null,"topics":["computer-vision","deep-learning","machine-learning","medical-image-processing"],"latest_commit_sha":null,"homepage":null,"language":"Matlab","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ad12.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-12T02:27:16.000Z","updated_at":"2021-03-14T11:35:38.000Z","dependencies_parsed_at":"2023-03-30T09:51:19.421Z","dependency_job_id":null,"html_url":"https://github.com/ad12/ALCV-pipeline","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ad12%2FALCV-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ad12%2FALCV-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ad12%2FALCV-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ad12%2FALCV-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ad12","download_url":"https://codeload.github.com/ad12/ALCV-pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242757002,"owners_count":20180204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","machine-learning","medical-image-processing"],"created_at":"2024-11-16T17:43:39.026Z","updated_at":"2025-03-09T21:53:08.320Z","avatar_url":"https://github.com/ad12.png","language":"Matlab","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ALCV Lens Segmentation\n\nThis repository hosts the code used for both the coarse anterior lens localization (CALL) and finer anterior lens localization (FALL) methods detailed in [Arjun D. Desai, Chunlei Peng, Leyuan Fang, Dibyendu Mukherjee, Andrew Yeung, Stephanie J. Jaffe, Jennifer B. Griffin, and Sina Farsiu, \"Open-source, machine and deep learning-based automated algorithm for gestational age estimation through smartphone lens imaging,\" Biomed. Opt. Express 9, 6038-6052 (2018)](https://www.osapublishing.org/boe/abstract.cfm?uri=boe-9-12-6038)\n\nPlease cite this paper if you use any component of this software.\n\nThis software and all corresponding code is copyrighted and may only be used for academic research.\nCOPYRIGHT (C) 2018 DUKE UNIVERSITY\n\nReleased under GPL v2 License without permission for commercial use.\n\n## Dependencies\n### Python\n - python            \u003e=        3.5.5\n - h5py            \u003e=          2.8.0                     \n - imageio           \u003e=        2.3.0                   \n - Keras               \u003e=      2.1.5              \n - matplotlib            \u003e=    2.2.2                  \n - numpy                   \u003e=  1.14.5                \n - opencv-python            \u003e= 3.4.1.15              \n - Pillow                   \u003e= 5.2.0                                        \n - pydot                  \u003e=   1.2.4                             \n - scikit-learn              0.19.2                   \n - scipy                     \u003e=1.1.0                  \n - tesnorflow/tensorflow-gpu          \u003e=  1.4.0      \n\n\n### MATLAB\n- MATLAB \u003e= R2018a\n- [LIBSVM for MATLAB](https://www.csie.ntu.edu.tw/~cjlin/libsvm/) \u003e= 3.18\n- [matconvnet](http://www.vlfeat.org/matconvnet/) \u003e= 1.0-beta25\n\n### Open source libraries\nWe thank all research pioneers who have open-sourced their work for public use. We have utilized several open-source libraries as part of our code, all of which are documented below.\n\n#### Blind image quality analyzers\n1. S. Gabarda and G. Cristóbal, \"Blind image quality assessment through anisotropy,\" J.\nOpt. Soc. Am. A 24, B42-B51 (2007). [link](https://www.osapublishing.org/josaa/abstract.cfm?uri=josaa-24-12-b42)\n2. A. Mittal, R. Soundararajan, and A. C. Bovik, \"Making a \"Completely Blind\" Image\nQuality Analyzer,\" IEEE Signal Processing Letters 20, 209-212 (2013). [link](https://ieeexplore.ieee.org/document/6353522)\n\n#### Circular hough transform\n3. D. Young, \"Hough Transform for Circles\", Mathworks File Exchange (2016). [link](https://www.mathworks.com/matlabcentral/fileexchange/26978-hough-transform-for-circles)\n\n#### DeeplabV3+ implementation\n4. Keras implementation of DeeplabV3+ GitHub repo (bonlime) - [link](https://github.com/bonlime/keras-deeplab-v3-plus)\n\n#### Extraneous\nThe following code was added as part of the library but not used in the final computation:\n\n5. Dense scene correspondence - C. Liu, J. Yuen, and A. Torralba. \"Sift flow: Dense correspondence across scenes and its applications.\" IEEE transactions on pattern analysis and machine intelligence 33.5, 978-994 (2011). [link](https://people.csail.mit.edu/celiu/SIFTflow/)\n\n6. Image fusion framework - Y. Liu, S. Liu and Z. Wang, \"A General Framework for Image Fusion Based on Multi-scale Transform and Sparse Representation\", Information Fusion (2014). [link](https://github.com/RexYing/infant-eye/tree/master/Dibyendu/Project%20-%20Neonatal%20Eye/External%20Codes/Fusion/MST_SR_fusion_toolbox)\n\n7. Intelligent Scissors [link](http://www.andrewnoske.com/wiki/Livewire_Segmentation)\n\n8. Cosfire segmentation - G. Azzopardi, N. Strisciuglio, M. Vento, and N. Petkov, \"Trainable COSFIRE filters for vessel delineation with application to retinal images\", Medical Image Analysis, Available online 3 September 2014, ISSN 1361-8415 [link](http://dx.doi.org/10.1016/j.media.2014.08.002) (available feature in MATLAB GUI code, but not used in lens segmentation)\n\n9. Dijkstra segmentation - L. Grady, Graph Analysis Toolbox. [link](http://leogrady.net/software/) (available feature in MATLAB GUI code, but not used in lens segmentation)\n\n10. Hysteresis segmentation - L. Xie, \"Hysteresis thresholding for 3D images (or 2D)\" Mathworks File Exchange (2013). [link](https://www.mathworks.com/matlabcentral/fileexchange/44648-hysteresis-thresholding-for-3d-images-or-2d)\n\n## Environment Setup\n### MATLAB\n\n1. Turn off hardware acceleration for video decoding\n`matlab.video.read.UseHardwareAcceleration('off')`\n\n2. Download and setup [matconvnet](http://www.vlfeat.org/matconvnet/)\n\n3. Download [resnet-152-dag](http://www.vlfeat.org/matconvnet/models/imagenet-resnet-152-dag.mat) and save in `mtlb/classification_utils` folder\n\n4. Download [libsvm](https://www.csie.ntu.edu.tw/~cjlin/libsvm/) for MATLAB. Add to matlab path.\n\n## Input files\n\nAll input files must be videos with the ```.mov``` extension. While the filenames do not have a fixed format, the recommended format is ```\u003cCASEID\u003e_\u003cDATE\u003e_\u003cTIME\u003e.mov``` without the carats (```\u003c```,`\u003e`). The ```CASEID``` is an 8 digit string such as ```01020031```. It is preferred if the format of the ```DATE``` string is maintained across videos. The standard form is ```YYYYDDMM```. ```TIME``` is recorded in military time (e.g. `9:20 p.m.` is `2120`).\n\nExamples:\n - `01010001_20180401_0920`: Subject 01010001, born April 01, 2018 at 9:20 AM\n - `01010002_Apr012018_1743`: Subject 01010002, born April 01, 2018 at 5:43 PM\n\n\n**Please put all videos in a single folder**. This folder will be the base folder for analysis.  \n\n## Lens segmentation\nThis section describes the pipeline used for both the coarse anterior lens localization (CALL) and finer anterior lens localization (FALL) methods detailed in [Desai, et al. - TODO](https://www.osapublishing.org/boe/abstract.cfm?uri=boe-9-12-6038)\n\n### Coarse anterior lens localization (CALL)\n\nTo use CALL, save videos with the ```.mov``` extension in a folder. Run the ```ALCV-debug.m``` method.\n\n##### Generate mat files\nSelect ```Video``` in the dropdown and click the ```Process Batch``` button. For more information, see the figure below. The dropdown and the button are circled.\n\n![alt text](readme_images/alcvsoft1.png)\n\nA popup selection window will appear. Select the folder containing the videos you would like to analyze. Confirm the selection and wait for the process to complete. Please note that this process does take some time. A waitbar will display the index of the video it is currently processing. If the process fails on a video, it will notify the user on the MATLAB console. The video folder should now contain a ```.mat``` file for every ```.mov``` video file.\n\n##### Extract top frames\nNext use the ALCV window to extract all top frames from the mat file. Select ```Utils --\u003e Extract Top Frames``` in the menu bar. The menu option is circled in the figure below. Choose the folder where all the mat files were stored. By default, these mat files will be in the same folder as the videos. The frames will be extracted and saved as png files in the subfolder ```ALCV_frames_top50/```. This folder will have additional directories labelled by case id. Each case id directory will contain the top frames selected by the CALL algorithm.\n\n **Do not change the path to or file format of these images.**\n\n![alt text](readme_images/alcvsoft2.png)\n\n### Finer anterior lens localization (FALL)\nThe finer anterior lens localization algorithm was trained using 25-fold [cross validation](https://www.openml.org/a/estimation-procedures/1). This means that 25 different optimal architectures (25 different weight files) were produced, one for each fold. As there are multiple models to choose from, you can choose which weights to load into the model.\n\nThere are a few ways to go about this issue. One approach is to use a single model initialized with a weight file of your choosing. Ideally, this would be the weight file that has the lowest validation loss, as that is our metric for performance. However, you can choose whichever file you wish.\n\nAnother approach is to use [ensemble learning](https://papers.nips.cc/paper/1001-neural-network-ensembles-cross-validation-and-active-learning.pdf), where each of the 25 models can be used to produce a binary prediction mask, resulting in 25 different masks. Then to determine the final mask, each of the 25 models (ensembles) votes whether the pixel is 1 or 0. So for any pixel *(i, j)*, there are 25 votes. A simple method would take the average of the probabilities across different models and return the class with the highest averaged probability. This is the ensemble method we have implemented. As ensemble learning utilizes multiple models, the inference time will be slower.\n\nNote that neither of these methods have been tested independently on a test set because the evaluation on our test set was done following the k-fold cross validation protocol.\n\nWe have provided the code for you to run both of these methods. Details on how to use this is described below\n\n#### Network Weights\nThe 25-fold cross validation scheme produced 25 different optimal weight files. These files can be found [here - Link weight](google.com). The optimal weights for each cross validation experiment are the weights that minimized the validation loss.\n\nAll weight files are of the H5 file format. The file names indicate the validation loss. For example, a weight file named `005-weights.019-0.1529.h5` produced a validation loss of 0.1529 after epoch 19 in cross validation experiment 5 (zero-indexed).\n\n#### Shell Interface\nTo run this program from a python-enabled shell or command line interface, run `python -m opt/path/pipeline` with the flags detailed below. `opt/path` is the path to the file `fall_pipeline.py`\n\nNote this program is meant to be run from the command line. As a result, all import statements are local imports.\n\n#### Setup\nTo run this program, open a shell/terminal window that is enabled with the python dependencies listed in the Dependencies section. A [python virtual environment](https://realpython.com/python-virtual-environments-a-primer/) is recommended.\n\n#### Shell Help\nThe command line flags are explained below:\n\n**Required**:\n- `-d D, --dir D`: The path to the directory that you want to process. Safer to use absolute paths.\n  1. If you are processing a single subject, `D` should be the path to that subject folder. For example, `\u003cpath_to_folder\u003e/ALCV_frames_top50/01010001_20180401_0920`.\n  2. If you are processing a batch of subjects (`-b, --batch` flag), then specify the path to the directory holding the multiple subjects (i.e. `ALCV_frames_top50`). For example, `\u003cpath_to_folder\u003e/ALCV_frames_top50`\n\n\n - `-w W, --weight W`: The path to weight(s) to use for the network.\n  1. If using a single weight file, specify the path to the weight file. For example, `\u003cpath_to_weights_directory\u003e/weights/weights.019-0.1529.h5`\n  2. If using multiple weight files (ensemble), specify the path to the weight directory. All H5 weight files in this directory will be used for ensembling. For example, `\u003cpath_to_weights_directory\u003e/weights`\n\n**Optional**:\n- `-r`: Resize frames to size of images used to train the network. The network is fully convolutional, so it can take input images of any size greater than 216x384. However, the performance was not tested on images of different sizes.\n\n- `-b, --batch`: Process multiple subjects. See details on `-d` flag for more information.\n\n- `--mini_batch_size`: Mini batch size for inference on the network. Default is 50.\n\n\nThe output of the command line help menu (```python -m pipeline -h```) can be seen below:\n```\nusage: pipeline.py [-h] -d D -w W [-r] [-b] [--mini_batch_size BS]\n\nSegment anterior lens regions in frames\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -d D, --dir D         Either 1. Path to subject directory storing frames. or\n                        2. Path to directory of subject directories\n  -w W, --weight W      path to weight(s). If path is a directory, use\n                        ensemble learning. Else path must be an h5 file\n  -r                    Resize images to default shape for segmentation. If\n                        input images are not of shape (216, 384, 3), then\n                        model may not perform optimally\n  -b, --batch           Batch process all subdirectories in D. Assumes all\n                        subdirectories in D are of subjects\n  --mini_batch_size BS  Mini batch size. Default is 50\n```\n\n#### Use cases\nAll use cases assume that the current working directory is this repository. If the working directory is different, make sure to specify the path to `Fall-segmentation/pipeline.py` when running the script. For example, `python -m ~/alcv-pipeline/Fall-segmentation/pipeline` if the repository is located in the user directory.\n\nAll cases also resize the input using `-r` flag, because this is recommended. if you don't want to, remove the `-r` flag from the use cases\n\n\n##### Case 1\n###### Process single subject in `ALCV_frames_top50` folder\n`python -m Fall-segmentation/pipeline -r -d \u003cpath\u003e/ALCV_frames_top50/\u003csubject_folder\u003e -w \u003cpath\u003e/weights/weights.019-0.1529.h5`\n\n##### Case 2\n###### Process multiple subjects in `ALCV_frames_top50` folder\n`python -m Fall-segmentation/pipeline -b -r -d \u003cpath\u003e/ALCV_frames_top50 -w \u003cpath\u003e/weights/weights.019-0.1529.h5`\n\n##### Case 3\n###### Process multiple subjects using ensemble inference\n`python -m Fall-segmentation/pipeline -b -r -d \u003cpath\u003e/ALCV_frames_top50 -w \u003cpath\u003e/weights`\n\n##### Case 4\n###### Process multiple subjects with mini-batch-size of 16\n`python -m Fall-segmentation/pipeline -b -r -d \u003cpath\u003e/ALCV_frames_top50 -w \u003cpath\u003e/weights/weights.019-0.1529.h5 --mini_batch_size 16`\n\n#### Output\nFollowing the finer anterior lens localization, each subject folder should contain two folders `mask-pred` and `mask-pred-visual`. The `mask-pred` folder contains the mask as 0s and 1s, but it is hard to visualize. So we create another folder `mask-pred-visual` where the masks can be seen as images. Note these are the mask outputs of the FALL segmentation, they have not been refined yet.\n\n### Extracting the lens\nReturn to the MATLAB GUI. Click on the `Segmentation` menu in the menu bar. This menu is circled in the figure below. There are two options to extract the lens- either subject by subject or by batch. The two menu options `Extract lens` and `Batch Extract lens` correspond to these options respectively.\n\n![alt text](readme_images/alcvsoft3.png)\n\n**`Extract Lens`**:\nIf you chose to extract lens by subject, you will be prompted the select the directory of the subject you wish to process.\n\n**`Batch Extract Lens`**:\nIf you choose to batch extract lens for multiple subjects, you will be prompted to select the parent directory of the subject folders (i.e. `ALCV_frames_top50`).\n\n#### Output\nThe extracted lens regions and the best lens region will be stored in the folder `lens` and `best_lens` for each subject\n\n### Classification\nIn the paper, the classification was tested using leave-1-out cross validation, which for 124 subjects, yielded 124 SVM models for each of the 6 thresholds (33-38 weeks). As this pipeline is only for inference purposes, we have retrained a single model for each threshold using the 124 subjects in the paper.\n\nIn the MATLAB GUI, go to the `Classification -\u003e Estimate Age` option as circled in the figure below. Select the `ALCV_frames_top50` directory to process classify the ages of all of the subjects with extracted lens regions in that folder. The results will be saved to a `results.csv` file.\n\n![alt text](readme_images/alcvsoft4.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fad12%2Falcv-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fad12%2Falcv-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fad12%2Falcv-pipeline/lists"}