{"id":16519534,"url":"https://github.com/kwanit1142/sars-cov-2-detection-radiology-based-multi-modal-multi-task-framework","last_synced_at":"2026-04-14T05:31:26.680Z","repository":{"id":129370299,"uuid":"402791218","full_name":"kwanit1142/SARS-CoV-2-Detection-Radiology-based-Multi-modal-Multi-task-Framework","owner":"kwanit1142","description":"Keras-based Implementation for \"SARS-CoV-2 Detection: Radiology based Multi-modal Multi-task Framework\" (Accepted in 45th IEEE EMBC 2k23)","archived":false,"fork":false,"pushed_at":"2023-11-26T12:27:21.000Z","size":167,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-13T14:44:42.622Z","etag":null,"topics":["gans","keras","matplotlib","numpy","opencv","os","pandas","pillow","scipy","sklearn","tensorflow","transfer-learning","wasserstein-distance"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kwanit1142.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-09-03T14:08:15.000Z","updated_at":"2023-11-26T12:28:12.000Z","dependencies_parsed_at":"2023-07-29T03:31:42.392Z","dependency_job_id":null,"html_url":"https://github.com/kwanit1142/SARS-CoV-2-Detection-Radiology-based-Multi-modal-Multi-task-Framework","commit_stats":null,"previous_names":["kwanit1142/sars-cov-2-detection-radiology-based-multi-modal-multi-task-framework"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kwanit1142/SARS-CoV-2-Detection-Radiology-based-Multi-modal-Multi-task-Framework","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwanit1142%2FSARS-CoV-2-Detection-Radiology-based-Multi-modal-Multi-task-Framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwanit1142%2FSARS-CoV-2-Detection-Radiology-based-Multi-modal-Multi-task-Framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwanit1142%2FSARS-CoV-2-Detection-Radiology-based-Multi-modal-Multi-task-Framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwanit1142%2FSARS-CoV-2-Detection-Radiology-based-Multi-modal-Multi-task-Framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kwanit1142","download_url":"https://codeload.github.com/kwanit1142/SARS-CoV-2-Detection-Radiology-based-Multi-modal-Multi-task-Framework/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwanit1142%2FSARS-CoV-2-Detection-Radiology-based-Multi-modal-Multi-task-Framework/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31784251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T02:24:21.117Z","status":"ssl_error","status_checked_at":"2026-04-14T02:24:20.627Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gans","keras","matplotlib","numpy","opencv","os","pandas","pillow","scipy","sklearn","tensorflow","transfer-learning","wasserstein-distance"],"created_at":"2024-10-11T16:47:07.320Z","updated_at":"2026-04-14T05:31:26.663Z","avatar_url":"https://github.com/kwanit1142.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SARS-CoV-2 Detection: Radiology based Multi-modal Multi-task Framework\n\n## About\n\n![MTLArch](https://user-images.githubusercontent.com/54277039/132307382-53335fdc-06ce-4c9b-b9d6-fa6a266cf219.png)\n\n## Required Python Dependencies\n\nThese were categorized according to their application purposes as following :-\n\n- Model Architectures and Deep Learning based Libraries\n  - tensorflow\n  - keras\n- Evaluation Metrics and Statistical Analysis based Libraries\n  - sklearn\n  - scipy\n- Image Processing and Data Visualization based Libraries\n  - opencv (cv2)\n  - pillow (PIL)\n  - matplotlib\n- Matrices and Data Engineering based Libraries\n  - numpy\n  - pandas\n- System and Directory Management based Library\n  - os\n  - tqdm\n\n## Adviced Structure of Dataset Folder\n\nFor the ease of Training and Implementation flow, following structure would be adviced :-\n\n```\n|--Dataset Folder/\n\n   |--X_Ray_Train/\n\n      |--COVID/\n         \n         |--x_c1_image_train\n         |--x_c2_image_train\n         ...\n         \n      |--Non_COVID/\n\n         |--x_nc1_image_train\n         |--x_nc2_image_train\n         ...\n         \n   |--X_Ray_Test/\n\n      |--COVID/\n      \n         |--x_c1_image_test\n         |--x_c2_image_test\n         ...\n         \n      |--Non_COVID/\n      \n         |--x_nc1_image_test\n         |--x_nc2_image_test\n         ...\n         \n   |--CT_Scan_Train/\n\n      |--COVID/\n      \n         |--ct_c1_image_train\n         |--ct_c2_image_train\n         ...\n         \n      |--Non_COVID/\n      \n         |--ct_nc1_image_train\n         |--ct_nc2_image_train\n         ...\n\n   |--CT_Scan_Test/\n\n      |--COVID/\n      \n         |--ct_c1_image_test\n         |--ct_c2_image_test\n         ...\n         \n      |--Non_COVID/\n      \n         |--ct_nc1_image_test\n         |--ct_nc2_image_test\n         ...\n\n```\n\n## Steps for User-Customized Training\n\n1. Arrange the Dataset according to the adviced structure, mentioned above.\n\n2. Run `Dataset_PreProcessing.py`, which consist of 2 functions, naming `Sample_Processing` and `Dataset_Processing`, with the following details :-\n   \n   - Sample_Processing(x_ray_directory,ct_scan_directory) :\n   \n      - `x_ray_directory` --\u003e A String of the X-Ray Image File Path with the name of the file and extension of it.\n      - `ct_scan_directory` --\u003e A String of the CT-Scan Image File Path with the name of the file and extension of it.\n   \n   - Dataset_Processing(x_base_directory,ct_base_directory) :\n   \n      - `x_base_directory` --\u003e A String of the X-Ray Folder Path (till COVID or Non-COVID Folder).\n      - `ct_base_directory` --\u003e A String of the CT-Scan Folder Path (till COVID or Non-COVID Folder).\n\n3. Run `Embeddings_Generator.py`, which consist of 2 functions, naming `Embedding_Model` and `Embedding_Save`, with the following details **(Replace Model_Name with keras-based Model Classes like keras.applications.VGG16, keras.applications.InceptionV3, etc.)** :-\n   \n   - Embedding_Model(Model_save_directory,Model_Class,Dataset_train_Directory,Dataset_test_Directory) :\n   \n      - `Model_save_directory` --\u003e A String of the Transfer Learning Model Path, to be saved in future, with the name of file and extension (.h5 or .hdf5) of it.\n      - `Model_Class` --\u003e A Class Name of the Transfer Learning Model, like keras.applications.VGG16 (To be changed by the user in Function File)\n      - `Dataset_train_Directory` --\u003e A String of the Train Dataset Directory Path (till x_ray_train or ct_scan_train folder).\n      - `Dataset_test_Directory` --\u003e A String of the Train Dataset Directory Path (till x_ray_test or ct_scan_test folder).\n   \n   - Embedding_Save(Embed_save_Directory,Trained_Model_Class,Dataset_Directory,Image_Size) :\n   \n      - `Embed_save_Directory` --\u003e A String of the Directory Path (with the name of file and extension (.npy) of it) where the Embeddings would be saved.\n      - `Trained_Model_Class` --\u003e The Class of the Trained Transfer Learning Model, which would be loaded from the user side using keras.models.load_model(\u003c--Model_Path--\u003e).\n      - `Image_Size` --\u003e The Size of images to be taken as input (331 for NasNetLarge and 224 for others) during embedding generation.\n      - `Dataset_Directory` --\u003e A String of the Dataset Directory Path (till x_ray_train, ct_scan_train, x_ray_test or ct_scan_test folder).\n\n4. Run `Task_Specific_Feature_Extractor.py`, which consist of 2 functions, naming `x_ray_task_specific` and `ct_scan_task_specific`, with the following details :-\n   \n   - x_ray_task_specific(Model_save_directory,train_embed,test_embed,train_labels,test_labels) :\n   \n      - `Model_save_directory` --\u003e A String of the X-Ray Task Specific Feature Extractor Path, to be saved in future, with the name of file and extension (.h5 or .hdf5) of it.\n      - `train_embed` --\u003e A numpy array of embeddings from training dataset images.\n      - `test_embed` --\u003e A numpy array of embeddings from testing dataset images.\n      - `train_labels` --\u003e A numpy array of task-specific labels (whether COVID or Non-COVID) from training dataset images.\n      - `test_labels` --\u003e A numpy array of task-specific labels (whether COVID or Non-COVID) from testing dataset images.\n   \n   - ct_scan_task_specific(Model_save_directory,train_embed,test_embed,train_labels,test_labels) :\n   \n      - `Model_save_directory` --\u003e A String of the CT-Scan Task Specific Feature Extractor Path, to be saved in future, with the name of file and extension (.h5 or .hdf5) of it.\n      - `train_embed` --\u003e A numpy array of embeddings from training dataset images.\n      - `test_embed` --\u003e A numpy array of embeddings from testing dataset images.\n      - `train_labels` --\u003e A numpy array of task-specific labels (whether COVID or Non-COVID) from training dataset images.\n      - `test_labels` --\u003e A numpy array of task-specific labels (whether COVID or Non-COVID) from testing dataset images.\n\n5. Decide whether to make Shared Features Extractors with or without Adversarial Training. Depending upon the choice, following would be the implementation :-\n\n   - Without Adversarial Training, Run `Shared_Features_Without_Adversarial.py`, which consist of 2 functions, naming `mlp_model` and `mlp_train`, with the following details :-\n   \n      - `mlp_model()` --\u003e It would make the MLP model with pre-defined parameters and architecture.\n      - `mlp_train(Model_save_Directory,x_train_embed,x_test_embed,ct_train_embed,ct_test_embed,train_labels,test_labels)` --\u003e A function to Train the pre-defined MLP Model with the help of training and testing embeddings of X-Ray and CT-Scan with their corresponding Task-Specific Labels (Whether COVID or Non-COVID), all being of the form of numpy array.\n\n   - With Adversarial Training, Run `Shared_Features_Extractor.py`, which consist of some functions, with their following details :- \n   \n      - `gen_model()` and `disc_model()` --\u003e These Functions would produce the MLP Architecture with pre-defined parameters.\n      - `disc_loss_func(actual_embed,gen_embed,train_labels,invert_labels)` --\u003e Discriminator Loss Function where the actual input and generator's output go alongside Actual training labels and inverted labels (interchanging classes --\u003e Whether X-Rays or CT-Scans), all of the form of numpy arrays.\n      - `gen_loss_func(gen_embed,invert_labels)` --\u003e Generator Loss Function where the Generated Output and inverted class labels (Whether X-Rays or CT-Scans) go, all of the form of numpy arrays.\n      - `train_step(actual_embed,old_gen_loss,old_disc_loss,train_labels,invert_labels)` --\u003e Adversarial Training happens here, alongside Actual Input, Old Values of Generator and Discriminator Loss, with Training Labels and Inverted Class Labels, all of the form of numpy arrays.\n      - `fine_tune(Model_save_Directory,model_gen,x_train_embed,x_test_embed,ct_train_embed,ct_test_embed)` --\u003e Here, the Shared Feature Extractor Module would be fine tuned and best model iteration would be saved according to \"Model_save_Directory\", with Class Labels (Whether X-Rays or CT-Scans).\n\n6. Run `Classifier_Head.py`, which consist of 2 functions, naming `Classifier_Head` and `Final_Embeddings`, with the following details :-\n\n   - Classifier_Head(Model_save_Directory,train_embed,test_embed,train_labels,test_labels) :\n   \n      - `Model_save_Directory` --\u003e A String of the Classifier Head Path, to be saved in future, with the name of file and extension (.h5 or .hdf5) of it.\n      - `train_embed` --\u003e A numpy array of embeddings from training dataset images, after passing through Task Specific and Shared Features Extractor Modules.\n      - `test_embed` --\u003e A numpy array of embeddings from testing dataset images, after passing through Task Specific and Shared Features Extractor Modules.\n      - `train_labels` --\u003e A numpy array of Labels (Whether COVID or Non-COVID) for training dataset images.\n      - `test_labels` --\u003e A numpy array of Labels (Whether COVID or Non-COVID) for testing dataset images.\n    \n   - Final_Embeddings(\u003c----Arguments----\u003e) :\n   \n      - `shared_model`, `x_task_model`, `ct_task_model` --\u003e Model Classes of Shared Features Extractor and Task-Specific X-Ray and CT-Scans Feature Extractors, loaded using keras.models.load_model(\u003c--Model_Path--\u003e).\n      - `shared_x_ray_train_input`,`shared_x_ray_test_input`,`shared_ct_train_input`,`shared_ct_test_input` --\u003e Numpy arrays of Embeddings Input for Shared Features Extractor Modules.\n      - `task_x_ray_train_input`,`task_x_ray_test_input`,`task_ct_train_input`,`task_ct_test_input` --\u003e Numpy arrays of Embeddings Input for Task-Specific Features Extractor Modules.\n\n7. Run `User_Customized_Pipeline.py`, which consist of some functions, with the following details :-\n\n   - Pre_Process(prompt,x,ct) :\n      - `prompt` --\u003e A Parameter which define the input conditions.\n      - `x` --\u003e Greyscaled X-Ray Image Matrix, loaded using cv2.imread(\u003c---Image Path---\u003e , 0)\n      - `ct` --\u003e Greyscaled CT-Scan Image Matrix, loaded using cv2.imread(\u003c---Image Path---\u003e , 0)\n   - Task_Specific_Embed(Save_Folder_Directory,x,ct) :\n      - `Save_Folder_Directory` --\u003e A String of the Container Folder Path where each component of the Pipeline was saved.\n      - `x` --\u003e Pre-Processed X-Ray Image Matrix.\n      - `ct` --\u003e Pre-Processed CT-Scan Image Matrix. \n   - Shared_Feature_Embed(Save_Folder_Directory,x,ct) :\n      - `Save_Folder_Directory` --\u003e A String of the Container Folder Path where each component of the Pipeline was saved.\n      - `x` --\u003e Pre-Processed X-Ray Image Matrix.\n      - `ct` --\u003e Pre-Processed CT-Scan Image Matrix.\n   - Classification(Save_Folder_Directory,x,ct) : \n      - `Save_Folder_Directory` --\u003e A String of the Container Folder Path where each component of the Pipeline was saved.\n      - `x` --\u003e Pre-Processed X-Ray Image Matrix.\n      - `ct` --\u003e Pre-Processed CT-Scan Image Matrix. \n   - Test_Run(Save_Folder_Directory,prompt,X_Ray_Grey_Image,CT_Scan_Grey_Image) : \n      - `Save_Folder_Directory` --\u003e A String of the Container Folder Path where each component of the Pipeline was saved.\n      - `X_Ray_Grey_Image` --\u003e Greyscaled X-Ray Image Matrix, loaded using cv2.imread(\u003c---Image Path---\u003e , 0)\n      - `CT_Scan_Grey_Image` --\u003e Greyscaled CT-Scan Image Matrix, loaded using cv2.imread(\u003c---Image Path---\u003e , 0)\n      - `prompt` --\u003e A Parameter which define the input conditions.\n\n8. For Evaluating and Analysing the Generalization of Implementation, consider using `Evaluation_and_Significance_Test.py`, which consist of 2 functions, naming `Evaluation_metrics` and `Significance_Test`, with the following details :-\n   \n   - Evaluation_metrics(prob_list,labels,threshold=0.5) :\n      - `prob_list` --\u003e List of Output Probabilities after implementing the whole pipeline.\n      - `labels` --\u003e Output Classes vector.\n      - `threshold` --\u003e Confidence Probability (generally 0.5 for ideal sigmoid scenario).\n   \n   - Significance_Test(Save_Directory,task_train_embed,shared_train_embed,task_test_embed,shared_test_embed,train_labels,test_labels) :\n      - `Save_Directory` --\u003e A String of Sample Run Iterations, with the name and extension (.h5 or .hdf5) on it.\n      - `task_train_embed`,`shared_train_embed` --\u003e Numpy arrays of Task Specific and Shared Features Embeddings from Training Data.\n      - `task_test_embed`,`shared_test_embed` --\u003e Numpy arrays of Task Specific and Shared Features Embeddings from Testing Data.\n      - `train_labels`,`test_labels` --\u003e Numpy arrays of Training and Testing Labels (Whether COVID or Non-COVID)\n\n## Pre-Processed Train-Test Dataset\n\nIn order to start with the suggested Dataset from the Research Paper, access this link and use it accordingly, with Label Annotations as `train.csv` and `test.csv`. As it was already pre-processed, Running `Dataset_PreProcessing.py` won't be a requirement.\n\nhttps://drive.google.com/drive/folders/1dNGICFvzV1gF66QIz8RVizMeyaON6APp?usp=sharing\n\n## Test Run using Pre-Trained Pipeline\n\nRun `Pre_Trained_Pipeline.py` with the Directory Paths mentioned accordingly.\n\nPre-Trained Model Weights ---\u003e https://drive.google.com/file/d/1tHIDYe83hk4k0y511zx8ELh7UlQ2KHWP/view?usp=sharing\n\n## Documentations and Resources\n\n1. https://keras.io/api/applications/\n2. https://docs.opencv.org/4.5.1/d2/d96/tutorial_py_table_of_contents_imgproc.html\n3. https://www.youtube.com/watch?v=tX-6CMNnT64\u0026list=LL\u0026index=102\n4. https://docs.scipy.org/doc/scipy/reference/stats.html\n5. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html\n6. https://towardsdatascience.com/hypothesis-testing-in-machine-learning-using-python-a0dc89e169ce\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwanit1142%2Fsars-cov-2-detection-radiology-based-multi-modal-multi-task-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkwanit1142%2Fsars-cov-2-detection-radiology-based-multi-modal-multi-task-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwanit1142%2Fsars-cov-2-detection-radiology-based-multi-modal-multi-task-framework/lists"}