{"id":13574364,"url":"https://github.com/oneapi-src/image-data-generation","last_synced_at":"2025-04-04T15:30:45.057Z","repository":{"id":152282011,"uuid":"610975667","full_name":"oneapi-src/image-data-generation","owner":"oneapi-src","description":"AI Starter Kit for Synthetic Image Generation using Intel® Optimized Tensorflow*","archived":true,"fork":false,"pushed_at":"2024-02-01T23:56:37.000Z","size":804,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-05T09:44:29.575Z","etag":null,"topics":["deep-learning","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oneapi-src.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-07T21:22:10.000Z","updated_at":"2024-04-08T18:36:59.000Z","dependencies_parsed_at":"2024-11-05T09:33:53.887Z","dependency_job_id":"a811bbdd-a3a5-460e-acf7-fd73b9bc874e","html_url":"https://github.com/oneapi-src/image-data-generation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oneapi-src%2Fimage-data-generation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oneapi-src%2Fimage-data-generation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oneapi-src%2Fimage-data-generation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oneapi-src%2Fimage-data-generation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oneapi-src","download_url":"https://codeload.github.com/oneapi-src/image-data-generation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247202528,"owners_count":20900794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","tensorflow"],"created_at":"2024-08-01T15:00:50.886Z","updated_at":"2025-04-04T15:30:40.048Z","avatar_url":"https://github.com/oneapi-src.png","language":"Python","readme":"PROJECT NOT UNDER ACTIVE MANAGEMENT\n\nThis project will no longer be maintained by Intel.\n\nIntel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.  \n\nIntel no longer accepts patches to this project.\n\nIf you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the open source software community, please create your own fork of this project.  \n\nContact: webadmin@linux.intel.com\n# Applications of Synthetic Image Generation \r\n## Introduction\r\nSynthetic image generation is the process of creating new images using algorithms or simulations to substitute real-world data. [More Info](https://en.wikipedia.org/wiki/Synthetic_data)\r\n\r\n\r\n## Table of Contents \r\n - [Purpose](#purpose)\r\n - [Reference Solution](#reference-solution)\r\n - [Reference Implementation](#reference-implementation)\r\n - [Intel® Optimized Implementation](#optimizing-the-end-to-end-solution-with-intel%C2%AE-oneapi-components)\r\n - [Performance Observations](#performance-observations)\r\n\r\n## Purpose\r\nThere is a growing interest in medical diagnosis systems using Deep Learning. However, due to proprietary and privacy reasons, the data access to the public has been very limited and thereby leading to limited public contributions in this field of study. Moreover, access to medical image data without expert annotations and accurate metadata, renders them useless for most modern applications. \r\nThis has led to a larger interest in generating synthetic data - since these can be used for deep learning model training and validation while vastly reducing the costs of data collection and labeling.\r\n\r\n\r\nA notable use case, in the medical industry, is to utilize synthetic data generation of retinal Images for vessel extraction. A detailed interpretation and study of the retinal vasculature rely on precise vessel segmentation, which is a very laborious and time taking task. Since public data sources of retinal images are very limited and are bound by strict privacy clauses. A synthetic data generation of the images helps to provide a larger data source to aid the model development from the existing and limited data sources.\r\n\r\nIn this reference kit, we demonstrate a reference implementation for this use case.\r\n\r\n\r\n## Reference Solution \r\nAn AI-enabled image generator aid helps to generate new accurate image and image segmentation datasets when the dataset is limited\r\n\r\nIn this reference kit, we consider a two-stage pipeline for generating synthetic medical images of retinal fundi using  generative adversarial networks :\r\n1. Stage 1 - Generate segmented images from a U-Net model using a real dataset of fundi images.\r\n2. Stage 2 - Generate synthetic retinal fundi image from a GAN model using a real dataset of fundi images and its segmented pairs generated in stage 1.\r\n\r\n\r\nIn Stage 1, a real fundus image is processed using a U-Net model to segment the vessel from the input fundus image. U-Net is a special type of CNN segmentation architecture that was designed to do biomedical image segmentation tasks. \r\nThese segmented masks are then fed into Stage 2 together with real fundus images to generate fundus images synthetically.\r\n\r\nStage 2 is utilized to convert the segmentation masks into photorealistic fundus images. Stage 2 starts with training with real fundi and segmented vessel masks to determine the mapping between the two image classes. During Inferencing, a segmented vessel mask is fed as an input to the model which will transform the given segmented mask into a synthetic photorealistic fundus image.\r\n\r\nSince GPUs are typically the choice for deep learning and AI processing to achieve a higher Frames Per Second (FPS) rate, to offer a more cost-effective \r\noption leveraging a CPU, we use the quantization technique, leveraging the Intel Analytics toolkit, to achieve high FPS by performing vectorized operations on the CPUs itself.\r\n\r\nBy quantizing/compressing the model (from floating point to integer model), while maintaining a similar level of accuracy as the floating point model, \r\nwe demonstrate efficient utilization of underlying resources when deployed on edge devices with low processing and memory capabilities.\r\n\r\n\r\n## Key Implementation Details\r\n- Highlighted the difference between using Intel® Optimizations for TensorFlow* v2.11.0 against the packages of the stock version of the TensorFlow v2.8.0.\r\n- Model has been Quantized using Intel® Neural Compressor which has shown high-performance vectorized operations on Intel® platforms.\r\n\r\n## Reference Implementation\r\n### Use Case End-To-End flow\r\n\r\n![Use_case_flow](assets/e2eflow.png)\r\n\r\n### Reference Sources\r\n\r\n*DataSet*: https://www5.cs.fau.de/research/data/fundus-images/ \u003cbr\u003e\r\n*Case Study \u0026 Repo*: https://github.com/diazandr3s/synthetic-medical-images (Retinal Fundus images)\u003cbr\u003e\r\n\r\n\u003e ***Please see this data set's applicable license for terms and conditions. Intel®Corporation does not own the rights to this data set and does not \r\nconfer any rights to it.***\r\n\r\n### Dataset\r\n\r\n| **Use case** | Synthetic Image Generation\r\n| :--- | :---\r\n| **Object of interest** | Synthetic Fundus Image Generation\r\n| **Size** | Total 15 Images \u003cbr\u003e\r\n| **Train : Test Split** | 10 images for training and 5 images for testing\r\n| **Source** | https://www5.cs.fau.de/research/data/fundus-images/\r\n\r\n### Expected Input-Output for Stage 1\r\nThe input and output for Stage 1 are given below.\r\n**Input**                                 | **Output** |\r\n| :---: | :---: |\r\n| Retinal Fundus Images        | Segmented Retinal Vessel Mask Images  |\r\n\r\n### Expected Input-Output for Stage 2\r\nThe input and output for Stage 2 are given below.\r\n**Input**                                 | **Output** |\r\n| :---: | :---: |\r\n| Segmented Retinal Vessel Mask Images        | Synthetic Retinal Fundus Images  |\r\n\r\n### Software Requirements\r\nNote that this reference kit implementation already provides the necessary scripts to set up the software requirements. \r\n1. Python - 3.9.7\r\n2. TensorFlow - 2.8.0\r\n\r\n\u003e **Note**: If you beginning to explore the reference kits on client machines such as a windows laptop, go to the \r\n[Running on Windows](#running-on-windows) section to ensure you are all set and come back here\r\n\r\n\u003e **Note**: The performance measurements were captured on Xeon-based processors. The instructions will work on WSL, \r\nhowever, some portions of the ref kits may run slower on a client machine, so utilize the flags supported to modify the \r\nepochs/batch size to run the training or inference faster. Additionally, performance claims reported may not be seen on a \r\nwindows based client machine.\r\n\r\n\u003e **Note**: In this reference kit implementation already provides the necessary conda environment configurations to set up the software requirements. \r\nTo utilize these environment scripts, first, install Anaconda/Miniconda by following the instructions at the following link  \r\n\u003e [Anaconda installation](https://docs.anaconda.com/anaconda/install/linux/)\r\n\r\n###  For cloning the repository  please execute below \r\nFirst clone the repository by executing the below command.\r\n```\r\ngit clone https://github.com/oneapi-src/image-data-generation\r\ncd image-data-generation\r\n```\r\n\r\n### Reference Solution setup\r\nThe below commands can be used to automate the creation of a conda environment for the execution of the algorithms.\r\n\r\n\u003eNote: It is assumed that the present **working directory is the root directory of this code repository**\r\n\r\nBelow are the developer environment used for this module on Azure. All the observations captured are based on these environment setups.\r\n\r\nThe below file is used to create a stock environment as follows:\r\n\r\n|     **YAML file**     | **Environment Name** |            **Configuration**            |\r\n|:---------------------:|----------------------|:---------------------------------------:|\r\n| `env/stock/syntheticimagegeneration.yml` | `syntheticimagegeneration_stock`        | Python=3.9.7 with Stock TensorFlow v2.8.0 without oneDNN enabled |\r\n\r\nCommand to create the stock environment\r\n```sh\r\nconda env create --file env/stock/syntheticimagegeneration_stock.yml\r\n```\r\nActivate the stock environment\r\n```sh\r\nconda activate syntheticimagegeneration_stock\r\n```\r\n\u003eNote: Please refer to known-issues section at the end in case of any libraries issues.\r\n\r\n#### Reference Solution Implementation\r\n\r\n### Data Preparation\r\nWe are using the 'High-Resolution Fundus (HRF) Image Database' from 'https://www5.cs.fau.de/research/data/fundus-images/'\r\n\r\n```sh\r\ncd data\r\nwget https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/all.zip\r\nunzip all.zip\r\n```\r\n\r\nThe folder structure Looks as below after the extraction of the dataset.\r\n```\r\n- data\r\n      - images\r\n      - manual1\r\n      - mask\r\n```\r\n\r\nThe downloaded dataset contains diseased and healthy retinal fundus images. Only the healthy retinal fundus images are considered in this refkit.\r\nOnce the dataset is downloaded, move the retinal fundus images and segmentation mask images into appropriate directories as required for \r\nmodel training and prediction. Run the shell script **gen_datasetfiles.sh** to perform the same.\r\n\r\n```sh\r\nchmod +x gen_datasetfiles.sh\r\n./gen_datasetfiles.sh\r\n```\r\n\r\nAfter the files are moved to their respective directories, preprocess the images so that they can be used as input for training by \r\nrunning the below python script.\r\n\r\n```\r\npython image_resize.py\r\n```\r\n\r\nThen use the shell script **split_train_test.sh** to split the dataset into training and testing images.\r\n\r\n```sh\r\nchmod +x split_train_test.sh\r\n./split_train_test.sh\r\ncd ..\r\n```\r\n\r\n### Model Building - Stage 1\r\n\r\nThe Python script given below needs to be executed to start training using the active environment enabled by using the above steps to set up the environment. \r\nThe details of the script and parameters are given below. The number of epochs, the path to save the model, and the path of the dataset file is given\r\nin the configuration file **cfg.json**. An example **cfg.json** file used during the training can be found [here.](data/cfg.json)\r\n\r\nUsers can train for more epochs to improve model performance, default number of epochs is \"10\".\r\n\r\nExecute the Python script as given below to start training the given dataset file.\r\n```shell\r\npython src/stage1/run_modeltraining.py\r\n```\r\nThe trained model will be saved in the **\"models\"** folder and dataset files will be read from **\"data/training\"** folder.\r\nBoth of these parameters can be set in the **cfg.json** file present in the data folder.\r\n\r\nThe user can collect the logs by redirecting the output to a file as illustrated below.\r\n\r\n```shell\r\npython src/stage1/run_modeltraining.py | tee \u003clog_file_name\u003e\r\n```\r\n\r\nThe output of the python script \u003ci\u003erun_modeltraining.py\u003c/i\u003e will be collected in the file \u003clog_file_name\u003e\r\n\r\n**Expected Output**\r\n\r\nThe trained model will be saved in the **\"models\"** folder.\r\n\r\n### Running Inference for Stage 1\r\n\r\nThe above script will train and save models to the **\"models\"** folder by default. The path can be changed by modifying \r\nthe saved model folder path in the configuration file **cfg.json**. To use this model to make predictions on new data, \r\nthe script **run_inference.py** needs to be used as given below.\r\n\r\n```shell\r\nusage: run_inference.py [-h] [-l LOAD_MODEL_PATH] [-b BATCH_SIZE] [-bf16 BF16] [-inc INC]\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -l LOAD_MODEL_PATH, --load_model_path LOAD_MODEL_PATH\r\n                        directory to load model from\r\n  -b BATCH_SIZE, --batch_size BATCH_SIZE\r\n                        inference batch size\r\n  -bf16 BF16, --bf16 BF16\r\n                        use 1 to enable bf16 capabilities, default is 0\r\n  -inc INC, --inc INC   Enable this flag for inc model inference. Default is 0\r\n ```\r\n\r\nThe inference will run on the image files stored in the **\"data/test\"** folder\r\n\r\n```shell\r\npython src/stage1/run_inference.py -l ./models/test_saved_model/ -b 32\r\n```\r\n\r\n**Expected Output**\r\n\r\nThe output would be generated by the Python script \u003ci\u003erun_inference.py\u003c/i\u003e which will capture the overall inference time in seconds.\r\nThe output can be redirected to a file as per the command above.\r\n\r\nThe lines below are from the sample output of the python script **run_inference.py** and it gives details of the inference of the model\r\n\r\n    ----------------------------------------\r\n    # Model Inference details:\r\n    # Average batch inference:\r\n    #   Time (in seconds): 5.85315097249349\r\n    #   Batch size: 64\r\n    ----------------------------------------\r\n\r\n### 4. Model Building - Stage 2\r\n\r\nThe pair of real fundus \u0026 segmented vessel images generated from Stage 1 is given as input for Stage 2.\r\n\r\n#### Data Loading\r\nBefore we start the training, we need to pre-process the data by converting the input image to tensors. Run the script **load_data_stage_2.py** with the arguments given below:\r\n\r\n```sh\r\nusage: load_data_stage_2.py [-h] [--count DATACOUNT] [--path PATH]  [--spit SPLIT]\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -count DATACOUNT, --count DATACOUNT\r\n                        The number of pairs of segmented \u0026 original images in the data folder\r\n  -path PATH, --path PATH\r\n                        The path of the data folder i.e. Stage 1 output\r\n  -split SPLIT, --split SPLIT\r\n                        The train - test split percentage  of data for stage2                   \r\n\r\n```\r\nRun the **load_data_stage_2.py** for generating the training and inference data tensors:\r\n```sh\r\ncd src/stage2\r\npython load_data_stage_2.py -count 15 -path ../../data/test/ -split 0.8\r\n```\r\n**original.npy** and **output.npy** tensors are generated for segmented masks and color images respectively for training.\r\n**original_test.npy** and **output_test.npy** tensors are generated for segmented masks and color images respectively for inference.\r\n\r\n#### Stage 2 Training\r\nOnce the data tensors are generated, run the Training script to generate models.\r\n```sh\r\nusage: training_stage_2.py [-h] -epochs EPOCHS [-bs BATCHSIZE] [-sz SIZE] [-mp MODEL_PATH] optional arguments:\r\n  -h, --help            show this help message and exit\r\n  -epochs EPOCHS, --epochs EPOCHS\r\n                        Define the number of epochs\r\n  -bs BATCHSIZE, --batchsize BATCHSIZE\r\n                        Define the batchsize for training\r\n  -sz SIZE, --size SIZE\r\n                        Dimension of image i.e.512\r\n  -mp MODEL_PATH, --model_path MODEL_PATH\r\n                        Path to save the trained models\r\n```\r\n\r\n**For example:**\r\n\r\n```sh\r\npython training_stage_2.py --epochs 10 -bs 8 -sz 512 -mp ../../models/Stage2_Models\r\n```\r\nThe generator, discriminator, and gan models will be saved in path : **models/Stage2_Models/**\r\n\r\n#### Stage 2 Frozen Graph Generation\r\n\r\nRun the **create_frozen_graph.py** to create a frozen graph from the \".h5\" model generated during the training phase.\r\n```sh\r\nusage: create_frozen_graph.py [-h] [-s SAVE_PATH] [-model MODEL_PATH]\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -s SAVE_PATH, --save_path SAVE_PATH\r\n                        Absolute path to the folder to save the frozen graph model\r\n  -model MODEL_PATH, --model_path MODEL_PATH\r\n                        Give the path of the Generator model to be used, which is in (.h5) file format.\r\n```\r\n\r\n**For example:**\r\n```sh\r\ncd ../../\r\npython src/stage2/create_frozen_graph.py -model ./models/Stage2_Models/pix2pix_g_epoch_10.h5 -s ./models/frozen\r\n```\r\n\u003e Note: above command will create \"saved_model.pb\" inside the \"./models/frozen\".\r\n\r\n#### Stage 2 Inference\r\nRun the **inference.py** to synthetically generate retinal fundus images.\r\n```sh\r\nusage: inference.py [-h] [-model MODEL_PATH] [-b BATCH_SIZE] [-nfrz] [-bf16 BF16]\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -model MODEL_PATH, --model_path MODEL_PATH\r\n                        Give the path of the Generator model to be used, it can either be keras model(.h5) or it can be frozen graph model (.pb).Note to set \"-nfrz\" if\r\n                        not using frozen graph models\r\n  -b BATCH_SIZE, --batch_size BATCH_SIZE\r\n                        inference batch size\r\n  -nfrz, --not_frozen_graph\r\n                        Sets True if model passed is not frozen graph ie.. \".pb\"\r\n  -bf16 BF16, --bf16 BF16\r\n                        use 1 to enable bf16 capabilities, default is 0\r\n```\r\n\r\n**For example:**\r\nInference using trained \".h5\" generator model\r\n```sh\r\npython src/stage2/inference.py -model ./models/Stage2_Models/pix2pix_g_epoch_10.h5 -nfrz -b 1\r\n```\r\nInference using converted trained frozen \".pb\" generator model\r\n```sh\r\npython src/stage2/inference.py -model ./models/frozen/saved_model.pb -b 1\r\n```\r\n\r\n**Expected Output**\r\n\r\nThe output would be generated by the Python script \u003ci\u003einference.py\u003c/i\u003e which will capture the overall inference time in seconds.\r\nThe output can be redirected to a file as per the command above.\r\n\r\nThe lines below are from the sample output of the python script **inference.py** and it gives details of the inference of the model\r\n\r\n\u003eNote: \"Generated image can be found in data/Inference_results\"\r\n\r\n\r\n    ----------------------------------------\r\n    # Model Inference details:\r\n    # Average batch inference:\r\n    #   Total Average Time (in seconds): 2.131949686471117\r\n    #   Batch size: 1\r\n    ----------------------------------------\r\n\r\n**Sample Output**\r\n\r\n![sample_output](assets/sample_output.png)\r\n\r\n## Optimizing the End To End solution with Intel® oneAPI components\r\n### Use Case End-To-End flow\r\n![Use_case_flow](assets/e2eflow_intel.png)\r\n\r\n### Optimized software requirements\r\n\r\n| **Package**                 | **Intel® Python**                   \r\n|:----------------------------| :---                               \r\n| Python                      | python=3.9.7\r\n| TensorFlow                  | tensorflow==2.11.0\r\n| Intel® Neural Compressor    | neural-compressor==2.0\r\n\r\n**Before setting up the Intel environment please deactivate the existing Stock environment by executing the below**\r\n```\r\nconda deactivate \r\n``` \r\n\r\n#### Intel Environment Creation\r\nThe below file is used to create an intel environment as follows:\r\n|     **YAML file**     | **Environment Name** |            **Configuration**            |\r\n|:---------------------:|----------------------|:---------------------------------------:|\r\n| `env/intel/syntheticimagegeneration_intel.yml` | `syntheticimagegeneration_intel`        | Python=3.9.7 with Stock TensorFlow v2.11.0 with oneDNN enabled by default |\r\n\r\n\r\nCommand to create the intel environment\r\n\r\n```sh\r\nconda env create --file env/intel/syntheticimagegeneration_intel.yml\r\n```\r\n\r\nActivate the intel environment\r\n```sh\r\nconda activate syntheticimagegeneration_intel\r\n```\r\n\r\n\u003eNote: Please refer to known-issues section at the end in case of any libraries issues.\r\n\r\n### Model Building Process for Stage 1\r\nThe Python script given below needs to be executed to start training using the active environment enabled by using the above steps to set up the environment. \r\nThe details of the script and parameters are given below. The number of epochs, the path to save the model, and the path of the dataset file is given\r\nin the configuration file **\"cfg.json\"**. An example **cfg.json** file used during the training can be found [here.](data/cfg.json)\r\n\r\nUsers can train for more epochs to improve model performance, default number of epochs is \"10\".\r\n\r\nExecute the python script as given below to start training the given dataset file.\r\n```shell\r\npython src/stage1/run_modeltraining.py\r\n\r\n```\r\nThe trained model will be saved in the **./models** folder and dataset files will be read from **./data/training folder**.\r\nBoth of these parameters can be set in the **cfg.json** file present in the data folder.\r\n\r\nThe user can collect the logs by redirecting the output to a file as illustrated below.\r\n\r\n```shell\r\npython src/stage1/run_modeltraining.py | tee \u003clog_file_name\u003e\r\n```\r\n\r\nThe output of the python script \u003ci\u003erun_modeltraining.py\u003c/i\u003e will be collected in the file \u003clog_file_name\u003e\r\n\r\n**Expected Output**\r\n\r\nThe trained model will be saved in the **./models** folder.\r\n\r\n### Running Inference for Stage 1\r\n\r\nThe above script will train and save models to the **./models** folder by default. The path can be changed by modifying \r\nthe saved model folder path in the configuration file **cfg.json**. To use this model to make predictions on new data, \r\nthe script **'run_inference.py'** need to be used as given below.\r\n\r\n```shell\r\nusage: run_inference.py [-h] [-l LOAD_MODEL_PATH] [-b BATCH_SIZE] [-bf16 BF16] [-inc INC]\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -l LOAD_MODEL_PATH, --load_model_path LOAD_MODEL_PATH\r\n                        directory to load model from\r\n  -b BATCH_SIZE, --batch_size BATCH_SIZE\r\n                        inference batch size\r\n  -bf16 BF16, --bf16 BF16\r\n                        use 1 to enable bf16 capabilities, default is 0\r\n  -inc INC, --inc INC   Enable this flag for inc model inference. Default is 0\r\n ```\r\n\r\nThe inference will run on the image files stored in the **./data/test** folder\r\n\r\n```shell\r\nTF_NUM_INTEROP_THREADS=1 TF_NUM_INTRAOP_THREADS=4 numactl -C 0,2,4,6 -m 0 python src/stage1/run_inference.py -l ./models/test_saved_model/ -b 32\r\n```\r\n\r\n**Expected Output**\r\n\r\nThe output would be generated by the python script \u003ci\u003erun_inference.py\u003c/i\u003e which will capture the overall inference time in seconds.\r\nThe output can be redirected to a file as per the command above.\r\n\r\nThe lines below are from the sample output of the python script **run_inference.py** and it gives details of the inference of the model\r\n\r\n    ----------------------------------------\r\n    # Model Inference details:\r\n    # Average batch inference:\r\n    #   Time (in seconds): 5.85315097249349\r\n    #   Batch size: 64\r\n    ----------------------------------------\r\n\r\nThe pair of real fundus \u0026 segmented images generated from Stage 1 is given as input for Stage 2.\r\n\r\n#### Stage 1 Intel® Neural Compressor\r\n\r\n### Step 1 : Intel® Neural Compressor Quantization\r\n\r\n```sh\r\nusage: inc_quantization.py [-h] -model MODEL_PATH [-o OUT_PATH] [-d DATA_PATH]\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -model MODEL_PATH, --model_path MODEL_PATH\r\n                        Give the path of the Generator model to be used, which is a saved model (.pb)\r\n  -o OUT_PATH, --out_path OUT_PATH\r\n                        Output quantized model will be save as 'saved_model.pb'in stage1_quantized_model.\r\n  -d DATA_PATH, --data_path DATA_PATH\r\n                        Absolute path to the dataset folder containing npy files, created while preprocessing.\r\n```\r\n\r\nCommand to run quantization\r\n\r\n```sh\r\npython src/stage1/inc_quantization.py -model ./models/test_saved_model/ -o ./models/stage1_quantized_model -d ./\r\n```\r\n\r\n### Step 2 : Intel® Neural Compressor Inference\r\n\r\nCommand to run inference using quantized model\r\n\r\n```sh\r\nTF_NUM_INTEROP_THREADS=1 TF_NUM_INTRAOP_THREADS=4 numactl -C 0,2,4,6 -m 0 python src/stage1/run_inference.py -l ./models/stage1_quantized_model/ -inc 1 -b 32\r\n```\r\n\r\n#### Data Loading\r\nBefore we start the training, we need to pre-process the data by converting the input image to tensors. Run the script **load_data_stage_2.py** with the arguments given below:\r\n\r\n```sh\r\nusage: load_data_stage_2.py [-h] [--count DATACOUNT] [--path PATH]  [--spit SPLIT]\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -count DATACOUNT, --count DATACOUNT\r\n                        The number of pairs of segmented \u0026 original images in the data folder\r\n  -path PATH, --path PATH\r\n                        The path of the data folder i.e. Stage 1 output\r\n  -split SPLIT, --split SPLIT\r\n                        The train - test split percentage  of data for stage2                   \r\n\r\n```\r\nRun the **load_data_stage_2.py** for generating the training and inference data tensors:\r\n```sh\r\ncd src/stage2\r\npython load_data_stage_2.py -count 15 -path ../../data/test/ -split 0.8\r\n```\r\n**original.npy** and **output.npy** tensors are generated for segmented masks and color images respectively for training.\r\n**original_test.npy** and **output_test.npy** tensors are generated for segmented masks and color images respectively for inference.\r\n\r\n#### Stage 2 Training\r\nOnce the data tensors are generated, run the training script to generate models.\r\n```sh\r\nusage: training_stage_2.py [-h] -epochs EPOCHS [-bs BATCHSIZE] [-sz SIZE] [-mp MODEL_PATH] optional arguments:\r\n  -h, --help            show this help message and exit\r\n  -epochs EPOCHS, --epochs EPOCHS\r\n                        Define the number of epochs\r\n  -bs BATCHSIZE, --batchsize BATCHSIZE\r\n                        Define the batchsize for training\r\n  -sz SIZE, --size SIZE\r\n                        Dimension of image i.e.512\r\n  -mp MODEL_PATH, --model_path MODEL_PATH\r\n                        Path to save the trained models\r\n```\r\n**For example:**\r\n```sh\r\npython training_stage_2.py --epochs 10 -bs 8 -sz 512 -mp ../../models/Stage2_Models\r\n```\r\nThe generator, discriminator, and gan models will be saved in the path of the present working directory: **models/Stage2_Models/**\r\n\r\n#### Stage 2 Frozen Graph Generation\r\n\r\nRun the **create_frozen_graph.py** to create a frozen graph from the \".h5\" model generated during the training phase.\r\n```sh\r\nusage: create_frozen_graph.py [-h] [-s SAVE_PATH] [-model MODEL_PATH]\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -s SAVE_PATH, --save_path SAVE_PATH\r\n                        Absolute path to the folder to save the frozen graph model\r\n  -model MODEL_PATH, --model_path MODEL_PATH\r\n                        Give the path of the Generator model to be used, which is in (.h5) file format.\r\n```\r\n\r\n**For example:**\r\n```sh\r\ncd ../../\r\npython src/stage2/create_frozen_graph.py -model ./models/Stage2_Models/pix2pix_g_epoch_10.h5 -s ./models/frozen\r\n```\r\n\u003e Note: above command will create \"saved_model.pb\" inside the \"./models/frozen\".\r\n\r\n#### Stage 2 Inference\r\nRun the **inference.py** to synthetically generate retinal fundus images.\r\n```sh\r\nusage: inference.py [-h] [-model MODEL_PATH] [-b BATCH_SIZE] [-nfrz] [-bf16 BF16]\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -model MODEL_PATH, --model_path MODEL_PATH\r\n                        Give the path of the Generator model to be used, it can either be keras model(.h5) or it can be frozen graph model (.pb).Note to set \"-nfrz\" if\r\n                        not using frozen graph models\r\n  -b BATCH_SIZE, --batch_size BATCH_SIZE\r\n                        inference batch size\r\n  -nfrz, --not_frozen_graph\r\n                        Sets True if model passed is not frozen graph ie.. \".pb\"\r\n  -bf16 BF16, --bf16 BF16\r\n                        use 1 to enable bf16 capabilities, default is 0\r\n```\r\n\r\n**For example:**\r\n\r\nInference using trained \".h5\" generator model\r\n```sh\r\nTF_NUM_INTEROP_THREADS=1 TF_NUM_INTRAOP_THREADS=4 numactl -C 0,2,4,6 -m 0 python src/stage2/inference.py -model ./models/Stage2_Models/pix2pix_g_epoch_10.h5 -nfrz -b 1\r\n```\r\nInference using converted trained frozen \".pb\" generator model\r\n```sh\r\nTF_NUM_INTEROP_THREADS=1 TF_NUM_INTRAOP_THREADS=4 numactl -C 0,2,4,6 -m 0 python src/stage2/inference.py -model ./models/frozen/saved_model.pb -b 1\r\n```\r\n\r\n**Expected Output**\r\n\r\nThe output would be generated by the Python script \u003ci\u003einference.py\u003c/i\u003e which will capture the overall inference time in seconds.\r\nThe output can be redirected to a file as per the command above.\r\n\r\nThe lines below are from the sample output of the python script **inference.py** and it gives details of the inference of the model\r\n\r\n\u003eNote: \"Generated image can be found in data/Inference_results\"\r\n\r\n    ----------------------------------------\r\n    # Model Inference details:\r\n    # Average batch inference:\r\n    #   Total Average Time (in seconds): 2.131949686471117\r\n    #   Batch size: 1\r\n    ----------------------------------------\r\n\r\n**Sample Output**\r\n\r\n![sample_output](assets/sample_output.png)\r\n\r\n#### Stage 2 Intel® Neural Compressor\r\n\r\n### Step 1 : Intel® Neural Compressor Quantization\r\n\r\n```sh\r\nusage: inc_quantization.py [-h] -model MODEL_PATH [-o OUT_PATH] [-d DATA_PATH]\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -model MODEL_PATH, --model_path MODEL_PATH\r\n                        Give the path of the Generator model to be used, which is a frozen model (.pb)\r\n  -o OUT_PATH, --out_path OUT_PATH\r\n                        Output quantized model will be save as 'quantized_SyncImgGen.pb'.\r\n  -d DATA_PATH, --data_path DATA_PATH\r\n                        Absolute path to the dataset folder containing npy files, created while preprocessing.\r\n```\r\n\r\nCommand to run quantization\r\n\r\n```sh\r\npython src/stage2/intel_neural_compressor/inc_quantization.py -model ./models/frozen/saved_model.pb -o ./models/inc -d ./\r\n```\r\n\r\n### Step 2 : Intel® Neural Compressor Inference\r\n\r\nCommand to run inference using quantized model\r\n\r\n```sh\r\nTF_NUM_INTEROP_THREADS=1 TF_NUM_INTRAOP_THREADS=4 numactl -C 0,2,4,6 -m 0 python src/stage2/inference.py -model ./models/inc/quantized_SyncImgGen.pb -b 1\r\n```\r\n\r\n### **Performance Observations**\r\n\r\n## Inference Stock TensorFlow v2.8.0 without oneDNN enabled  vs Stock TensorFlow v2.11.0 with oneDNN enabled by default for Stage 1.\r\n\r\n![image](assets/BenchmarkStage1InferenceGraph.png)\r\n\r\n\u003eNote: Stage1 inference benchmarking is based on saved model format models.\r\n\r\n## Inference Stock TensorFlow v2.8.0 without oneDNN enabled vs Stock TensorFlow v2.11.0 with oneDNN enabled by default vs Intel® Neural Compressor 2.0 for Stage 2.\r\n\r\n![image](assets/inference_stage_2.png)\r\n\r\n\u003eNote: Stage2 inference benchmarking is based on frozen graph format models.\r\n\r\n\u003cbr\u003e**Key Takeaway**\u003cbr\u003e\r\n- Stock TensorFlow v2.11.0 with oneDNN enabled by default offers batch inference time speed-up up to 1.29x compared to stock \r\nTensorFlow 2.8.0 for different batch sizes for Stage 1.\r\n- Stock TensorFlow v2.11.0 with oneDNN enabled by default and Intel® Neural Compressor 2.0 batch inference time speed-up up to 1.33x compared to stock \r\nTensorFlow 2.8.0 for different batch sizes for Stage 1.\r\n- Stock TensorFlow v2.11.0 with oneDNN enabled by default offers batch inference time speed-up up to 1.11 compared to stock \r\nTensorFlow 2.8.0 for different batch sizes for Stage 2.\r\n- Stock TensorFlow v2.11.0 with oneDNN enabled by default and Intel® Neural Compressor 2.0 batch inference time speed-up up to 1.25x compared to stock \r\nTensorFlow 2.8.0 for different batch sizes for Stage 2.\r\n\r\n#### **Conclusion**\r\nSynthetic image generation has huge potential in present and in the future also due to the multiple applications it caters to. The search ends with image generation \r\nwhen augmenting images for training in case of the imbalanced dataset and when the dataset size is very small, for medical image synthesis due to a shortage\r\nof real-world data and for rare medical conditions it becomes difficult to obtain the dataset required. Like ways there are an enormous number of use \r\ncases that require synthetic image generation. In all these scenarios, efforts have been made to use AI for solving these tasks in fewer data regimes.\r\nSynthetic image generation allows one to explore and identify significant actionable insights in a wide variety of application domains.\r\n\r\nTo build a solution for synthetic image generation using the Deep Learning approach, data scientists will need to train the models with \r\nsubstantial large datasets and run inference more frequently. They would need to derive actionable insights from the model predictions. \r\nThis requires a lot of training and retraining, making their job tedious. The ability to accelerate training will allow them to train \r\ntheir models more frequently to achieve better accuracy. Additionally, faster inference will allow them to run predictions in real-time \r\nscenarios more frequently. Thus, speeding up the entire DL pipeline.\r\n\r\nThis reference kit implementation provides a performance-optimized guide around synthetic image generation use cases that can \r\neasily be scaled across similar use cases in various domains.\r\n\r\n**Note on Performance Gain:**\r\n\r\n The performance gain is dependent on multiple factors such as hardware infrastructure, hyperparameters, trained models, data size and \r\n structure, etc. The hardware chosen here is a typical VM configuration that customers use and the rest of the parameters are choices \r\n governed by the dataset as well as the computational capability. Changing any of these, which users are free to do, will change the \r\n performance numbers. The main objective of this reference kit is to demonstrate that a deep-learning model using \r\n Intel-optimized packages will speed up that process. The presented results confirm that.\r\n\r\n\r\n## Notices \u0026 Disclaimers\r\nPerformance varies by use, configuration, and other factors. Learn more on the [Performance Index site](https://edc.intel.com/content/www/us/en/products/performance/benchmarks/overview/). \r\nPerformance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates.  \r\nSee backup for configuration details. No product or component can be secure. Your costs and results may vary. Intel technologies \r\nmay require enabled hardware, software, or service activation.\u003cbr\u003e\r\n© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and \r\nbrands may be claimed as the property of others.\r\n\r\nTo the extent that any public or non-Intel datasets or models are referenced by or accessed using tools or code on this site those datasets or models are provided by the third party indicated as the content source. Intel does not create the content and does not warrant its accuracy or quality. By accessing the public content, or using materials trained on or with such content, you agree to the terms associated with that content and that your use complies with the applicable license.\r\n \r\nIntel expressly disclaims the accuracy, adequacy, or completeness of any such public content, and is not liable for any errors, omissions, or defects in the content, or for any reliance on the content. Intel is not liable for any liability or damages relating to your use of public content.\r\n\r\n## Appendix\r\n\r\n### **Running on Windows**\r\n\r\nThe reference kits commands are linux based, in order to run this on Windows, goto Start and open WSL and follow the same steps as running on a linux machine starting from git clone instructions. If WSL is not installed you can [install WSL](https://learn.microsoft.com/en-us/windows/wsl/install).\r\n\r\n\u003e **Note** If WSL is installed and not opening, goto Start ---\u003e Turn Windows feature on or off and make sure Windows Subsystem for Linux is checked. Restart the system after enabling it for the changes to reflect.\r\n\r\n### Experiment Setup\r\n- **Date Testing Performed:** March 2023\r\n- **Configuration Details and Workload Setup:** Azure D8v5 (Intel® Xeon® Platinum 8370C CPU @ 2.80GHz), 1 Socket, 4 Cores per Socket, \r\n2 Threads per Core, Turbo:On, Total Memory: 32 GB, OS: Ubuntu 20.04, Kernel: Linux 5.15.0-1019-azure, Software: TensorFlow 2.11.0, TensorFlow 2.8.0, \r\nIntel® Neural Compressor 2.0\r\n- **Testing performed by** Intel Corporation\r\n\r\n| Platform                          | Microsoft Azure: Standard_D8_v5 (Ice Lake)\u003cbr\u003eUbuntu 20.04\r\n| :---                              | :---\r\n| Hardware                          | Azure Standard_D8_V5\r\n| CPU cores                         | 8\r\n| Memory                            | 32GB\r\n| Software                          | TensorFlow v2.11.0 with oneDNN eanbled by default, Intel® Neural Compressor v2.0 \u003cbr\u003e\r\n| What you will learn               | oneDNN eanbled TensorFlow performance advantage over the stock packages\r\n\r\n**Operating System:**\r\nWe recommend using a Linux OS for executing the reference solution presented here, e.g. RHEL or Ubuntu.\r\n\r\n### Known Issues\r\n\r\n1. Environment Creation Issue: Could not build wheels for pycocotools\r\n\r\n    **Issue:**\r\n      When creating an intel environment using `syntheticimagegeneration_intel` \r\n      ```\r\n      error: command 'gcc' failed: No such file or directory\r\n      ERROR: Could not build wheels for pycocotools\r\n      ```\r\n\r\n    **Solution:**\r\n\r\n    Install gcc.  For Ubuntu, this will be: \r\n\r\n      ```bash\r\n      apt install gcc\r\n      ```\r\n\r\n2. ImportError: libGL.so.1: cannot open shared object file: No such file or directory\r\n   \r\n    **Issue:**\r\n      While running the script \"python src/run_quantize_inc.py\":,\r\n      ```\r\n      ImportError: libGL.so.1: cannot open shared object file: No such file or directory\r\n      or\r\n      libgthread-2.0.so.0: cannot open shared object file: No such file or directory\r\n      ```\r\n\r\n    **Solution:**\r\n\r\n      Install the libgl11-mesa-glx and libglib2.0-0 libraries. For Ubuntu this will be:\r\n\r\n      ```bash\r\n     sudo apt install libgl1-mesa-glx\r\n     sudo apt install libglib2.0-0\r\n      ```\r\n3. Numactl not found\r\n\r\n    **Issue:**\r\n      Could not run numactl command due to numactl not found error, during inference stage\r\n\r\n    **Solution:**\r\n\r\n    Install numactl.  For Ubuntu, this will be: \r\n\r\n      ```bash\r\n      apt install numactl\r\n      ```\r\n","funding_links":[],"categories":["Table of Contents"],"sub_categories":["AI - Frameworks and Toolkits"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foneapi-src%2Fimage-data-generation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foneapi-src%2Fimage-data-generation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foneapi-src%2Fimage-data-generation/lists"}