{"id":19899159,"url":"https://github.com/tootouch/normalization-test","last_synced_at":"2025-10-08T21:41:21.620Z","repository":{"id":160312292,"uuid":"634768290","full_name":"TooTouch/Normalization-Test","owner":"TooTouch","description":"Comparison normalization when using a pre-trained model","archived":false,"fork":false,"pushed_at":"2023-05-11T12:15:38.000Z","size":2276,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-27T18:15:43.113Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TooTouch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-01T06:13:10.000Z","updated_at":"2023-07-23T11:32:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"e3340019-d929-4000-aa27-4bccb6a3fba2","html_url":"https://github.com/TooTouch/Normalization-Test","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TooTouch/Normalization-Test","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TooTouch%2FNormalization-Test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TooTouch%2FNormalization-Test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TooTouch%2FNormalization-Test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TooTouch%2FNormalization-Test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TooTouch","download_url":"https://codeload.github.com/TooTouch/Normalization-Test/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TooTouch%2FNormalization-Test/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000706,"owners_count":26082837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T20:07:23.149Z","updated_at":"2025-10-08T21:41:21.598Z","avatar_url":"https://github.com/TooTouch.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Normalization-Test\nComparison normalization to pre-process images when using a pre-trained model\n\n**Motivation**\n\nIt is generally known that when using a pre-trained model for fine-tuning, the image for fine-tuning is normalized using the statistics of the data to be used for fine-tuning. This is because the data for fine-tuning is different from the distribution of the data used for pre-training. However, sometimes the data to be fine-tuned is normalized by the statistics of the data used for pre-training, and the performance is better or similar. Therefore, I would like to experiment to see in which cases this difference occurs.\n\n# Experiments\n\n**Baseline**\n\nIn this experiments, I used different types of CNN models, ResNet and DenseNet, and varied the size of the models to use them as baseline models.\n\n| model       |             #Params |\n|:------------|--------------------:|\n| ResNet18    |                 11M |\n| ResNet50    |                 24M |\n| DenseNet121 |                  7M |\n| DenseNet161 |                 27M |\n\n\n\n\n**Dataset**\n\nWe use four datasets for our experiments. We also use the mean and standard deviation to check the statistics after normalization, or the mean and standard deviation of ImageNet-1k used for model pre-training.\n\n\n\u003c **Table 1**. Dataset Description.  \u003e\n| Dataset           | #Images(trainset) | #Images(testset) | #Classes | Image Size  | Fine-tune stats.                                   | ImageNet-1k stats.                                    |\n|-------------------|-------------------|------------------|----------|-------------|---------------------------------------------------|-------------------------------------------------------|\n| CIFAR10           | 50,000            | 10,000           | 10       | 32 X 32 X 3 | mean: (0.00, 0.00, 0.00)\u003cbr\u003estd : (1.00, 1.00, 1.00) | mean: (0.03, 0.12, 0.18)\u003cbr\u003estd : (1.08, 1.09, 1.16)     |\n| CIFAR100          | 50,000            | 10,000           | 100      | 32 X 32 X 3 | mean: (0.00, 0.00, 0.00)\u003cbr\u003estd : (1.00, 1.00, 1.00) | mean: (0.10, 0.14, 0.16)\u003cbr\u003estd : (1.17, 1.15, 1.23)     |\n| SVHN              | 73,257            | 26,032           | 10       | 32 X 32 X 3 | mean: (0.00, 0.00, 0.00)\u003cbr\u003estd : (1.00, 1.00, 1.00) | mean: (-0.21, -0.06, 0.30)\u003cbr\u003estd : ( 0.87,  0.90, 0.88) |\n| Tiny ImageNet-200 | 100,000           | 10,000           | 200      | 64 X 64 X 3 | mean: (0.00, 0.00, 0.00)\u003cbr\u003estd : (1.00, 1.00, 1.00) | mean: (-0.38, -0.04, 0.33)\u003cbr\u003estd : ( 1.23,  1.20, 1.23) |\n\n\n**Training Settings**\n\nAll models are trained with `./default_configs`.\n\n```\nSEED: 223\n\nDATASET:\n  datadir: /datasets\n\nOPTIMIZER:\n  opt_name: Adam\n  lr: 0.001\n\nTRAINING:\n  batch_size: 256\n  test_batch_size: 256\n  epochs: 100\n  log_interval: 10\n  use_scheduler: true\n\nRESULT:\n  savedir: './saved_model'\n```\n\nthe statistics for each datasets to be fine-tuned and ImageNet-1k used to pre-trained can be find in `./stats.py`.\n\n\n```\ndataset_stats = {\n    \"imagenet\":{\n        \"num_classes\" : 1000,\n        \"img_size\"    : 224,\n        \"mean\"        : (0.485, 0.456, 0.406),\n        \"std\"         : (0.229, 0.224, 0.225)\n    },\n    \"cifar10\":{\n        \"num_classes\" : 10,\n        \"img_size\"    : 32,\n        \"mean\"        : (0.4914, 0.4822, 0.4465),\n        \"std\"         : (0.247, 0.2435, 0.2616)\n    },\n    \"cifar100\":{\n        \"num_classes\" : 100,\n        \"img_size\"    : 32,\n        \"mean\"        : (0.5071, 0.4867, 0.4408),\n        \"std\"         : (0.2675, 0.2565, 0.2761)\n    },\n    \"svhn\":{\n        \"num_classes\" : 10,\n        \"img_size\"    : 32,\n        \"mean\"        : (0.4377, 0.4438, 0.4728), \n        \"std\"         : (0.1980, 0.2010, 0.1970)\n    },\n    \"tiny_imagenet_200\":{\n        \"num_classes\" : 200,\n        \"img_size\"    : 64,\n        \"mean\"        : (0.4802, 0.4481, 0.3975), \n        \"std\"         : (0.2764, 0.2689, 0.2816)\n    }\n}\n```\n\n**Results**\n\n\n\u003cp align='cetner'\u003e\n    \u003cimg src=\"https://github.com/TooTouch/Normalization-Test/blob/main/assets/figure1.jpg?raw=true\"\u003e\n    \u003c \u003cstrong\u003eFigure 1\u003c/strong\u003e. Compare history of test accuracy based on normalization settings by model and dataset \u003e\n\u003c/p\u003e\n\n\nFigure 1 shows the accuracy history of the testset between different models and datasets. From the results, it seems that using normalization on the fine-tuning dataset converges to higher performance in most cases.\n\n\n\u003c **Table 2**. Accuracy(%) for four datasets. $\\textcolor{green}{green}$ indicates how much better the performance is than pre-trained setting. $\\textcolor{red}{red}$ indicates how much worse the performance is than pre-trained setting. \u003e\n\n| Model           | Setting           |   CIFAR10\u003cbr\u003eAccuracy(%) |  CIFAR100\u003cbr\u003eAccuracy(%) |      SVHN\u003cbr\u003eAccuracy(%) |       Tiny ImageNet-200\u003cbr\u003eAccuracy(%) |\n|:----------------|:------------------|---------------------:|----------------------:|---------------------:|-------------------------------:|\n| DenseNet121     | Fine-tuned         |   **89.05** ($\\textcolor{green}{+00.02}$) |        63.34 ($\\textcolor{red}{-00.24}$) |    **95.16** ($\\textcolor{green}{+00.25}$)|              **59.66** ($\\textcolor{green}{+00.42}$)|\n|                 | Pre-trained        |                89.03 |             **63.58** |                94.91 |                          59.24 |\n|                 |                   |                      |                       |                      |                                |\n| DenseNet161     | Fine-tuned         |         90.91 ($\\textcolor{red}{-00.15}$) |        67.68 ($\\textcolor{red}{-00.19}$) |    **95.51** ($\\textcolor{green}{+00.05}$)|              **61.90** ($\\textcolor{green}{+00.03}$)|\n|                 | Pre-trained        |            **91.06** |             **67.87** |                95.46 |                          61.87 |\n|                 |                   |                      |                       |                      |                                |\n| ResNet18        | Fine-tuned         |         87.88 ($\\textcolor{red}{-00.10}$) |    **61.67** ($\\textcolor{green}{+00.12}$) |        95.01 ($\\textcolor{red}{-00.16}$)|              **53.62** ($\\textcolor{green}{+00.73}$)|\n|                 | Pre-trained        |            **87.98** |                 61.55 |            **95.17** |                          52.89 |\n|                 |                   |                      |                       |                      |                                |\n| ResNet50        | Fine-tuned         |         89.78 ($\\textcolor{red}{-00.42}$) |    **67.58** ($\\textcolor{green}{+00.64}$) |        94.39 ($\\textcolor{red}{-00.09}$)|              **67.32** ($\\textcolor{green}{+00.16}$)|\n|                 | Pre-trained        |            **90.20** |                 66.94 |            **94.48** |                          67.16 |\n\n\nTable 1 compares the best accuracy for the testset. When comparing the results quantitatively, I found that using normalization based on the fine-tuning dataset resulted in higher performance in three out of four cases for CIFAR10 and higher performance in two out of four cases for CIFAR100 and SVHN. However, Tiny ImageNet-200 performed better with all of the fine-tuned settings.\n\nIn general, image normalization for model training is done to fit a normal distribution. Therefore, the pre-training model was also trained on images normalized to a normal distribution. However, in the case of tiny imagenet-200, when normalized to the statistics of imagenet-1k, it is farther from the normal distribution than the other datasets, so I assume that the fine-tuning setting yielded better results than the pretraining setting.\n\n\n\u003c **Table 3**. Accuracy(%) for four datasets and for normalization settings.\n| Model           | Setting           |   CIFAR10\u003cbr\u003eAccuracy(%) |   CIFAR100\u003cbr\u003eAccuracy(%) |   SVHN\u003cbr\u003eAccuracy(%) |   Tiny ImageNet-200\u003cbr\u003eAccuracy(%) |\n|:----------------|:------------------|-------------------------:|--------------------------:|----------------------:|-----------------------------------:|\n| DenseNet121     | finetune          |               **0.8905** |                    0.6334 |            **0.9516** |                         **0.5966** |\n|                 | pretrain          |            \u003cu\u003e0.8903\u003c/u\u003e |             \u003cu\u003e0.6358\u003c/u\u003e |                0.9491 |                      \u003cu\u003e0.5924\u003c/u\u003e |\n|                 | instance          |                   0.8768 |                    0.6056 |         \u003cu\u003e0.9515\u003c/u\u003e |                             0.5659 |\n|                 | minmax            |                   0.8889 |                **0.6367** |                0.9512 |                             0.5917 |\n|                 |                   |                          |                           |                       |                                    |\n| DenseNet161     | finetune          |            \u003cu\u003e0.9091\u003c/u\u003e |             \u003cu\u003e0.6768\u003c/u\u003e |            **0.9551** |                      \u003cu\u003e0.6190\u003c/u\u003e |\n|                 | pretrain          |               **0.9106** |                **0.6787** |         \u003cu\u003e0.9546\u003c/u\u003e |                             0.6187 |\n|                 | instance          |                   0.8994 |                    0.6572 |                0.9537 |                             0.6034 |\n|                 | minmax            |                   0.9087 |                    0.6763 |                0.9525 |                         **0.6245** |\n|                 |                   |                          |                           |                       |                                    |\n| ResNet18        | finetune          |                   0.8788 |                **0.6167** |         \u003cu\u003e0.9501\u003c/u\u003e |                         **0.5362** |\n|                 | pretrain          |               **0.8798** |             \u003cu\u003e0.6155\u003c/u\u003e |            **0.9517** |                      \u003cu\u003e0.5289\u003c/u\u003e |\n|                 | instance          |                   0.8655 |                    0.5764 |                0.9455 |                             0.5083 |\n|                 | minmax            |            \u003cu\u003e0.8797\u003c/u\u003e |                    0.6118 |                0.9497 |                             0.5254 |\n|                 |                   |                          |                           |                       |                                    |\n| ResNet50        | finetune          |                   0.8978 |                **0.6758** |                0.9439 |                         **0.6732** |\n|                 | pretrain          |            \u003cu\u003e0.9020\u003c/u\u003e |                    0.6694 |                0.9448 |                             0.6716 |\n|                 | instance          |                   0.8932 |                    0.6469 |            **0.9465** |                             0.6540 |\n|                 | minmax            |               **0.9023** |             \u003cu\u003e0.6750\u003c/u\u003e |         \u003cu\u003e0.9449\u003c/u\u003e |                      \u003cu\u003e0.6724\u003c/u\u003e |\n\nFollowing on from Table 2, we experimented with two additional image normalization methods in Table 3. The first instance setting performs normalization on a per-image basis, and the second performs MinMax normalization. The results were surprisingly good with minmax normalization in some cases.\n\n\n# Conclusion\n\nThe results were, unsurprisingly, strongly influenced by whether the image was normalized to a normal distribution. The general rule of thumb is to normalize to the distribution of the data we want to fine-tune. Sometimes I do it without bothering with normalization, and I see it in other repositories as well, so I need to reflect on it and make sure the model is trained well according to the canonicals.\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftootouch%2Fnormalization-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftootouch%2Fnormalization-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftootouch%2Fnormalization-test/lists"}