{"id":13490128,"url":"https://github.com/pudae/kaggle-hpa","last_synced_at":"2025-03-28T05:32:10.788Z","repository":{"id":145892648,"uuid":"165990648","full_name":"pudae/kaggle-hpa","owner":"pudae","description":"Code for 3rd place solution in Kaggle Human Protein Atlas Image Classification Challenge.","archived":false,"fork":false,"pushed_at":"2019-01-16T07:12:01.000Z","size":8107,"stargazers_count":248,"open_issues_count":2,"forks_count":91,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-08-01T19:33:11.049Z","etag":null,"topics":["kaggle","python","pytorch"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pudae.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-01-16T06:57:34.000Z","updated_at":"2024-06-22T15:26:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"7d95e602-1cb8-4998-b190-781f101f96bd","html_url":"https://github.com/pudae/kaggle-hpa","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pudae%2Fkaggle-hpa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pudae%2Fkaggle-hpa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pudae%2Fkaggle-hpa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pudae%2Fkaggle-hpa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pudae","download_url":"https://codeload.github.com/pudae/kaggle-hpa/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222351205,"owners_count":16970636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kaggle","python","pytorch"],"created_at":"2024-07-31T19:00:41.239Z","updated_at":"2024-10-31T03:31:17.366Z","avatar_url":"https://github.com/pudae.png","language":"Python","funding_links":[],"categories":["Image Classification"],"sub_categories":["[[2019] Human Protein Atlas Image Classification](https://www.kaggle.com/c/human-protein-atlas-image-classification/)"],"readme":"# kaggle-hpa-image-classification\nCode for 3rd place solution in Kaggle Human Protein Atlas Image Classification Challenge.\n\nTo read the detailed solution, please, refer to [the Kaggle post](https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77320)\n\n## Hardware\nThe following specs were used to create the original solution.\n- Ubuntu 16.04 LTS\n- Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz\n- 3x NVIDIA TitanX \n\n## Reproducing Submission\nTo reproduct my submission without retrainig, do the following steps:\n1. [Installation](#installation)\n2. [Download Official Image](#download-official-image)\n3. [Make RGBY Images](#make-rgby-images) for official.\n4. [Download Pretrained models](#pretrained-models)\n5. [Inference](#inference)\n6. [Make Submission](#make-submission)\n\n## Installation\nAll requirements should be detailed in requirements.txt. Using Anaconda is strongly recommended.\n```\nconda create -n hpa python=3.6\nsource activate hpa\npip install -r requirements.txt\n```\n\n## Dataset Preparation\nAll required files except images are already in data directory.\nIf you generate CSV files (duplicate image list, split, leak.. ), original files are overwritten. The contents will be changed, but It's not a problem.\n\n### Prepare Images\nAfter downloading and converting images, the data directory is structured as:\n```\ndata\n  +- raw\n  |  +- train\n  |  +- test\n  |  +- external\n  +- rgby\n  |  +- train\n  |  +- test\n  |  +- external\n```\n#### Download Official Image\nDownload and extract *train.zip* and *test.zip* to *data/raw* directory.\nIf the Kaggle API is installed, run following command.\n```\n$ kaggle competitions download -c human-protein-atlas-image-classification -f train.zip\n$ kaggle competitions download -c human-protein-atlas-image-classification -f test.zip\n$ mkdir -p data/raw\n$ unzip train.zip -d data/raw/train\n$ unzip test.zip -d data/raw/test\n```\n\n#### Download External Images\nTo download external images, run following command. The external images will be located in *data/raw/external*\n```\n$ python tools/download.py\n```\n\n#### Make RGBY Images\nTo train or inference, converting to RGBY image is required. Run following commands.\n\nFor official:\n```\n$ python tools/make_rgby.py --input_dir=data/raw/train --output_dir=data/rgby/train\n$ python tools/make_rgby.py --input_dir=data/raw/test --output_dir=data/rgby/test\n```\nFor external:\n```\n$ python tools/make_rgby.py --input_dir=data/raw/external --output_dir=data/rgby/external\n```\n\n### Generate CSV files\n*You can skip this step. All CSV files are prepared in data directory.*\n\n#### Duplicated Image List\nThere are duplicated images. To search them, run following commands. *duplicates.ahash.csv* and *duplicates.phash.csv* will be generated.\n```\n$ python tools/find_duplicate_images.py\n```\n\n#### Split Dataset\nCreate 5 folds CV set. One for training, the other for searching augmentation. *split.stratified.[0-4].csv* and *split.stratified.small.[0-4].csv* will be generated.\n```\n$ python stratified_split.py\n$ python stratified_split.py --use_external=0\n```\n\n#### Search Data Leak\nTo learn more about data leak, please, refer to [this post](https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534). Following comand will create *data_leak.ahash.csv* and *data_leak.phash.csv*. [The other leak](https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/73395y) is already in *data* directory.\n```\n$ python find_data_leak.py\n```\n\n## Training\nIn configs directory, you can find configurations I used train my final models. My final submission is ensemble of resnet34 x 5, inception-v3 and se-resnext50, but ensemble of inception-v3 and se-resnext50's performance is better.\n\n### Search augmentation\nTo find suitable augmentation, 256x256 image and resnet18 are used.\nIt takes about 2 days on TitanX. The result(best_policy.data) will be located in *results/search* directory.\nThe policy that I used is located in *data* directory.\n```\n$ python train.py --config=configs/search.yml\n```\n\n### Train models\nTo train models, run following commands.\n```\n$ python train.py --config={config_path}\n```\nTo train all models, run `sh train.sh`\n\nThe expected training times are:\n\nModel | GPUs | Image size | Training Epochs | Training Time\n------------ | ------------- | ------------- | ------------- | -------------\nresnet34 | 1x TitanX | 512 | 40 | 16 hours\ninception-v3 | 3x TitanX | 1024 | 27 | 1day 15 hours\nse-resnext50 | 2x TitanX | 1024 | 22 | 2days 15 hours\n\n### Average weights\nTo average weights, run following commands.\n```\n$ python swa.py --config={config_path}\n```\nTo average weights of all models, simply run `sh swa.sh`\nThe averages weights will be located in *results/{train_dir}/checkpoint*.\n\n### Pretrained models\nYou can download pretrained model that used for my submission from [link](https://www.dropbox.com/s/qo65gw8kml5hgag/results.tar.gz?dl=0). Or run following command.\n```\n$ wget https://www.dropbox.com/s/qo65gw8kml5hgag/results.tar.gz\n$ tar xzvf results.tar.gz\n```\nUnzip them into results then you can see following structure:\n```\nresults\n  +- resnet34.0.policy\n  |  +- checkpoint\n  +- resnet34.1.policy\n  |  +- checkpoint\n  +- resnet34.2.policy\n  |  +- checkpoint\n  +- resnet34.3.policy\n  |  +- checkpoint\n  +- resnet34.4.policy\n  |  +- checkpoint\n  +- inceptionv3.attention.policy.per_image_norm.1024\n  |  +- checkpoint\n  +- se_resnext50.attention.policy.per_image_norm.1024\n  |  +- checkpoint\n```\n\n## Inference\nIf trained weights are prepared, you can create files that contains class probabilities of images.\n```\n$ python inference.py \\\n  --config={config_filepath} \\\n  --num_tta={number_of_tta_images, 4 or 8} \\\n  --output={output_filepath} \\\n  --split={test or test_val}\n```\nTo make submission, you must inference test and test_val splits. For example:\n```\n$ python inference.py --config=configs/resnet34.0.policy.yml --num_tta=8 --output=inferences/resnet34.0.test_val.csv --split=test_val\n$ python inference.py --config=configs/resnet34.0.policy.yml --num_tta=8 --output=inferences/resnet34.0.test.csv --split=test\n```\nTo inference all models, simply run `sh inference.sh`\n\n## Make Submission\nFollowing command will ensemble of all models and make submissions.\n```\n$ python make_submission.py\n```\nIf you don't want to use, modify *make_submission.py*.\nFor example, if you want to use inception-v3 and se-resnext50 then modify *test_val_filenames, test_filenames and weights* in *make_submission.py*.\n```\ntest_val_filenames = ['inferences/inceptionv3.0.test_val.csv',\n                      'inferences/se_resnext50.0.test_val.csv']\n                      \ntest_filenames = ['inferences/inceptionv3.0.test.csv',\n                  'inferences/se_resnext50.0.test.csv']\n                  \nweights = [1.0, 1.0]\n```\nThe command generate two files. One for original submission and the other is modified using data leak.\n- submissions/submission.csv\n- submissions/submission.csv.leak.csv\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpudae%2Fkaggle-hpa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpudae%2Fkaggle-hpa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpudae%2Fkaggle-hpa/lists"}