{"id":16505215,"url":"https://github.com/qidiso/mxnet-retrain","last_synced_at":"2025-07-26T21:37:30.704Z","repository":{"id":153113315,"uuid":"145223834","full_name":"qidiso/mxnet-retrain","owner":"qidiso","description":"Create mxnet finetuner (retrain) for mac/linux ,no need install docker and supports CPU, GPU(eGpu/cudnn).support the inception,resnet ,squeeznet,mobilenet...","archived":false,"fork":false,"pushed_at":"2018-08-19T03:16:24.000Z","size":2272,"stargazers_count":32,"open_issues_count":0,"forks_count":29,"subscribers_count":26,"default_branch":"master","last_synced_at":"2025-03-18T20:49:02.173Z","etag":null,"topics":["fine-tuning","gpu","inception","mac","mobienet","mxnet","nin","resnet","retrain","squeeznet"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qidiso.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-08-18T13:51:50.000Z","updated_at":"2021-05-06T04:24:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"2d2a662f-47b8-4198-9414-bdda0aeaf593","html_url":"https://github.com/qidiso/mxnet-retrain","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qidiso%2Fmxnet-retrain","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qidiso%2Fmxnet-retrain/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qidiso%2Fmxnet-retrain/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qidiso%2Fmxnet-retrain/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qidiso","download_url":"https://codeload.github.com/qidiso/mxnet-retrain/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245108266,"owners_count":20562018,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fine-tuning","gpu","inception","mac","mobienet","mxnet","nin","resnet","retrain","squeeznet"],"created_at":"2024-10-11T15:09:53.926Z","updated_at":"2025-03-23T13:32:05.996Z","avatar_url":"https://github.com/qidiso.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mxnet-finetuner-for-mac-no-docker\nCreate mxnet fine tuner for mac ,no need install docker and support GPU(egpu).\n## Prerequisites\npython 2.7 mxnet(cpu) or mxnet-cu80 (you need setup egpu for mac)\nRUN pip install \\\n  attrdict \\\n  awscli \\\n  jupyter \\\n  matplotlib \\\n  nose \\\n  nose-timer \\\n  numpy \\\n  opencv-python \\\n  pandas \\\n  pandas_ml \\\n  Pillow \\\n  pylint \\\n  pyyaml \\\n  requests \\\n  seaborn \\\n  sklearn-pandas \\\n  slackclient \\\n  tqdm\n  \n  ## Setup\n\n  \n  ### 1. Arrange images into their respective directories\n\nA training data directory (`images/train`), validation data directory (`images/valid`), and test data directory (`images/test`) should containing one subdirectory per image class.\n\nFor example, arrange training, validation, and test data as follows.\n\n```\nimages/\n    train/\n        airplanes/\n            airplane001.jpg\n            airplane002.jpg\n            ...\n        watch/\n            watch001.jpg\n            watch002.jpg\n            ...\n    valid/\n        airplanes/\n            airplane101.jpg\n            airplane102.jpg\n            ...\n        watch/\n            watch101.jpg\n            watch102.jpg\n            ...\n    test/\n        airplanes/\n            airplane201.jpg\n            airplane202.jpg\n            ...\n        watch/\n            watch201.jpg\n            watch202.jpg\n            ...\n```\n\n\n### 2. Edit config/config.yml\n\nEdit `config.yml` as you like.\n\nFor example\n```\ncommon:\n  num_threads: 4\n  gpus: 0\n\ndata:\n  quality: 100\n  shuffle: 1\n  center_crop: 0\n\nfinetune:\n  models:\n    - imagenet1k-resnet-50\n  optimizers:\n    - sgd\n  num_epochs: 30\n  lr: 0.0001\n  lr_factor: 0.1\n  lr_step_epochs: 10,20\n  mom: 0.9\n  wd: 0.00001\n  batch_size: 10\n```\n\n### 3. Do Fine-tuning\n\n```\n$ sh finetune.sh\n```\n\nmxnet-finetuner will automatically execute the followings according to `config.yml`.\n\n- Create RecordIO data from images\n- Download pretrained models\n- Replace the last fully-connected layer with a new one that outputs the desired number of classes\n- Data augumentaion\n- Do Fine-tuning\n- Make training accuracy graph\n- Make confusion matrix\n- Upload training accuracy graph and confusion matrix to Slack\n\nTraining accuracy graph and/or confusion matrix are save at `logs/` directory.  \nTrained models are save at `model/` directory.\n\nTrained models are saved with the following file name for each epoch.\n```\nmodel/201705292200-imagenet1k-nin-sgd-0000.params\n```\n\nIf you want to upload results to Slack, set `SLACK_API_TOKEN` environment variable and edit `config.yml` as below.\n```\nfinetune:\n  train_accuracy_graph_slack_upload: 1\ntest:\n  confusion_matrix_slack_upload: 1\n```\n\n![p1](mxnet-retain.png)\ncontinue:\n![p1](mxnet-retain1.png)\n### 4. Predict with trained models\n\nSelect the trained model and epoch you want to use for testing and edit `config.yml`\n\nIf you want to use `model/201705292200-imagenet1k-nin-sgd-0001.params`, edit `config.yml` as blow.\n\n```\ntest:\n  model: 201705292200-imagenet1k-nin-sgd-0001\n```\n\nWhen you want to use the latest highest validation accuracy trained model, edit `config.yml` as below.\n\n```\ntest:\n  use_latest: 1\n```\n\nIf set this option, `model` is ignored.\n\nWhen you are done, you can predict with the following command\n\n```\n$ docker-compose run finetuner test\n```\n\nPredict result and classification report and/or confusion matrix are save at `logs/` directory.\n\n\n## Available pretrained models\n\n|model                          |pretrained model name            |\n|:------------------------------|:--------------------------------|\n|CaffeNet                       |imagenet1k-caffenet              |\n|SqueezeNet                     |imagenet1k-squeezenet            |\n|NIN                            |imagenet1k-nin                   |\n|VGG16                          |imagenet1k-vgg16                 |\n|Inception-BN                   |imagenet1k-inception-bn          |\n|ResNet-50                      |imagenet1k-resnet-50             |\n|ResNet-152                     |imagenet1k-resnet-152            |\n|Inception-v3                   |imagenet1k-inception-v3          |\n|DenseNet-169                   |imagenet1k-densenet-169          |\n|SE-ResNeXt-50                  |imagenet1k-se-resnext-50         |\n\nTo use these pretrained models, specify the following pretrained model name in `config.yml`.\n\nFor details, please check [Available pretrained models](docs/pretrained_models.md)\n\n\n## Available optimizers\n\n- SGD\n- NAG\n- RMSProp\n- Adam\n- AdaGrad\n- AdaDelta\n- Adamax\n- Nadam\n- DCASGD\n- SGLD\n- Signum\n- FTML\n- Ftrl\n\nTo use these optimizers, specify the optimizer name in lowercase in `config.yml`.\n\n\n## Benchmark (Speed and Memory Footprint)\n\nSingle TITAN X (Maxwell) with batch size 40\n\n|Model       |speed (images/sec)|memory (MiB)|\n|:-----------|:-----------------|:-----------|\n|CaffeNet    |1077.63           |716         |\n|ResNet-50   |111.04            |5483        |\n|Inception-V3|82.34             |6383        |\n|ResNet-152  |48.28             |11330       |\n\nFor details, please check [Benchmark](docs/benchmark.md)\n\n## Utilities\n\n### counter.sh\n\nCount the number of files in each subdirectory.\n\n```\n$ counter.sh testdir\ntestdir contains 4 directories\nLeopards    197\nMotorbikes  198\nairplanes   199\nwatch       200\n```\n\n### move_images.sh\n\nMove the specified number of jpeg images from the target directory to the output directory while maintaining the directory structure.\n\n```\n$ move_images.sh 20 testdir newdir\nprocessing Leopards\nprocessing Motorbikes\nprocessing airplanes\nprocessing watch\n$ counter.sh newdir\nnewdir contains 4 directories\nLeopards    20\nMotorbikes  20\nairplanes   20\nwatch       20\n```\n\n### Prepare sample images for fine-tuning\n\nDownload [Caltech 101] dataset, and split part of it into the `example_images` directory.\n\n```\n$ caltech101_prepare.sh\n```\n\n- `example_images/train` is train set of 60 images for each classes\n- `example_images/valid` is validation set of 20 images for each classes\n- `example_imags/test` is test set of 20 images for each classes\n\n```\n$ counter.sh example_images/train\nexample_images/train contains 10 directories\nFaces       60\nLeopards    60\nMotorbikes  60\nairplanes   60\nbonsai      60\ncar_side    60\nchandelier  60\nhawksbill   60\nketch       60\nwatch       60\n```\n\nWith this data you can immediately try fine-tuning.\n\n```\n$ caltech101_prepare.sh\n$ rm -rf images\n$ mv exmaple_images images\n$ sh finetune.sh\n```\n\n\n\n## Misc\n\n### How to freeze layers during fine-tuning\n\nIf you set the number of target layer to `finetune.num_active_layers` in `config.yml` as below, only layers whose number is not greater than the number of the specified layer will be train.\n\n```\nfinetune:\n  models:\n    - imagenet1k-nin\n  optimizers:\n    - sgd\n  num_active_layers: 6\n```\n\nThe default for `finetune.num_active_layers` is `0`, in which case all layers are trained.\n\nIf you set `1` to `finetune.num_active_layers`, only the last fully-connected layers are trained.\n\nYou can check the layer numbers of various pretrained models with `num_layers` command.\n\n```\n$ sh finetune.sh num_layers \u003cpretrained model name\u003e\n```\n\nFor details, please check [How to freeze layers during fine-tuning](docs/freeze_layers.md)\n\n\n### Training from scratch\n\nEdit `config.yml` as below.\n\n```\nfinetune:\n  models:\n    - scratch-alexnet\n```\n\nYou can also run fine-tuning and training from scratch together.\n\n```\nfinetune:\n  models:\n    - imagenet1k-inception-v3\n    - scratch-inception-v3\n```\n\nFor details, please check [Available models training from scratch](docs/train_from_scratch.md)\n\n## Refrence\n\nMxnet-finetuner(https://github.com/knjcode/mxnet-finetuner)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqidiso%2Fmxnet-retrain","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqidiso%2Fmxnet-retrain","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqidiso%2Fmxnet-retrain/lists"}