https://github.com/freedomofkeima/transfer-learning-anime
Transfer Learning for Anime Characters Recognition
https://github.com/freedomofkeima/transfer-learning-anime
anime classification computer-vision opencv python tensorflow transfer-learning transferlearning
Last synced: 3 months ago
JSON representation
Transfer Learning for Anime Characters Recognition
- Host: GitHub
- URL: https://github.com/freedomofkeima/transfer-learning-anime
- Owner: freedomofkeima
- Created: 2017-11-10T10:03:25.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2017-11-18T11:32:27.000Z (almost 8 years ago)
- Last Synced: 2024-04-17T21:47:42.447Z (over 1 year ago)
- Topics: anime, classification, computer-vision, opencv, python, tensorflow, transfer-learning, transferlearning
- Language: Python
- Homepage: https://freedomofkeima.com/moeflow/
- Size: 62.3 MB
- Stars: 187
- Watchers: 8
- Forks: 24
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Transfer Learning for Anime Characters
**Warning**: This repository size is quite big (approx. 100 MB) since it includes training and test images.
## Introduction
This repository is the continuation of [Flag #15 - Image Recognition for Anime Characters](http://freedomofkeima.com/blog/posts/flag-15-image-recognition-for-anime-characters).
In Flag #15, we can see that Transfer Learning works really well with 3 different anime characters: Nishikino Maki, Kotori Minami, and Ayase Eli.
In this experiment, we will try to push Transfer Learning further, by using 3 different anime characters which have hair color similarity: Nishikino Maki, Takimoto Hifumi, and Sakurauchi Riko.
This experiment has 3 main steps:
1. Utilize `lbpcascade_animeface` to recognize character face from each images
2. Resize each images to 96 x 96 pixels
3. Split images into training & test before creating the final model
`raw` directory contains 36 images for each characters (JPG & PNG format). The first 30 images are used for training while the last 6 images are used for test.
As an example, we got the following result after applying Step 1 (`cropped` directory is shown at the right side):
`lbpcascade_animeface` can detect character faces with an accuracy of around **83%**. Failed images are stored in `raw (unrecognized)` for future improvements.
Since we have 3 characters and 6 test images for each which are not part of training, `resized_for_test` contains 18 images in total. Surprisingly, almost **all characters** are detected properly!
**Update (Nov 13, 2017)**: See `animeface-2009` section below, which push face detection accuracy to **93**%.
## Requirements
- OpenCV (https://github.com/opencv/opencv)
- TensorFlow (https://github.com/tensorflow/tensorflow)
## Steps
1. The following command is used to populate `cropped` directory.
```
$ python bulk_convert.py raw/[character_name] cropped
```
2. The following command is used to populate `resized_for_training` & `resized_for_test` directory.
```
$ python bulk_resize.py cropped/[character_name] resized
```
After running the step above, you can decide how many images will be used in `resized_for_training` and how many images will be used in `resized_for_test`.
3. Re-train the Inception model by using transfer learning:
```
$ bazel-bin/tensorflow/examples/image_retraining/retrain --image_dir ~/transfer-learning-anime/resized_for_traning/
$ bazel build tensorflow/examples/image_retraining:label_image
```
4. At this point, the model is ready to use. We can run the following command to get the classification result:
```
$ bazel-bin/tensorflow/examples/image_retraining/label_image --graph=/tmp/output_graph.pb --labels=/tmp/output_labels.txt --output_layer=final_result:0 --image=$HOME/transfer-learning-anime/resized_for_test/[character name]/[image name]
```
If everything works properly, you will get the classification result. See [TensorFlow Documentation](https://www.tensorflow.org/tutorials/image_retraining) for more options.
Optionally, sample model can be downloaded by running `download_model.sh` script inside `models (example)` directory.
## Result Analysis
Initially, we run the experiment with 2 characters: Nishikino Maki and Takimoto Hifumi.
```
INFO:tensorflow:2017-11-10 08:50:36.151387: Step 3999: Train accuracy = 100.0%
INFO:tensorflow:2017-11-10 08:50:36.151592: Step 3999: Cross entropy = 0.002191
INFO:tensorflow:2017-11-10 08:50:36.210147: Step 3999: Validation accuracy = 100.0% (N=100)
INFO:tensorflow:Final test accuracy = 92.9% (N=14)
```
The result is as the following:
|Image|Classification| OK/NG |
| --- | --- | --- |
|  | nishikino maki (score = 0.99874)
takimoto hifumi (score = 0.00126) | OK |
|  | nishikino maki (score = 0.75519)
takimoto hifumi (score = 0.24481) | OK |
|  | nishikino maki (score = 0.99513)
takimoto hifumi (score = 0.00487) | OK |
|  | nishikino maki (score = 0.98629)
takimoto hifumi (score = 0.01371) | OK |
|  | nishikino maki (score = 0.99723)
takimoto hifumi (score = 0.00277) | OK |
|  | nishikino maki (score = 0.99695)
takimoto hifumi (score = 0.00305) | OK |
|Image|Classification| OK/NG |
| --- | --- | --- |
|  | takimoto hifumi (score = 0.63084)
nishikino maki (score = 0.36916) | OK|
|  | takimoto hifumi (score = 0.99728)
nishikino maki (score = 0.00272) | OK |
|  | takimoto hifumi (score = 0.99972)
nishikino maki (score = 0.00028) | OK |
|  | takimoto hifumi (score = 0.98852)
nishikino maki (score = 0.01148) | OK |
|  | takimoto hifumi (score = 0.99456)
nishikino maki (score = 0.00544) | OK |
|  | takimoto hifumi (score = 0.96630)
nishikino maki (score = 0.03370) | OK |
From the result above, 10 out of 12 have threshold > 0.95, while the lowest threshold is 0.63.
At this point, I decided to add Sakurauchi Riko, which is known for its similarity to Nishikino Maki.
```
INFO:tensorflow:2017-11-10 13:13:59.270717: Step 3999: Train accuracy = 100.0%
INFO:tensorflow:2017-11-10 13:13:59.270912: Step 3999: Cross entropy = 0.005526
INFO:tensorflow:2017-11-10 13:13:59.328139: Step 3999: Validation accuracy = 100.0% (N=100)
INFO:tensorflow:Final test accuracy = 80.0% (N=15)
```
With 3 similar characters, the result is as the following:
|Image|Classification| OK/NG |
| --- | --- | --- |
|  | nishikino maki (score = 0.99352)
sakurauchi riko (score = 0.00612)
takimoto hifumi (score = 0.00036) | OK |
|  | nishikino maki (score = 0.47391)
sakurauchi riko (score = 0.37913)
takimoto hifumi (score = 0.14696) | OK |
|  | nishikino maki (score = 0.95976)
sakurauchi riko (score = 0.02797)
takimoto hifumi (score = 0.01227) | OK |
|  | nishikino maki (score = 0.88851)
sakurauchi riko (score = 0.07526)
takimoto hifumi (score = 0.03623) | OK |
|  | nishikino maki (score = 0.99025)
sakurauchi riko (score = 0.00766)
takimoto hifumi (score = 0.00209) | OK |
|  | nishikino maki (score = 0.96782)
sakurauchi riko (score = 0.02783)
takimoto hifumi (score = 0.00435) | OK |
As you can see above, the similarity between Nishikino Maki and Sakurauchi Miko starts to lower down the confidence level of the resulted model. Nevertheless, all classifications are still correct, where 4 out of 6 maintain the threshold of > 0.95.
|Image|Classification| OK/NG |
| --- | --- | --- |
|  | takimoto hifumi (score = 0.86266)
nishikino maki (score = 0.13632)
sakurauchi riko (score = 0.00102) | OK |
|  | takimoto hifumi (score = 0.87614)
sakurauchi riko (score = 0.12334)
nishikino maki (score = 0.00051) | OK |
|  | takimoto hifumi (score = 0.99964)
sakurauchi riko (score = 0.00023)
nishikino maki (score = 0.00013) | OK |
|  | takimoto hifumi (score = 0.99417)
nishikino maki (score = 0.00472)
sakurauchi riko (score = 0.00110) | OK |
|  | takimoto hifumi (score = 0.94923)
sakurauchi riko (score = 0.04842)
nishikino maki (score = 0.00235) | OK |
|  | takimoto hifumi (score = 0.96029)
sakurauchi riko (score = 0.02822)
nishikino maki (score = 0.01150) | OK |
Interestingly, the addition of 3rd character increases the confidence level of several Takimoto Hifumi testcases (see 1st and 4th result). Overall, this character can be easily differentiated compared to the other two.
|Image|Classification| OK/NG |
| --- | --- | --- |
|  | sakurauchi riko (score = 0.98747)
takimoto hifumi (score = 0.01054)
nishikino maki (score = 0.00199) | OK |
|  | sakurauchi riko (score = 0.96840)
takimoto hifumi (score = 0.02895)
nishikino maki (score = 0.00265) | OK |
|  | sakurauchi riko (score = 0.97713)
nishikino maki (score = 0.02167)
takimoto hifumi (score = 0.00119) | OK |
|  | sakurauchi riko (score = 0.90159)
nishikino maki (score = 0.06989)
takimoto hifumi (score = 0.02852) | OK |
|  | sakurauchi riko (score = 0.99713)
takimoto hifumi (score = 0.00184)
nishikino maki (score = 0.00103) | OK |
|  | sakurauchi riko (score = 0.79957)
nishikino maki (score = 0.19310)
takimoto hifumi (score = 0.00733) | OK |
From this experiment, it seems that the current bottleneck is located at Step 1 (face detection), which have the overall accuracy of 83% in face detection.
## animeface-2009
[nagadomi/animeface-2009](https://github.com/nagadomi/animeface-2009) provides another method of face detection. 13 out of 21 unrecognized images are now recognized in `cropped (unrecognized)` directory.
**Current found limitations**: it seems the script requires more memory and slower to run compared to `lbpcascade_animeface.xml`.
|Image|Classification| OK/NG |
| --- | --- | --- |
|  | nishikino maki (score = 0.99296)
sakurauchi riko (score = 0.00694)
takimoto hifumi (score = 0.00010) | OK |
|  | nishikino maki (score = 0.93702)
sakurauchi riko (score = 0.04017)
takimoto hifumi (score = 0.02281) | OK |
|  | nishikino maki (score = 0.99406)
sakurauchi riko (score = 0.00565)
takimoto hifumi (score = 0.00030) | OK |
|Image|Classification| OK/NG |
| --- | --- | --- |
|  | takimoto hifumi (score = 0.99242)
nishikino maki (score = 0.00431)
sakurauchi riko (score = 0.00327) | OK |
|  | takimoto hifumi (score = 0.99596)
sakurauchi riko (score = 0.00403)
nishikino maki (score = 0.00001) | OK |
|  | takimoto hifumi (score = 0.98369)
sakurauchi riko (score = 0.01498)
nishikino maki (score = 0.00133) | OK |
|  | takimoto hifumi (score = 0.99796)
sakurauchi riko (score = 0.00189)
nishikino maki (score = 0.00015) | OK |
|  | takimoto hifumi (score = 0.99601)
nishikino maki (score = 0.00335)
sakurauchi riko (score = 0.00064) | OK |
|  | takimoto hifumi (score = 0.99960)
sakurauchi riko (score = 0.00029)
nishikino maki (score = 0.00011) | OK |
|  | takimoto hifumi (score = 0.99995)
nishikino maki (score = 0.00004)
sakurauchi riko (score = 0.00001) | OK |
|Image|Classification| OK/NG |
| --- | --- | --- |
|  | sakurauchi riko (score = 0.84480)
nishikino maki (score = 0.12101)
takimoto hifumi (score = 0.03419) | OK |
|  | sakurauchi riko (score = 0.94310)
nishikino maki (score = 0.04296)
takimoto hifumi (score = 0.01393) | OK |
|  | sakurauchi riko (score = 0.96176)
takimoto hifumi (score = 0.03217)
nishikino maki (score = 0.00607) | OK |
Since this method gives better result in detecting anime character face and classification still works with almost the same result, the overall face detection accuracy is now around **93%**.
## License
 is created by [nagadomi/lbpcascade_animeface](https://github.com/nagadomi/lbpcascade_animeface).
Copyright for all images are owned by their respective creators.