Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/ltfschoen/dog-project

Term 2 Project 1 Dog Breed Classifier and human face detector using ImageNet, superhuman CNNs, and Haar Cascades
https://github.com/ltfschoen/dog-project

aws-ec2 bottleneck convolutional-neural-networks gpu-computing haar image-augmentation image-classification keras multi-class-classification resnet-50 transfer-learning

Last synced: about 2 months ago
JSON representation

Term 2 Project 1 Dog Breed Classifier and human face detector using ImageNet, superhuman CNNs, and Haar Cascades

Awesome Lists containing this project

README 

        
[//]: # (Image References)



[image1]: ./images/sample_dog_output.png "Sample Output"

[image2]: ./images/vgg16_model.png "VGG-16 Model Keras Layers"

[image3]: ./images/vgg16_model_draw.png "VGG16 Model Figure"



## Project Overview



Welcome to the Convolutional Neural Networks (CNN) project in the AI Nanodegree! In this project, you will learn how to build a pipeline that can be used within a web or mobile app to process real-world, user-supplied images.  Given an image of a dog, your algorithm will identify an estimate of the canine’s breed.  If supplied an image of a human, the code will identify the resembling dog breed.  



![Sample Output][image1]



Along with exploring state-of-the-art CNN models for classification, you will make important design decisions about the user experience for your app.  Our goal is that by completing this lab, you understand the challenges involved in piecing together a series of models designed to perform various tasks in a data processing pipeline.  Each model has its strengths and weaknesses, and engineering a real-world application often involves solving many problems without a perfect answer.  Your imperfect solution will nonetheless create a fun user experience!



## Screenshots from Submission



![alt tag](https://raw.githubusercontent.com/ltfschoen/dog-project/master/screenshots/1.png)



![alt tag](https://raw.githubusercontent.com/ltfschoen/dog-project/master/screenshots/2.png)



![alt tag](https://raw.githubusercontent.com/ltfschoen/dog-project/master/screenshots/3.png)



![alt tag](https://raw.githubusercontent.com/ltfschoen/dog-project/master/screenshots/4.png)



![alt tag](https://raw.githubusercontent.com/ltfschoen/dog-project/master/screenshots/5.png)



## Project Instructions



### Instructions



1. Clone the repository and navigate to the downloaded folder.

	

	```	

		git clone https://github.com/udacity/dog-project.git

		cd dog-project

	```

2. Download the [dog dataset](https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/dogImages.zip).  Unzip the folder and place it in the repo, at location `path/to/dog-project/dogImages`. 

3. Download the [human dataset](http://vis-www.cs.umass.edu/lfw/lfw.tgz).  Unzip the folder and place it in the repo, at location `path/to/dog-project/lfw`.  If you are using a Windows machine, you are encouraged to use [7zip](http://www.7-zip.org/) to extract the folder. 

4. Download the [VGG-16 bottleneck features](https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/DogVGG16Data.npz) for the dog dataset.  Place it in the repo, at location `path/to/dog-project/bottleneck_features`.

5. Obtain the necessary Python packages, and switch Keras backend to Tensorflow.  

	

	For __Mac/OSX__:

	```

		conda env create -f requirements/aind-dog-mac.yml

		source activate aind-dog

		KERAS_BACKEND=tensorflow python -c "from keras import backend"

	```



	For __Linux__:

	```

		conda env create -f requirements/aind-dog-linux.yml

		source activate aind-dog

		KERAS_BACKEND=tensorflow python -c "from keras import backend"

	```



	For __Windows__:

	```

		conda env create -f requirements/aind-dog-windows.yml

		activate aind-dog

		set KERAS_BACKEND=tensorflow

		python -c "from keras import backend"

	```

6. Open the notebook and follow the instructions.

	

	```

		jupyter notebook dog_app.ipynb

	```



__NOTE:__ While some code has already been implemented to get you started, you will need to implement additional functionality to successfully answer all of the questions included in the notebook. __Unless requested, do not modify code that has already been included.__



## Amazon Web Services



Instead of training your model on a local CPU (or GPU), you could use Amazon Web Services to launch an EC2 GPU instance.  Please refer to the [Udacity instructions](https://classroom.udacity.com/nanodegrees/nd889/parts/16cf5df5-73f0-4afa-93a9-de5974257236/modules/53b2a19e-4e29-4ae7-aaf2-33d195dbdeba/lessons/2df3b94c-4f09-476a-8397-e8841b147f84/project) for setting up a GPU instance for this project.



* SSH into the EC2 GPU Instance 

	```

	ssh aind2@

	```



* Clone and Activate environment

	```

	git clone https://github.com/ltfschoen/dog-project; cd dog-project

	source activate aind2;

	```



* Fetch and unzip Dog and Human datasets

	```

	wget https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/dogImages.zip; unzip dogImages.zip; rm dogImages.zip;



	wget http://vis-www.cs.umass.edu/lfw/lfw.tgz; tar -xvzf lfw.tgz; rm lfw.tgz

	```



* Fetch and unzip VGG-16 Bottleneck and ResNet50 Bottleneck

	```

	cd bottleneck_features; wget https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/DogVGG16Data.npz; cd ..



	cd bottleneck_features; wget https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/DogResnet50Data.npz; cd ..

	```



* Start the Jupyter Notebook

	```

	jupyter notebook --ip=0.0.0.0 --no-browser

	```



* Copy/Paste the URL into your browser

(i.e. `http://0.0.0.0:8888/?token=3156e...`)

but REPLACE the 0.0.0.0 with the "IPv4 Public IP" from AWS EC2 GPU

instance shown in the EC2 Dashboard



* Click dog_app.ipynb in the browser to edit the notebook



* IMPORTANT NOTE: 

	* STOP the EC2 GPU Instance when not in use since "p2.xlarge" GPU Instance costs ~AU$1.5 per hour to run [according to EC2 On-Demand pricing](https://aws.amazon.com/ec2/pricing/on-demand/)

	* TERMINATE when no longer using since otherwise may be subjected to EBS Storage costs



## Copying data from AWS GPU Instance to Local Computer



* Run the following on local machine:

```

scp -r aind2@:/home/aind2/dog-project /Users/Ls/Desktop

```



## Evaluation



Your project will be reviewed by a Udacity reviewer against the CNN project [rubric](#rubric) found [here](https://review.udacity.com/#!/rubrics/810/view).  Review this rubric thoroughly, and self-evaluate your project before submission.  All criteria found in the rubric must meet specifications for you to pass.



## Project Submission



When you are ready to submit your project, collect the following files and compress them into a single archive for upload:

- The `dog_app.ipynb` file with fully functional code, all code cells executed and displaying output, and all questions answered.

- An HTML or PDF export of the project notebook with the name `report.html` or `report.pdf`.

- Any additional images used for the project that were not supplied to you for the project. __Please do not include the project data sets in the `dogImages/` or `lfw/` folders.  Likewise, please do not include the `bottleneck_features/` folder.__



Alternatively, your submission could consist of the GitHub link to your repository.





## Project Rubric



#### Files Submitted



| Criteria       		|     Meets Specifications	        			            | 

|:---------------------:|:---------------------------------------------------------:| 

| Submission Files      | The submission includes all required files.		|



#### Documentation



| Criteria       		|     Meets Specifications	        			            | 

|:---------------------:|:---------------------------------------------------------:| 

| Comments         		| The submission includes comments that describe the functionality of the code.  	|



#### Step 1: Detect Humans



| Criteria       		|     Meets Specifications	        			            | 

|:---------------------:|:---------------------------------------------------------:| 

| __Question 1:__ Assess the Human Face Detector |  The submission returns the percentage of the first 100 images in the dog and human face datasets with a detected human face.          |

| __Question 2:__ Assess the Human Face Detector |  The submission opines whether Haar cascades for face detection are an appropriate technique for human detection.    |



#### Step 2: Detect Dogs



| Criteria       		|     Meets Specifications	        			            | 

|:---------------------:|:---------------------------------------------------------:| 

| __Question 3:__ Assess the Dog Detector |  The submission returns the percentage of the first 100 images in the dog and human face datasets with a detected dog.          |



#### Step 3: Create a CNN to Classify Dog Breeds (from Scratch)



| Criteria       		|     Meets Specifications	        			            | 

|:---------------------:|:---------------------------------------------------------:| 

| Model Architecture | The submission specifies a CNN architecture. |

| Train the Model | The submission specifies the number of epochs used to train the algorithm. |

| Test the Model | The trained model attains at least 1% accuracy on the test set. |



#### Step 5: Create a CNN to Classify Dog Breeds (using Transfer Learning)



| Criteria       		|     Meets Specifications	        			            | 

|:---------------------:|:---------------------------------------------------------:| 

| Obtain Bottleneck Features | The submission downloads the bottleneck features corresponding to one of the Keras pre-trained models (VGG-19, ResNet-50, Inception, or Xception). |

| Model Architecture | The submission specifies a model architecture.  |

| __Question 5__: Model Architecture | The submission details why the chosen architecture succeeded in the classification task and why earlier attempts were not as successful.  |

| Compile the Model | The submission compiles the architecture by specifying the loss function and optimizer. |

| Train the Model    | The submission uses model checkpointing to train the model and saves the model with the best validation loss. |

| Load the Model with the Best Validation Loss    | The submission loads the model weights that attained the least validation loss. |

| Test the Model    | Accuracy on the test set is 60% or greater. |

| Predict Dog Breed with the Model | The submission includes a function that takes a file path to an image as input and returns the dog breed that is predicted by the CNN. |



#### Step 6: Write your Algorithm



| Criteria       		|     Meets Specifications	        			            | 

|:---------------------:|:---------------------------------------------------------:| 

| Write your Algorithm   | The submission uses the CNN from Step 5 to detect dog breed.  The submission has different output for each detected image type (dog, human, other) and provides either predicted actual (or resembling) dog breed. |



#### Step 7: Test your Algorithm

| Criteria       		|     Meets Specifications	        			            | 

|:---------------------:|:---------------------------------------------------------:| 

| Test Your Algorithm on Sample Images!   | The submission tests at least 6 images, including at least two human and two dog images. |

| __Question 6__: Test Your Algorithm on Sample Images! | The submission discusses performance of the algorithm and discusses at least three possible points of improvement. |



## Suggestions to Make your Project Stand Out!



(Presented in no particular order ...)



#### (1) Augment the Training Data 



[Augmenting the training and/or validation set](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html) might help improve model performance. 



#### (2) Turn your Algorithm into a Web App



Turn your code into a web app using [Flask](http://flask.pocoo.org/) or [web.py](http://webpy.org/docs/0.3/tutorial)!  



#### (3) Overlay Dog Ears on Detected Human Heads



Overlay a Snapchat-like filter with dog ears on detected human heads.  You can determine where to place the ears through the use of the OpenCV face detector, which returns a bounding box for the face.  If you would also like to overlay a dog nose filter, some nice tutorials for facial keypoints detection exist [here](https://www.kaggle.com/c/facial-keypoints-detection/details/deep-learning-tutorial).



#### (4) Add Functionality for Dog Mutts



Currently, if a dog appears 51% German Shephard and 49% poodle, only the German Shephard breed is returned.  The algorithm is currently guaranteed to fail for every mixed breed dog.  Of course, if a dog is predicted as 99.5% Labrador, it is still worthwhile to round this to 100% and return a single breed; so, you will have to find a nice balance.  



#### (5) Experiment with Multiple Dog/Human Detectors



Perform a systematic evaluation of various methods for detecting humans and dogs in images.  Provide improved methodology for the `face_detector` and `dog_detector` functions.