Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/g0bel1n/megazord-backend
Megazord (Backend) is the backend of an industrial pieces image recognition app using deep learning.
https://github.com/g0bel1n/megazord-backend
computer-vision convolutional-neural-networks deep-learning
Last synced: about 1 month ago
JSON representation
Megazord (Backend) is the backend of an industrial pieces image recognition app using deep learning.
- Host: GitHub
- URL: https://github.com/g0bel1n/megazord-backend
- Owner: g0bel1n
- License: mit
- Created: 2021-06-23T15:07:52.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-01-07T09:26:26.000Z (almost 3 years ago)
- Last Synced: 2024-11-07T10:07:24.636Z (about 2 months ago)
- Topics: computer-vision, convolutional-neural-networks, deep-learning
- Language: Python
- Homepage:
- Size: 26.3 MB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Megazord
![plot](./ressources/megazord_pic.png?raw=true)[![Unit tests](https://github.com/iSab01/megazord/actions/workflows/python-app.yml/badge.svg)](https://github.com/iSab01/megazord/actions/workflows/python-app.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
Deep learning project realised by Lucas Saban and Augustin Cramer for Soft Next.Please mention us if you use this code.
Feel free to contact us at : [email protected] or [email protected].**The goal is to classify the maximum amount of industrial pieces while minimizing misclassifcations.**
## Demonstration
![Demo](https://github.com/g0bel1n/megazord-backend/blob/main/ressources/demo_megazord.gif)
## Lexicon :
**Zords** : Convolutionnal Neural Networks (CNN) trained to classify labels from the same class. For example,
**handle** distinguish the labels *handle_lock* and *handle_lockless*.**Main Zord** : CNN trained to distinguish classes. (*ex: handles from motors*)
**MegaZord** : Assemblage of the Main Zord and the different Zord
## Concept :
Our model makes predictions following a two-stage process.
Firstly, the image is inputed into the main_zord which determines its class. If it's a single classe label, then
the job is done and the prediction is made.
If it's not the case, if the image belongs to a class with several labels, a mask is applied so that the input is
inputed in the adequate zord and only this one. Then a label prediction is made and outputted.**However, it will not be valid for the frontend as the confidence of the output must be superior to 0.7 to be
considered as such**## OnBoarding
### Installation```
pip install git+https://github.com/iSab01/megazord-backend
```### Creating the dataset
In order to get enough data to train our models, we used *video_to_frame.py*. It allowed to video-scan our pieces in
different environnements quite quickly and to extract pictures from it.For example :
```python
from MegaZord.utilitaries.video_to_frame import framerdirectory = "my_dir_with_videos"
framer(fps_goal=8, directory=directory, rescale=True, shape=(256,256))
```
You might want to put your video into labels folders and iterate upon the directories to automatize the process.### Make the app
#### Training and assembling a modelThe main file is megazord.py in the directory SwissKnife
``` python
from MegaZord.SwissKnife.megazord import SwissKnife
```Instantiates the SwissKnife object with your main directory path and the base_model used for zord.
For now, InceptionV3, MobileNetV2, EfficientNetV2 and EfficientNet are supported (the input must be in low cap)
>The main_zord will be trained with inceptionV3 no matter the base_model chosen.
> Making good main predictions on the main zord is crucial.
``` python
swiss_knife = SwissKnife(directory = "/Users/lucas/swiss_knife", base_model = "mobilenetv2")
```
If some zords are missing, they will be trained with the following command :
```python
swiss_knife.train_zords(epochs= 2)
```
Then, it's time to assemble the main_zord and every zords :
```python
megazord = swiss_knife.assemble_megazord()
```
To save the megazord :
```python
swiss_knife.save(megazord)
```
> **Warnings** : If the base_model is efficientNetV2, you might encounter errors when loading the saved file.Finally, the user can convert the megazord to .mlmodel with :
```python
swiss_knife.megazord_to_coreml(megazord)
```#### Configuring the app
Now that the mlmodel has been saved, you need to create a plist file containing info about your labels for the app to
work.
Please run the following command :```python
execfile("Megazord/utilitaries/ProductCatalog_Maker.py")
```Then you have to open the [xcode project](https://github.com/iSab01/megazord_frontend) and
update the files *ProductCatalog.plist* and *MegaZord_mobilenetv2.mlmodel*.IOS >= 13.0.0 is required.
Finally, build the app on the simulator or your iDevice after signing the project with your developper account.
## Versions
| Module | Version |
| :--- | ----: |
| Python | 3.9 |
| tensorflow | 2.5.0 |
| coremltools | 4.1 |
| opencv | 4.5.2.54 |
| tqdm | 4.61.1 |
| numpy | 1.19.5 |
| matplotlib | 3.4.2 |## The frontend
The app is based on code sample found on the Apple Developper website. We only modified it to work with our model and to
adapt the detection threshold.
It's not energy efficient and is quite heavy.
For now, it's only for visualization but we might spend more time on it later on.The code can be found here, in the [frontend repo](https://github.com/iSab01/megazord_frontend)
The .plist files needed to show predictions are the output of :
```python
execfile('MegaZord/utilitaries/ProductCatalog_Maker.py')
execfile('MegaZord/utilitaries/UndefinedProductCatalog_Maker.py')
execfile('MegaZord/utilitaries/ref_to_id_maker.py')
```## Techniques used
In this project we used Data Augmentation, Transfer Learning (Inception V3,EfficentNet, EfficientNetV2, MobileNetV2),
Fine Tuning.
## Errors knownThe importation processes in-between files is quite messy, you might check that first if you encounter errors.
## Why *Megazord* and *zords* ?
Well, it is because of the architecture of our neural network. It combines several small neural networks, trained to
distinguish object from the same class (ex: motor_s and motor_m) to make a big neural networks that classify each label.
It made us remember the good old days, when we used to watch the *Power Rangers series*. They used to merge their zords
(each Ranger had its own zord aka fighter robot) into a megazord which could defeat the bad guys. In our case, the
megazord is the final model and the zords are the secondary models.## Tips
### Directory organization
Please remember that your main directory must look like this :
main_directory
main_directory/class_1
main_directory/class_1/label_1
main_directory/class_1/label_1/pic_1 ...
main_directory/class_1/label_2
main_directory/class_2
main/directory/class_2/label_1
main/directory/class_2/label_1/pic_2 ..