Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/radosz99/scrabble-board-detector
RPC API created using FastAPI for detecting letters based on image with scrabble board in it.
https://github.com/radosz99/scrabble-board-detector
fastapi python rpc scrabble
Last synced: 22 days ago
JSON representation
RPC API created using FastAPI for detecting letters based on image with scrabble board in it.
- Host: GitHub
- URL: https://github.com/radosz99/scrabble-board-detector
- Owner: radosz99
- Created: 2021-09-30T20:54:48.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2021-10-08T11:24:26.000Z (over 3 years ago)
- Last Synced: 2024-11-12T22:20:16.104Z (3 months ago)
- Topics: fastapi, python, rpc, scrabble
- Language: Python
- Homepage:
- Size: 57 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# General info
RPC API created using FastAPI for detecting letters based on image with scrabble board in it.
In| Out
:-------------------------:|:-------------------------:
| 
# Endpoints
## Get letters pattern (POST)
The only endpoint in API requires an image in request body.
### URL
```
http://127.0.0.1:8000/detect_letters
```
### Example using Python requests
In request there must be parameter `files` included, with key `image` and binary representation of image as value.
```
import requests
import ast
IMG_PATH = "test.jpg"
SERVER_URL = "http://127.0.0.1:8000"
files = {'image': open(IMG_PATH,'rb')}
response = requests.post(url=f"{SERVER_URL}/detect_letters", files=files)
json_response = ast.literal_eval(response.text)
board = json_response['board']
```
As a response there is a json with only one key-value pair - `board` and its representation as a list of list:
```
{
"board": [[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', 'o', 'p', 'e', 'r', 'a', 't', 'i', 'o', 'n', 'a', 'l', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ']]
}
```
# Run
## FastAPI (Windows)
From the root folder:
```
$ uvicorn project.main:app --reload
```
# Principle
## Image with board in it
To detect something firstly is required to have an image with board (all edges required) in it:
## Board detection in image
Then function `align_images` from `project/board_detector.py` firstly cutting out board from the image using some OpenCV stuff(`BRISK_create`, `detectAndCompute`, `DescriptorMatcher_create`, `findHomography`, `warpPerspective`) to this form:
Secondly board is being thresholded using OpenCV `threshold` method to facilitate later analysis and also right board is extracted from board using manually calculated proportions:
```
BOARD_LEFT_UP_CORNER_HEIGHT = 0.048575
BOARD_RIGHT_DOWN_CORNER_HEIGHT = 0.949556
BOARD_LEFT_UP_CORNER_WIDTH = 0.081336
BOARD_RIGHT_DOWN_CORNER_WIDTH = 0.916524
```
## Getting cells from the board
Then using function `divide_board_in_cells` from `project/letter_detector.py` board is divided in cells (15x15) which are saved to the files if they are 'white enough' which means if `check_if_cell_is_used` function returns `True`. That function checks average of pixels value in some random row and if it is too black it return False and cell is considered as not used. 'White enough' cells are saved to chosen directory like this with their coordinates in filename:
## Extracting letters from cells
Next stage relies on finding contours in cells that can be considered as letters. With function `leave_only_cells_with_contours_similar_to_letter` from `project/letter_detector.py` first all contours are extracted using OpenCV `findContours` method, then amount of this contours is checked (`check_if_valid_amount_of_contours`) and valid contour is resized and saved to file which name is the same as name of the file with cell from which contour is:
## Few words about training classifier
For classifier training was created script `train.py`. Training is nearly fully automated so the only thing that user must do is to take photos for every letter putted on board in many configurations. Photos should be placed in appropriate directories for every letter. Suppose that we created directory `training` in root directory for training workspace and inside it directory for every letter in alphabet. So project structure now should look like this:
```
scrabble-board-detector/
|
|── training/
| |── a/
| |── b/
| |── c/
| |── ...
| |── y/
| |── z/
|
|── classifiers/
|── images/
|── project/
|── tests/
|── resources/
|
```
Under each of 'letter directory' there should be images with board in it and board should contain only tiles with this letter, no matter how many, there can be all possible tiles with that letter. Filenames of images with board might in any convention (or no convention):
```
scrabble-board-detector/
|
|── training/
| |── a/
| |── board1.png
| |── ...
| |── board2.png
| |── b/
| |── 814814.jpg
| |── ...
| |── 82899949.png
| |── ...
| |── y/
| |── brd.png
| |── ...
| |── brd0.png
| |── z/
| |── brd_with_z_1.png
| |── ...
| |── brd_with_z_67.png
|
|── classifiers/
|── images/
|── project/
|── tests/
|── resources/
|
```
It is basically all, last thing is run script `project/train.py` by:
```
$ python project/test.py
```
Calling `collect_samples_from_boards(TRAINING_WORKSPACE_DIR)` (`TRAINING_WORKSPACE_DIR` by default is `training`) from that script will prepare data for training:
1. Goes through every directory in `TRAINING_WORKSPACE_DIR` and cut board from every image in letter directories and placed cutted boards in `TRAINING_WORKSPACE_DIR/{letter}/cropped_boards`, for example cutted board from `TRAINING_WORKSPACE_DIR/{letter}/{file_name}` will be saved as `TRAINING_WORKSPACE_DIR/{letter}/cropped_boards/{file_name}_cropped.png`. Function `training_utils.get_boards_from_images`
2. In each directory in `TRAINING_WORKSPACE_DIR/{letter}/cropped_boards` directories where cells will be placed are created. For example `TRAINING_WORKSPACE_DIR/{letter}/cropped_boards/{file_name}_cropped_cells`
3. All valid cells from each image are being saved to appropriate directory. Filename clearly indicates coordinates of the cell. Function `training_utils.divide_boards_in_cells`
4. In each directory in format `TRAINING_WORKSPACE_DIR/{letter}/cropped_boards/{file_name}_cropped_cells` directories where valid contours that can be considered as a letter will be placed are created. For example `TRAINING_WORKSPACE_DIR/{letter}/cropped_boards/{file_name}_cropped_cells/cleared`.
5. Then calling function `training_utils.leave_only_cells_probably_with_letter` will collect new cells in created directories.
6. Finally function `training_utils.collect_letters_to_one_directory` will look for `cleared` directories in each letter directory and collect all letter samples to one directory:
Calling `training_utils.get_trained_classifier(TRAINING_WORKSPACE_DIR)` will make a classifier based on prepared data:
1. First it will collect data from all `samples` directories and reformat it appropriate to **Convolutional Neural Network**. All samples are now stored as 12x12 pixels image.
2. Each sample is being converted to 4-bit grayscale (function `convert_image_to_4_bit_array`) and stored in vector 1x144 and each sample is putted to samples vector, so finally vector Ax1x144 is created where A is number of samples. Also vector with target (letters corresponding to vector at the same index) is being created with size Ax1.
3. Then classifier is being initiated, train data is being generated and classifier is being tested (`TEST_SIZE` by default is 0.5):
```
clf = svm.SVC(gamma=0.001, probability=True)
X_train, _, y_train, _ = train_test_split(data, target, test_size=TEST_SIZE, shuffle=True)
clf.fit(X_train, y_train)
```
4. Next it can be saved in file using `training_utils.save_classifier_to_file`.
# How to recognize letters in image?
Calling `recognize_letters_from_image` from `project/train.py`