Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/gyuho/dplearn
Learn Deep Learning The Hard Way
https://github.com/gyuho/dplearn
deep-learning statistical-learning
Last synced: about 2 months ago
JSON representation
Learn Deep Learning The Hard Way
- Host: GitHub
- URL: https://github.com/gyuho/dplearn
- Owner: gyuho
- License: other
- Created: 2017-06-04T04:22:17.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2022-12-29T04:45:32.000Z (almost 2 years ago)
- Last Synced: 2024-06-18T23:05:26.793Z (6 months ago)
- Topics: deep-learning, statistical-learning
- Language: Go
- Homepage: http://dplearn.com
- Size: 85.6 MB
- Stars: 64
- Watchers: 3
- Forks: 14
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## dplearn
[](https://goreportcard.com/report/github.com/gyuho/dplearn)
[](https://travis-ci.org/gyuho/dplearn)
[](https://godoc.org/github.com/gyuho/dplearn)
Learn Deep Learning The Hard Way.
It is a set of small projects on [Deep Learning](https://en.wikipedia.org/wiki/Deep_learning).
- [System Overview](#system-overview)
- [Cats vs. Non-Cat](#cats-vs-non-cat)
- [Workflow](#workflow)
### System Overview
- [`frontend`](https://github.com/gyuho/dplearn/tree/master/frontend) implements user-facing UI, sends user requests to [`backend/*`](https://github.com/gyuho/dplearn/tree/master/backend).
- [`backend/web`](https://github.com/gyuho/dplearn/tree/master/backend/web) schedules user requests on [`pkg/etcd-queue`](https://github.com/gyuho/dplearn/tree/master/pkg/etcd-queue).
- [`backend/worker`](https://github.com/gyuho/dplearn/tree/master/backend/worker) processes jobs from queue, and writes back the results.
- Data serialization from `frontend` to `backend/web` is defined in [`backend/web.Request`](https://github.com/gyuho/dplearn/blob/master/backend/web/handler.go) and [`frontend/app/request.service.Request`](https://github.com/gyuho/dplearn/blob/master/frontend/app/request.service.ts).
- Data serialization from `backend/web` to `frontend` is defined in [`pkg/etcd-queue.Item`](https://github.com/gyuho/dplearn/blob/master/pkg/etcd-queue/queue.go) and [`frontend/app/request.service.Item`](https://github.com/gyuho/dplearn/blob/master/frontend/app/request.service.ts).
- Data serialization between `backend/web` and `backend/worker` is defined in [`pkg/etcd-queue.Item`](https://github.com/gyuho/dplearn/blob/master/pkg/etcd-queue/queue.go) and [`backend/worker/worker.py`](https://github.com/gyuho/dplearn/blob/master/backend/worker/worker.py).
Notes:
- **Why is the queue service needed?** To process concurrent users requests. Worker has limited resources, serializing requests into the queue.
- **Why Go?** To natively use [`embedded etcd`](https://github.com/coreos/etcd/tree/master/embed).
- **Why etcd?** To use [etcd Watch API](https://godoc.org/github.com/coreos/etcd/clientv3#Watcher). `pkg/etcd-queue` uses Watch to stream updates to `backend/worker` and `frontend`. This minimizes TCP socket creation and slow TCP starts (e.g. streaming vs. polling).
This is a *proof-of-concept*. In production, I would use: [Tensorflow/serving](https://www.tensorflow.org/serving/) to serve the pre-trained models, distributed [`etcd`](https://github.com/coreos/etcd) for higher availability.
[↑ top](#dplearn)
### Cats vs. Non-Cat
To train `cats` 5-layer Deep Neural Network model:
```bash
DATASETS_DIR=./datasets \
CATS_PARAM_PATH=./datasets/parameters-cats.npy \
python3 -m unittest backend.worker.cats.model_test
```
This persists trained model parameters on disk that can be loaded by workers later.
[↑ top](#dplearn)
### Workflow
To run application (backend, web UI) locally, on http://localhost:4200:
```bash
./scripts/docker/run-app.sh
./scripts/docker/run-worker-python3-cpu.sh
<