{"id":19487498,"url":"https://github.com/gyuho/dplearn","last_synced_at":"2025-08-25T08:08:22.244Z","repository":{"id":27306138,"uuid":"93289870","full_name":"gyuho/dplearn","owner":"gyuho","description":"Learn Deep Learning The Hard Way","archived":false,"fork":false,"pushed_at":"2022-12-29T04:45:32.000Z","size":89739,"stargazers_count":64,"open_issues_count":8,"forks_count":14,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-04T01:32:11.203Z","etag":null,"topics":["deep-learning","statistical-learning"],"latest_commit_sha":null,"homepage":"http://dplearn.com","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gyuho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-06-04T04:22:17.000Z","updated_at":"2024-06-17T04:18:57.000Z","dependencies_parsed_at":"2023-01-14T07:30:16.865Z","dependency_job_id":null,"html_url":"https://github.com/gyuho/dplearn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyuho%2Fdplearn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyuho%2Fdplearn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyuho%2Fdplearn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyuho%2Fdplearn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gyuho","download_url":"https://codeload.github.com/gyuho/dplearn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250872326,"owners_count":21500801,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","statistical-learning"],"created_at":"2024-11-10T20:46:10.255Z","updated_at":"2025-04-25T18:32:25.795Z","avatar_url":"https://github.com/gyuho.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## dplearn\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/gyuho/dplearn?style=flat-square)](https://goreportcard.com/report/github.com/gyuho/dplearn)\n[![Build Status](https://img.shields.io/travis/gyuho/dplearn.svg?style=flat-square)](https://travis-ci.org/gyuho/dplearn)\n[![Godoc](https://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)](https://godoc.org/github.com/gyuho/dplearn)\n\nLearn Deep Learning The Hard Way.\n\nIt is a set of small projects on [Deep Learning](https://en.wikipedia.org/wiki/Deep_learning).\n\n\u003cbr\u003e\u003cbr\u003e\n\n- [System Overview](#system-overview)\n- [Cats vs. Non-Cat](#cats-vs-non-cat)\n- [Workflow](#workflow)\n\n\u003cbr\u003e\u003cbr\u003e\n\n### System Overview\n\n\u003cimg src=\"./dplearn-architecture.png\" alt=\"dplearn-architecture\" width=\"620\"\u003e\n\n- [`frontend`](https://github.com/gyuho/dplearn/tree/master/frontend) implements user-facing UI, sends user requests to [`backend/*`](https://github.com/gyuho/dplearn/tree/master/backend).\n- [`backend/web`](https://github.com/gyuho/dplearn/tree/master/backend/web) schedules user requests on [`pkg/etcd-queue`](https://github.com/gyuho/dplearn/tree/master/pkg/etcd-queue).\n- [`backend/worker`](https://github.com/gyuho/dplearn/tree/master/backend/worker) processes jobs from queue, and writes back the results.\n- Data serialization from `frontend` to `backend/web` is defined in [`backend/web.Request`](https://github.com/gyuho/dplearn/blob/master/backend/web/handler.go) and [`frontend/app/request.service.Request`](https://github.com/gyuho/dplearn/blob/master/frontend/app/request.service.ts).\n- Data serialization from `backend/web` to `frontend` is defined in [`pkg/etcd-queue.Item`](https://github.com/gyuho/dplearn/blob/master/pkg/etcd-queue/queue.go) and [`frontend/app/request.service.Item`](https://github.com/gyuho/dplearn/blob/master/frontend/app/request.service.ts).\n- Data serialization between `backend/web` and `backend/worker` is defined in [`pkg/etcd-queue.Item`](https://github.com/gyuho/dplearn/blob/master/pkg/etcd-queue/queue.go) and [`backend/worker/worker.py`](https://github.com/gyuho/dplearn/blob/master/backend/worker/worker.py).\n\nNotes:\n\n- **Why is the queue service needed?** To process concurrent users requests. Worker has limited resources, serializing requests into the queue.\n- **Why Go?** To natively use [`embedded etcd`](https://github.com/coreos/etcd/tree/master/embed).\n- **Why etcd?** To use [etcd Watch API](https://godoc.org/github.com/coreos/etcd/clientv3#Watcher). `pkg/etcd-queue` uses Watch to stream updates to `backend/worker` and `frontend`. This minimizes TCP socket creation and slow TCP starts (e.g. streaming vs. polling).\n\nThis is a *proof-of-concept*. In production, I would use: [Tensorflow/serving](https://www.tensorflow.org/serving/) to serve the pre-trained models, distributed [`etcd`](https://github.com/coreos/etcd) for higher availability.\n\n[↑ top](#dplearn)\n\u003cbr\u003e\u003cbr\u003e\n\n\n### Cats vs. Non-Cat\n\nTo train `cats` 5-layer Deep Neural Network model:\n\n```bash\nDATASETS_DIR=./datasets \\\n  CATS_PARAM_PATH=./datasets/parameters-cats.npy \\\n  python3 -m unittest backend.worker.cats.model_test\n```\n\nThis persists trained model parameters on disk that can be loaded by workers later.\n\n\u003cimg src=\"./dplearn-cats.gif\" alt=\"dplearn-cats\" width=\"550\"\u003e\n\n[↑ top](#dplearn)\n\u003cbr\u003e\u003cbr\u003e\n\n\n### Workflow\n\nTo run application (backend, web UI) locally, on http://localhost:4200:\n\n```bash\n./scripts/docker/run-app.sh\n./scripts/docker/run-worker-python3-cpu.sh\n\n\u003c\u003cCOMMENT\n# to serve on port :80\n./scripts/docker/run-reverse-proxy.sh\nCOMMENT\n```\n\nOpen http://localhost:4200/cats and try other cat photos:\n\n- https://static.pexels.com/photos/127028/pexels-photo-127028.jpeg\n- https://static.pexels.com/photos/126407/pexels-photo-126407.jpeg\n- https://static.pexels.com/photos/54632/cat-animal-eyes-grey-54632.jpeg\n\nTo update dependencies:\n\n```bash\n./scripts/dep/go.sh\n./scripts/dep/frontend.sh\n```\n\nTo update [`Dockerfile`](./dockerfiles):\n\n```bash\n# update 'container.yaml' and then\n./scripts/docker/gen.sh\n```\n\nTo build Docker container images:\n\n```bash\n./scripts/docker/build-app.sh\n./scripts/docker/build-python3-cpu.sh\n./scripts/docker/build-python3-gpu.sh\n./scripts/docker/build-r.sh\n./scripts/docker/build-reverse-proxy.sh\n```\n\nTo run tests:\n\n```bash\n./scripts/tests/frontend.sh\n./scripts/tests/go.sh\n\ngo install -v ./cmd/backend-web-server\n\nDATASETS_DIR=./datasets \\\n  CATS_PARAM_PATH=./datasets/parameters-cats.npy \\\n  ETCD_EXEC=/opt/bin/etcd \\\n  SERVER_EXEC=${GOPATH}/bin/backend-web-server \\\n  ./scripts/tests/python3.sh\n```\n\nTo run tests in container:\n\n```bash\n./scripts/docker/test-app.sh\n./scripts/docker/test-python3-cpu.sh\n```\n\nTo run [IPython Notebook](https://ipython.org) locally, on http://localhost:8888/tree:\n\n```bash\n./scripts/docker/run-ipython-python3-cpu.sh\n./scripts/docker/run-ipython-python3-gpu.sh\n./scripts/docker/run-r.sh\n```\n\nTo deploy `dplearn` and IPython Notebook on Google Cloud Platform CPU or GPU:\n\n```bash\nGCP_KEY_PATH=/etc/gcp-key-dplearn.json ./scripts/gcp/ubuntu-python3-cpu.gcp.sh\nGCP_KEY_PATH=/etc/gcp-key-dplearn.json ./scripts/gcp/ubuntu-python3-gpu.gcp.sh\n\n# create a Google Cloud Platform Compute Engine VM with a start-up script\n# to provision GPU, init system, reverse proxy, and others\n# (see ./scripts/gcp/ubuntu-python3-gpu.ansible.sh for more detail)\n```\n\n[↑ top](#dplearn)\n\u003cbr\u003e\u003cbr\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgyuho%2Fdplearn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgyuho%2Fdplearn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgyuho%2Fdplearn/lists"}