{"id":19136466,"url":"https://github.com/chris-chris/haiku-scalable-example","last_synced_at":"2025-05-06T20:08:24.837Z","repository":{"id":44195456,"uuid":"242456530","full_name":"chris-chris/haiku-scalable-example","owner":"chris-chris","description":"Scalable distributed reinforcement learning agents on kubernetes","archived":false,"fork":false,"pushed_at":"2022-09-23T22:36:11.000Z","size":308,"stargazers_count":55,"open_issues_count":1,"forks_count":4,"subscribers_count":6,"default_branch":"master","last_synced_at":"2023-03-06T22:10:22.519Z","etag":null,"topics":["deepmind","distributed-reinforcement-learning","impala","kubernetes","machine-learning","reinforcement-learning","scalable"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chris-chris.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-23T04:32:45.000Z","updated_at":"2022-09-29T17:32:31.000Z","dependencies_parsed_at":"2023-01-18T16:15:09.301Z","dependency_job_id":null,"html_url":"https://github.com/chris-chris/haiku-scalable-example","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chris-chris%2Fhaiku-scalable-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chris-chris%2Fhaiku-scalable-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chris-chris%2Fhaiku-scalable-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chris-chris%2Fhaiku-scalable-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chris-chris","download_url":"https://codeload.github.com/chris-chris/haiku-scalable-example/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223795737,"owners_count":17204140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deepmind","distributed-reinforcement-learning","impala","kubernetes","machine-learning","reinforcement-learning","scalable"],"created_at":"2024-11-09T06:34:24.806Z","updated_at":"2024-11-09T06:34:25.340Z","avatar_url":"https://github.com/chris-chris.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# haiku-scalable-example\nScalable reinforcement learning agents on container orchestration\n\n[![chris-chris](https://circleci.com/gh/chris-chris/haiku-scalable-example.svg?style=shield)](\u003chttps://circleci.com/gh/chris-chris/haiku-scalable-example\u003e)\n[![Coverage Status](https://coveralls.io/repos/github/chris-chris/haiku-scalable-example/badge.svg?branch=master)](https://coveralls.io/github/chris-chris/haiku-scalable-example?branch=master)\n\n## 1. Purpose of the project\nImplement scalable reinforcement learning agent on the container orchestraion system like k8s.\n\n## 2. Container Orchestraion\n- [x] Kubernetes\n- [ ] Google Cloud Platform\n\n## 3. Reinforcement Learning Algorithms\n- [x] IMPALA\n- [ ] A3C\n- [ ] TBD\n\n## 4. Architecture\n\nThis example will introduce a clear way to deploy scalable reinforcement learning agents to the computing clusters.\n\n![alt text](img/k8s.png \"Logo Title Text 1\")\n\n## 5. Install\n\n```$bash\n$ git clone https://github.com/chris-chris/haiku-scalable-example\n$ cd haiku-scalable-example\n$ pip install -r requirements.txt\n```\n\n## 6. Execute\n\n#### v1. Learner + Multi Actor IMPALA wiring through gRPC.\n\n```$bash\n$ python learner_server.py\n```\n\n```$bash\n$ GRPC_HOST=localhost:50051 python actor_client.py \u0026\n$ GRPC_HOST=localhost:50051 python actor_client.py \u0026\n```\n\n#### v2. 1 Learner + Multi Actor IMPALA wiring through gRPC on docker VMs.\n\nprepare\n```bash\n$ docker pull chrisai/haiku-scalable-example-learner:test\n$ docker pull chrisai/haiku-scalable-example-actor:test\n\n$ docker network create --subnet 172.20.0.0/16 --ip-range 172.20.240.0/20 multi-host-network\n```\n\nrun\n```bash\n$ docker run -d -p 127.0.0.1:50051:50051 --network=multi-host-network --ip=172.20.240.1 chrisai/haiku-scalable-example-learner:test\n$ docker run -d --env GRPC_HOST=172.20.240.1:50051 --network=multi-host-network chrisai/haiku-scalable-example-actor:test\n```\n\nwanna see logs?\n```bash\n$ docker ps\n$ docker attach [CONTAINER ID]\n```\n\n#### v3. 1 Learner + Multi Actor IMPALA wiring through gRPC on k8s.\n\n- Install minikube\n\nhttps://kubernetes.io/docs/tasks/tools/install-minikube/\n\n- Run\n```bash\n$ kubectl apply -f impala.yml\n```\n\n- Wanna see logs?\n```bash\n$ kubectl logs -f impala learner\n$ kubectl logs -f impala actor\n```\n\n## 7. To-dos\n\n- [x] v1. 1 Learner + Multi Actor IMPALA wiring through gRPC.\n- [x] v2. 1 Learner + Multi Actor IMPALA wiring through gRPC on docker VMs.\n- [x] v3. 1 Learner + Multi Actor IMPALA wiring through gRPC on k8s.\n- [x] Optimize the model weight serialization for the performance.\n- [ ] v4. Multi Learner + Multi Actor IMPALA wiring through gRPC on k8s.\n- [ ] Implement other distributed RL algorithms\n- [ ] Asynchronous Processing via Queue\n- [ ] Monitor the computing resource usages \n\n## 8. Reference\n\nI used Deepmind's open sources [haiku](https://github.com/deepmind/dm-haiku), [rlax](https://github.com/deepmind/rlax), and google [jax](https://github.com/google/jax)\n\n- https://github.com/google/jax\n- https://github.com/deepmind/rlax\n- https://github.com/deepmind/haiku\n- https://github.com/kubernetes/kubernetes\n- https://github.com/kent-williams/grpc-python-kubernetes\n- https://github.com/simondlevy/OpenAI-Gym-Hacks/\n- https://blog.tensorflow.org/2018/07/deep-reinforcement-learning-keras-eager-execution.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchris-chris%2Fhaiku-scalable-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchris-chris%2Fhaiku-scalable-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchris-chris%2Fhaiku-scalable-example/lists"}