{"id":29210642,"url":"https://github.com/embarkstudios/emote","last_synced_at":"2025-07-02T21:07:24.791Z","repository":{"id":62715462,"uuid":"454925775","full_name":"EmbarkStudios/emote","owner":"EmbarkStudios","description":"Reinforcement learning library from Embark Studios","archived":false,"fork":false,"pushed_at":"2024-09-25T18:34:14.000Z","size":3443,"stargazers_count":19,"open_issues_count":22,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-06-24T19:14:23.010Z","etag":null,"topics":["embark","reinforcement-learning"],"latest_commit_sha":null,"homepage":"https://embarkstudios.github.io/emote/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EmbarkStudios.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE-APACHE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-02T20:31:15.000Z","updated_at":"2025-04-04T03:05:44.000Z","dependencies_parsed_at":"2023-10-11T13:35:39.791Z","dependency_job_id":"18c1e8a6-2d0b-44c7-ab47-1f08e888eb75","html_url":"https://github.com/EmbarkStudios/emote","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/EmbarkStudios/emote","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EmbarkStudios%2Femote","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EmbarkStudios%2Femote/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EmbarkStudios%2Femote/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EmbarkStudios%2Femote/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EmbarkStudios","download_url":"https://codeload.github.com/EmbarkStudios/emote/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EmbarkStudios%2Femote/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263215299,"owners_count":23431895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embark","reinforcement-learning"],"created_at":"2025-07-02T21:07:24.015Z","updated_at":"2025-07-02T21:07:24.778Z","avatar_url":"https://github.com/EmbarkStudios.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- Allow this file to not have a first line heading --\u003e\n\u003c!-- markdownlint-disable-file MD041 --\u003e\n\n\u003c!-- inline html --\u003e\n\u003c!-- markdownlint-disable-file MD033 --\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n# `🍒 emote`\n\n**E**mbark's **Mo**dular **T**raining **E**ngine - a flexible framework for\nreinforcement learning\n\n[![Embark](https://img.shields.io/badge/embark-open%20source-blueviolet.svg)](https://embark.dev)\n[![Embark](https://img.shields.io/badge/discord-ark-%237289da.svg?logo=discord)](https://discord.gg/dAuKfZS)\n[![Build status](https://badge.buildkite.com/968ac3c0bb075fb878f9f973ed91406c8b257b0f050c197542.svg?theme=github\u0026branch=main)](https://buildkite.com/embark-studios/emote)\n[![Docs status](https://img.shields.io/badge/Docs-latest-brightgreen)](https://embarkstudios.github.io/emote/)\n\n🚧 This project is a  **work in progress**. Things can and will change. 🚧\n\n\u003c/div\u003e\n\n\n## What it does\n\nEmote provides a way to build reusable components for creating reinforcement learning algorithms, and a\nlibrary of premade componenents built in this way. It is strongly inspired by the callback setup used\nby Keras and FastAI.\n\nAs an example, let us see how the SAC, the Soft Actor Critic algorithm by\n[Haarnoja et al.](https://arxiv.org/abs/1801.01290) can be written using Emote. The main algorithm in\nSAC is given in [Soft Actor-Critic Algorithms and Applications](https://arxiv.org/abs/1812.05905) and\nlooks like this:\n\n\u003cdiv align=\"center\"\u003e\n\n![Main SAC algorithm](./docs/haarnoja_sac.png)\n\n\u003c/div\u003e\n\nUsing the components provided with Emote, we can write this as\n\n```python\ndevice = torch.device(\"cpu\")\nenv = DictGymWrapper(AsyncVectorEnv(10 * [HitTheMiddle]))\ntable = DictObsTable(spaces=env.dict_space, maxlen=1000, device=device)\nmemory_proxy = MemoryTableProxy(table)\ndataloader = MemoryLoader(table, 100, 2, \"batch_size\")\n\nq1 = QNet(2, 1)\nq2 = QNet(2, 1)\npolicy = Policy(2, 1)\nln_alpha = torch.tensor(1.0, requires_grad=True)\nagent_proxy = FeatureAgentProxy(policy, device)\n\ncallbacks = [\n    QLoss(name=\"q1\", q=q1, opt=Adam(q1.parameters(), lr=8e-3)),\n    QLoss(name=\"q2\", q=q2, opt=Adam(q2.parameters(), lr=8e-3)),\n    PolicyLoss(pi=policy, ln_alpha=ln_alpha, q=q1, opt=Adam(policy.parameters())),\n    AlphaLoss(pi=policy, ln_alpha=ln_alpha, opt=Adam([ln_alpha]), n_actions=1),\n    QTarget(pi=policy, ln_alpha=ln_alpha, q1=q1, q2=q2),\n    SimpleGymCollector(env, agent_proxy, memory_proxy, warmup_steps=500),\n    FinalLossTestCheck([logged_cbs[2]], [10.0], 2000),\n]\n\ntrainer = Trainer(callbacks, dataloader)\ntrainer.train()\n```\n\nHere each callback in the `callbacks` list is its own reusable class that can readily be used\nfor other similar algorithms. The callback classes themselves are very straight forward to write.\nAs an example, here is the `PolicyLoss` callback.\n\n```python\nclass PolicyLoss(LossCallback):\n    def __init__(\n        self,\n        *,\n        pi: nn.Module,\n        ln_alpha: torch.tensor,\n        q: nn.Module,\n        opt: optim.Optimizer,\n        max_grad_norm: float = 10.0,\n        name: str = \"policy\",\n        data_group: str = \"default\",\n    ):\n        super().__init__(\n            name=name,\n            optimizer=opt,\n            network=pi,\n            max_grad_norm=max_grad_norm,\n            data_group=data_group,\n        )\n        self.policy = pi\n        self._ln_alpha = ln_alpha\n        self.q1 = q\n        self.q2 = q2\n\n    def loss(self, observation):\n        p_sample, logp_pi = self.policy(**observation)\n        q_pi_min = self.q1(p_sample, **observation)\n        # using reparameterization trick\n        alpha = torch.exp(self._ln_alpha).detach()\n        policy_loss = alpha * logp_pi - q_pi_min\n        policy_loss = torch.mean(policy_loss)\n        assert policy_loss.dim() == 0\n        return policy_loss\n```\n\n## Installation\n\nFor package management and environment handling we use `pants`. Install it from [pants](https://v1.pantsbuild.org/install.html). After `pants` is set up, verify that it is setup by running\n\n```bash\npants tailor ::\n```\n\n### Common problems\n\n**Box2d complains:** Box2d needs swig and python bindings. On apt-based systems try\n\n```bash\nsudo apt install swig\nsudo apt install python3.10-dev\n```\n\n**Python 3.10 is tricky to install:** For Ubuntu based distros try adding the deadsnakes PPA.\n\n## Contribution\n\n[![Contributor Covenant](https://img.shields.io/badge/contributor%20covenant-v1.4-ff69b4.svg)](../main/CODE_OF_CONDUCT.md)\n\nWe welcome community contributions to this project.\n\nPlease read our [Contributor Guide](CONTRIBUTING.md) for more information on how to get started.\nPlease also read our [Contributor Terms](CONTRIBUTING.md#contributor-terms) before you make any contributions.\n\nAny contribution intentionally submitted for inclusion in an Embark Studios project, shall comply with the Rust standard licensing model (MIT OR Apache 2.0) and therefore be dual licensed as described below, without any additional terms or conditions:\n\n### License\n\nThis contribution is dual licensed under EITHER OF\n\n* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or \u003chttp://www.apache.org/licenses/LICENSE-2.0\u003e)\n* MIT license ([LICENSE-MIT](LICENSE-MIT) or \u003chttp://opensource.org/licenses/MIT\u003e)\n\nat your option.\n\nFor clarity, \"your\" refers to Embark or any other licensee/user of the contribution.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembarkstudios%2Femote","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fembarkstudios%2Femote","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembarkstudios%2Femote/lists"}