{"id":13435372,"url":"https://github.com/araffin/srl-zoo","last_synced_at":"2025-04-14T12:50:34.684Z","repository":{"id":74413953,"uuid":"108873440","full_name":"araffin/srl-zoo","owner":"araffin","description":"State Representation Learning (SRL) zoo with PyTorch - Part of S-RL Toolbox","archived":false,"fork":false,"pushed_at":"2019-08-06T14:58:09.000Z","size":4728,"stargazers_count":163,"open_issues_count":1,"forks_count":18,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-03-28T01:53:39.821Z","etag":null,"topics":["autoencoder","deep-learning","forward-model","inverse-model","neural-network","pytorch","reinforcement-learning","representation-learning","srl","state-representation-learning","vae"],"latest_commit_sha":null,"homepage":"https://srl-zoo.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/araffin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-30T15:52:28.000Z","updated_at":"2025-03-18T08:16:48.000Z","dependencies_parsed_at":"2023-03-04T21:45:28.653Z","dependency_job_id":null,"html_url":"https://github.com/araffin/srl-zoo","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/araffin%2Fsrl-zoo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/araffin%2Fsrl-zoo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/araffin%2Fsrl-zoo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/araffin%2Fsrl-zoo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/araffin","download_url":"https://codeload.github.com/araffin/srl-zoo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248884506,"owners_count":21177427,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoencoder","deep-learning","forward-model","inverse-model","neural-network","pytorch","reinforcement-learning","representation-learning","srl","state-representation-learning","vae"],"created_at":"2024-07-31T03:00:35.199Z","updated_at":"2025-04-14T12:50:34.621Z","avatar_url":"https://github.com/araffin.png","language":"Python","funding_links":[],"categories":["Libraries"],"sub_categories":[],"readme":"# State Representation Learning Zoo with PyTorch (part of S-RL Toolbox)\n\nA collection of State Representation Learning (SRL) methods for Reinforcement Learning, written using PyTorch.\n\n\u003cimg src=\"imgs/enjoy-latent-real-baxter.gif\" align=\"right\" width=\"50%\"/\u003e\n\nSRL Zoo Documentation: https://srl-zoo.readthedocs.io/\n\nS-RL Toolbox Documentation: https://s-rl-toolbox.readthedocs.io/\n\nS-RL Toolbox Repository: https://github.com/araffin/robotics-rl-srl\n\n\nAvailable methods:\n\n- Autoencoder (reconstruction loss)\n- Denoising Autoencoder (DAE)\n- Forward Dynamics model\n- Inverse Dynamics model\n- Reward prediction loss\n- Variational Autoencoder (VAE) and beta-VAE\n- SRL with Robotic Priors + extensions (stereovision, additional priors)\n- Supervised Learning\n- Principal Component Analysis (PCA)\n- Triplet Network (for stereovision only)\n- Combination and stacking of methods\n- Random Features\n- **[experimental]** Reward Prior, Episode-prior, Perceptual Similarity loss (DARLA), Mutual Information loss\n\nRelated papers:\n- \"Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based robotics\" (Raffin et al. 2018) [https://arxiv.org/abs/1901.08651](https://arxiv.org/abs/1901.08651)\n- \"S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning\" (Raffin et al., 2018) [https://arxiv.org/abs/1809.09369](https://arxiv.org/abs/1809.09369)\n- \"State Representation Learning for Control: An Overview\" (Lesort et al., 2018), link: [https://arxiv.org/pdf/1802.04181.pdf](https://arxiv.org/pdf/1802.04181.pdf)\n\n\u003ca href=\"https://www.youtube.com/watch?v=qNsHMkIsqJc\"\u003e\u003cimg src=\"imgs/rl_toolboxplay.jpg\"/\u003e\u003c/a\u003e\n\n## Documentation\n\nDocumentation is available online: [https://srl-zoo.readthedocs.io/](https://srl-zoo.readthedocs.io/)\n\n## Installation\n\nPlease read the [documentation](https://s-rl-toolbox.readthedocs.io/) for more details, we provide anaconda env files and docker images.\n\n## Learning a State Representation\n\nTo learn a state representation, you need to enforce constrains on the representation using one or more losses. For example, to train an autoencoder, you need to use a reconstruction loss.\nMost losses are not exclusive, that means you can combine them.\n\nAll losses are defined in `losses/losses.py`. The available losses are:\n\n- autoencoder: reconstruction loss, using current and next observation\n- denoising autoencoder (dae): same as for the auto-encoder, except that the model reconstruct inputs from\n                             noisy observations containing a random zero-pixel mask\n- vae: (beta)-VAE loss (reconstruction + kullback leiber divergence loss)\n- inverse: predict the action given current and next state\n- forward: predict the next state given current state and taken action\n- reward: predict the reward (positive or not) given current and next state\n- priors: robotic priors losses (see \"Learning State Representations with Robotic Priors\")\n- triplet: triplet loss for multi-cam setting (see *Multiple Cameras* section in the doc)\n\n**[Experimental]**\n- reward-prior: Maximises the correlation between states and rewards (does not make sense for sparse reward)\n- episode-prior: Learn an episode-agnostic state space, thanks to a discriminator distinguishing states from same/different episodes\n- perceptual similarity loss (for VAE):  Instead of the reconstruction loss in the beta-VAE loss, it\nuses the distance between the reconstructed input and real input in the embedding of a pre-trained DAE.\n - mutual information loss: Maximises the mutual information between states and rewards\n\n\nAll possible arguments can be display using `python train.py --help`. You can limit the training set size (`--training-set-size` argument), change the minibatch size (`-bs`), number of epochs (`--epochs`), ...\n\n## Datasets: Simulated Environments and Real Robots\n\nAlthough the data can be generated easily using the RL repo in simulation (cf [Generating Data](https://s-rl-toolbox.readthedocs.io/en/latest/guide/envs.html#generating-data)), we provide datasets with a real baxter:\n\n- [Dataset 1](https://mega.nz/#!kN8gTbDQ!s0dRO5TmsNAorIhSsPvPeflaH-r7uYPKgUF2c5Fxxqk)\n- [Dataset 2](https://mega.nz/#!AcVkCJgA!mPzXkY3jkM3BPfCN5LGSi9pZfD6urf0m5wTCtkk1_uk) with multiple cameras\n\n\n### Examples\n\nYou can download an example dataset [here](https://drive.google.com/open?id=154qMJHgUnzk0J_Hxmr2jCnV1ipS7o1D5).\n\nTrain an inverse model:\n```\npython train.py --data-folder data/path/to/dataset --losses inverse\n```\n\nTrain an autoencoder:\n```\npython train.py --data-folder data/path/to/dataset --losses autoencoder\n```\n\nCombining an autoencoder with an inverse model is as easy as:\n```\npython train.py --data-folder data/path/to/dataset --losses autoencoder inverse\n```\n\nYou can as well specify the weight of each loss:\n```\npython train.py --data-folder data/path/to/dataset --losses autoencoder:1 inverse:10\n```\n\nPlease read the [documentation](https://s-rl-toolbox.readthedocs.io/) for more examples.\n\n\n## Running Tests\n\nDownload the test datasets [kuka_gym_test](https://drive.google.com/open?id=154qMJHgUnzk0J_Hxmr2jCnV1ipS7o1D5) and [kuka_gym_dual_test](https://drive.google.com/open?id=15Fhqr4-kai4b8qQWiq2mEAWW5ZqH5qID) and put it in `data/` folder.\n```\n./run_tests.sh\n```\n\n\n## Troubleshooting\n\n### CUDA out of memory error\n\n1.  python train.py --data-folder data/staticButtonSimplest\n```\nRuntimeError: cuda runtime error (2) : out of memory at /b/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.cu:66\n```\n\nSOLUTION 1: Decrease the batch size, e.g. 32-64 in GPUs with little memory.\n\nSOLUTION 2 Use simple 2-layers neural network model\npython train.py --data-folder data/staticButtonSimplest --model-type mlp\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faraffin%2Fsrl-zoo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faraffin%2Fsrl-zoo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faraffin%2Fsrl-zoo/lists"}