{"id":43493173,"url":"https://github.com/amathislab/lattice","last_synced_at":"2026-02-03T10:17:43.491Z","repository":{"id":168670897,"uuid":"644446012","full_name":"amathislab/lattice","owner":"amathislab","description":"[NeurIPS 2023] Latent Exploration for Reinforcement Learning","archived":false,"fork":false,"pushed_at":"2024-02-23T08:57:42.000Z","size":21152,"stargazers_count":20,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-25T23:38:08.330Z","etag":null,"topics":["exploration","inductive-bias","reinforcement-learning","sensorimotor-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amathislab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-05-23T14:29:18.000Z","updated_at":"2024-04-25T07:46:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"9a33966a-76bd-4b05-bce8-f1cf71f54731","html_url":"https://github.com/amathislab/lattice","commit_stats":null,"previous_names":["amathislab/lattice"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/amathislab/lattice","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amathislab%2Flattice","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amathislab%2Flattice/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amathislab%2Flattice/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amathislab%2Flattice/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amathislab","download_url":"https://codeload.github.com/amathislab/lattice/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amathislab%2Flattice/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29041057,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-03T10:09:22.136Z","status":"ssl_error","status_checked_at":"2026-02-03T10:09:16.814Z","response_time":96,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["exploration","inductive-bias","reinforcement-learning","sensorimotor-learning"],"created_at":"2026-02-03T10:17:42.714Z","updated_at":"2026-02-03T10:17:43.483Z","avatar_url":"https://github.com/amathislab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lattice (Latent Exploration for Reinforcement Learning)\n\nThis repository includes the implementation of Lattice exploration from the paper [Latent Exploration for Reinforcement Learning](https://arxiv.org/abs/2305.20065), published at NeurIPS 2023. \n\nLattice introduces random perturbations in the latent state of the policy network, which result in correlated noise across the system's actuators. This form of latent noise can facilitate exploration when controlling high-dimensional systems, especially with redundant actuation, and may find low-effort solutions. A short video explaining the project can be found on [YouTube](https://www.youtube.com/watch?v=_CCF3GDM9jY). \n\nLattice builds on top of Stable Baselines 3 (version 1.6.1) and it is here implemented for Recurrent PPO and SAC. Integration with a more recent version of Stable Baselines 3 and compatibility with more algorithms is [currently under development](https://github.com/albertochiappa/stable-baselines3).\n\nThis project was developed by Alberto Silvio Chiappa, Alessandro Marin Vargas, Ann Zixiang Huang and Alexander Mathis (EPFL).\n\n## MyoChallenge 2023\n\nWe used Lattice train the top submission of the NeurIPS 2023 competition [MyoChallenge](https://sites.google.com/view/myosuite/myochallenge/myochallenge-2023?authuser=0), object manipulation track. With curriculum learning, reward shaping and Lattice exploration we trained a policy to control a biologically-realistic arm with 63 muscles and 27 degrees of freedom to place random objects inside a box of variable shape:\n\n![relocate](/data/images/myochallenge_2023.gif)\n\nWe outperformed the other best solutions both in score and effort:\n\n\u003cimg src=\"/data/images/myochallenge_ranking.png\" alt=\"drawing\" width=\"70%\"/\u003e\n\nWe have also created [a dedicated repository](https://github.com/amathislab/myochallenge_2023eval) for the solution, where we have released the pretrained weights of all the curriculum steps.\n\n## Installation\n\nWe recommend using a Docker container to execute the code of this repository. We provide both the docker image [albertochiappa/myo-cuda-pybullet](https://hub.docker.com/repository/docker/albertochiappa/myo-cuda-pybullet) in DockerHub and the Dockerfile in the [docker](/docker/) folder to create the same docker image locally.\n\nIf you prefer to manually create a Conda environment, you can do so with the commands:\n\n```bash\nconda create --name lattice python=3.8.10\nconda activate lattice\npip install -r docker/requirements.txt\npip install myosuite==1.2.4 \npip install --upgrade cloudpickle==2.2.0 pickle5==0.0.11 pybullet==3.2.5\n```\n\nPlease note that there is a version error with some packages, e.g. `stable_baselines3`, requiring a later version of `gym` which `myosuite` is incompatible with. For this reason we could not include all the requirements in `docker/requirements.txt`. In our experiments the stated incompatibility did not cause any error.\n\n## Training a policy with Lattice\n\nWe provide scripts for various environments of the [MyoSuite](https://sites.google.com/view/myosuite) and [PyBullet](https://pybullet.org). \n\nTraining a policy is as easy as\n\n```bash\npython main_pose_elbow.py --use_lattice\n```\n\nif you have created a conda environment.\n\nIf you prefer to use the readily available docker container, you can train like this:\n\n```bash\ndocker run --rm --gpus all -it \\ \n--mount type=bind,src=\"$(pwd)/src\",target=/src \\\n--mount type=bind,src=\"$(pwd)/data\",target=/data \\ \n--mount type=bind,src=\"$(pwd)/output\",target=/output \\ \nalbertochiappa/myo-cuda-pybullet \\\npython3 src/main_pose_elbow.py --use_lattice\n```\n\nThe previous command will start training in the `Elbow Pose` enviornment using Recurrent PPO. Simply change the main script name to start training for a different environment. The output of the training, including the configuration used to select the hyperparameters and the tensorboard logs, are saved in a subfolder of `output/`, named as the current date. The code outputs useful information to monitor the training in Tensorboard format. You can run Tensorboard in the output folder to visualize the learning curves and much more. The different configuration hyperparameters can be set from the command line, e.g., by running \n\n```bash\npython src/main_humanoid.py --use_sde --use_lattice --freq=8\n```\n\nIn this case, a policy will be trained with SAC in the Humanoid environment, using state-dependent Lattice with update period 8.\n\n## Structure of the repository\n\n* src/\n  * envs/\n    * Modified MyoSuite environments, when we used different parameters from the default (cf. manuscript)\n    * Factory to instantiate all the environemtns used in the project\n  * metrics/\n    * Stable Baselines 3 callbacks to register useful information during the training\n  * models/\n    * Implementation of Lattice\n    * Adaptation of SAC and PPO to use Lattice\n  * train/\n    * Trainer class used to manage the trainings\n  * main_*\n    * One main file per environment, to start a training\n* data/\n  * configuration files for the MyoSuite environments\n* docker-cuda\n  * Definition of the Dockerfile to create the image used to run the experiments, with GPU support\n\n## Reference\n\nIf our work was useful to your research, please cite:\n\n```\n@article{chiappa2023latent,\n  title={Latent exploration for reinforcement learning},\n  author={Chiappa, Alberto Silvio and Vargas, Alessandro Marin and Huang, Ann Zixiang and Mathis, Alexander},\n  journal={Advances in Neural Information Processing Systems (NeurIPS)},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famathislab%2Flattice","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famathislab%2Flattice","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famathislab%2Flattice/lists"}