{"id":13458340,"url":"https://github.com/octo-models/octo","last_synced_at":"2025-03-24T15:31:10.396Z","repository":{"id":212265948,"uuid":"731075982","full_name":"octo-models/octo","owner":"octo-models","description":"Octo is a transformer-based robot policy trained on a diverse mix of 800k robot trajectories.","archived":false,"fork":false,"pushed_at":"2024-07-31T00:26:15.000Z","size":26039,"stargazers_count":1138,"open_issues_count":84,"forks_count":200,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-03-20T04:45:27.610Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://octo-models.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/octo-models.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-13T09:58:56.000Z","updated_at":"2025-03-19T13:22:32.000Z","dependencies_parsed_at":"2023-12-27T05:30:28.910Z","dependency_job_id":"903bc604-5543-4d14-bf2b-47461aeeb5aa","html_url":"https://github.com/octo-models/octo","commit_stats":null,"previous_names":["octo-models/octo"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octo-models%2Focto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octo-models%2Focto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octo-models%2Focto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octo-models%2Focto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/octo-models","download_url":"https://codeload.github.com/octo-models/octo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245297975,"owners_count":20592509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T09:00:51.116Z","updated_at":"2025-03-24T15:31:08.360Z","avatar_url":"https://github.com/octo-models.png","language":"Python","funding_links":[],"categories":["Robot Arm","Model","🔗 Useful Resources","🧩 Foundation Models for Robotics","Robot Learning Policies with Visual Input"],"sub_categories":["Robotics Foundation Model","Base VLA Models","Large Robot Foundation Models"],"readme":"# Octo\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Static Badge](https://img.shields.io/badge/Project-Page-a)](https://octo-models.github.io/)\n![](https://github.com/rail-berkeley/octo/workflows/run-debug/badge.svg)\n![](https://github.com/rail-berkeley/octo/workflows/pre-commit/badge.svg)\n\nThis repo contains code for training and finetuning Octo generalist robotic policies (GRPs).\nOcto models are transformer-based diffusion policies, trained on a diverse mix of 800k robot trajectories.\n\n## Get Started\n\nFollow the installation instructions, then load a pretrained Octo model! See [examples](examples/) for guides to zero-shot evaluation and finetuning and [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1z0vELj_lX9OWeoMG_WvXnQs43aPOEAhz?usp=sharing)\nfor an inference example.\n\n```python\nfrom octo.model.octo_model import OctoModel\nmodel = OctoModel.load_pretrained(\"hf://rail-berkeley/octo-base-1.5\")\nprint(model.get_pretty_spec())\n```\n\n![Octo model](docs/assets/teaser.jpg)\n\nOut of the box, Octo supports multiple RGB camera inputs, can control various robot arms,\nand can be instructed via language commands or goal images.\nOcto uses a modular attention structure in its transformer backbone, allowing it to be effectively finetuned\nto robot setups with new sensory inputs, action spaces, and morphologies, using only a small target domain\ndataset and accessible compute budgets.\n\n\n## Installation\n```bash\nconda create -n octo python=3.10\nconda activate octo\npip install -e .\npip install -r requirements.txt\n```\nFor GPU:\n```bash\npip install --upgrade \"jax[cuda11_pip]==0.4.20\" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html\n```\n\nFor TPU\n```bash\npip install --upgrade \"jax[tpu]==0.4.20\" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html\n```\nSee the [Jax Github page](https://github.com/google/jax) for more details on installing Jax.\n\nTest the installation by finetuning on the debug dataset:\n```bash\npython scripts/finetune.py --config.pretrained_path=hf://rail-berkeley/octo-small-1.5 --debug\n```\n\n## Checkpoints\n\nYou can find pretrained Octo checkpoints [here](https://huggingface.co/rail-berkeley).\nAt the moment we provide the following model versions:\n\n| Model                                                         | Inference on 1x NVIDIA 4090 | Size       |\n|---------------------------------------------------------------|-----------------------------|------------|\n| [Octo-Base](https://huggingface.co/rail-berkeley/octo-base)   | 13 it/sec                   | 93M Params |\n| [Octo-Small](https://huggingface.co/rail-berkeley/octo-small) | 17 it/sec                   | 27M Params |\n\n\n## Examples\n\nWe provide simple [example scripts](examples) that demonstrate how to use and finetune Octo models,\nas well as how to use our data loader independently. We provide the following examples:\n\n|                                                                      |                                                                                                                    |\n|----------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|\n| [Octo Inference](examples/01_inference_pretrained.ipynb)             | Minimal example for loading and running a pretrained Octo model                                                    |\n| [Octo Finetuning](examples/02_finetune_new_observation_action.py)    | Minimal example for finetuning a pretrained Octo models on a small dataset with a new observation and action space |\n| [Octo Rollout](examples/03_eval_finetuned.py)                        | Run a rollout of a pretrained Octo policy in a Gym environment                                                     |\n| [Octo Robot Eval](examples/04_eval_finetuned_on_robot.py)            | Evaluate a pretrained Octo model on a real WidowX robot                                                            |\n| [OpenX Dataloader Intro](examples/05_dataloading.ipynb)              | Walkthrough of the features of our Open X-Embodiment data loader                                                   |\n| [OpenX PyTorch Dataloader](examples/06_pytorch_oxe_dataloader.ipynb) | Standalone Open X-Embodiment data loader in PyTorch                                                                |\n\n\n## Octo Pretraining\n\nTo reproduce our Octo pretraining on 800k robot trajectories, run:\n```bash\npython scripts/train.py --config scripts/configs/octo_pretrain_config.py:\u003csize\u003e --name=octo --config.dataset_kwargs.oxe_kwargs.data_dir=... --config.dataset_kwargs.oxe_kwargs.data_mix=oxe_magic_soup ...\n```\n\nTo download the pretraining dataset from the [Open X-Embodiment Dataset](https://robotics-transformer-x.github.io/),\ninstall the [rlds_dataset_mod package](https://github.com/kpertsch/rlds_dataset_mod)\nand run the [prepare_open_x.sh script](https://github.com/kpertsch/rlds_dataset_mod/blob/main/prepare_open_x.sh).\nThe total size of the pre-processed dataset is ~1.2TB.\n\nWe run pretraining using a TPUv4-128 pod in 8 hours for the Octo-S model and in 14 hours for Octo-B.\n\n\n## Octo Finetuning\n\nWe provide a [minimal example](examples/02_finetune_new_observation_action.py) for finetuning with a new observation and action space.\n\nWe also provide a more advanced finetuning script that allows you to change hyperparameters via a config file and logs finetuning\nmetrics. To run advanced finetuning, use:\n```bash\npython scripts/finetune.py --config.pretrained_path=hf://rail-berkeley/octo-small-1.5\n```\n\nWe offer three finetuning modes depending on the parts of the model that are kept frozen: ```head_only```, ```head_mlp_only```, and ```full``` to finetune the full model.\nAdditionally, one can specify the task type to finetune with: ```image_conditioned```, ```language_conditioned``` or ```multimodal``` for both.\nFor example, to finetune the full transformer with image inputs only use:\n```--config=finetune_config.py:full,image_conditioned```.\n\n\n## Octo Evaluation\n\nLoading and running a trained Octo model is as easy as:\n```python\nfrom octo.model import OctoModel\n\nmodel = OctoModel.load_pretrained(\"hf://rail-berkeley/octo-small-1.5\")\ntask = model.create_tasks(texts=[\"pick up the spoon\"])\naction = model.sample_actions(observation, task, rng=jax.random.PRNGKey(0))\n```\n\nWe provide examples for evaluating Octo [in a simulated Gym environment](examples/03_eval_finetuned.py) as well\nas [on a real WidowX robot](examples/04_eval_finetuned_on_robot.py).\n\nTo evaluate on your own environment, simply wrap it in a Gym interface and follow the instructions in the\n[Eval Env README](examples/envs/README.md).\n\n\n## Code Structure\n\n|                     | File                                                    | Description                                                                   |\n|---------------------|---------------------------------------------------------|-------------------------------------------------------------------------------|\n| Hyperparameters     | [config.py](scripts/configs/config.py)                  | Defines all hyperparameters for the training run.                             |\n| Pretraining Loop    | [train.py](scripts/train.py)                            | Main pretraining script.                                                         |\n| Finetuning Loop     | [finetune.py](scripts/finetune.py)                      | Main finetuning script.                                                       |\n| Datasets            | [dataset.py](octo/data/dataset.py)                      | Functions for creating single / interleaved datasets + data augmentation.     |\n| Tokenizers          | [tokenizers.py](octo/model/components/tokenizers.py)    | Tokenizers that encode image / text inputs into tokens.                       |\n| Octo Model          | [octo_model.py](octo/model/octo_model.py)               | Main entry point for interacting with Octo models: loading, saving, and inference. |\n| Model Architecture  | [octo_module.py](octo/model/octo_module.py)             | Combines token sequencing, transformer backbone and readout heads.            |\n| Visualization       | [visualization_lib.py](octo/utils/visualization_lib.py) | Utilities for offline qualitative \u0026 quantitative eval.                        |\n\n## FAQ\n#### What is the `timestep_pad_mask` in the observation dictionary?\nThe `timestep_pad_mask` indicates which observations should be attended to, which is important when using multiple timesteps of observation history. Octo was trained with a history window size of 2, meaning the model can predict an action using both the current observation and the previous observation. However, at the very beginning of the trajectory, there is no previous observation, so we need to set `timestep_pad_mask=False` at the corresponding index. If you use Octo with a window size of 1, `timestep_pad_mask` should always just be `[True]`, indicating that the one and only observation in the window should be attended to. Note that if you wrap your robot environment with the `HistoryWrapper` (see [gym_wrappers.py](octo/utils/gym_wrappers.py)), the `timestep_pad_mask` key will be added to the observation dictionary for you.\n#### What is `pad_mask_dict` in the observation dictionary?\nWhile `timestep_pad_mask` indicates which observations should be attended to on a timestep level, `pad_mask_dict` indicates which elements of the observation should be attended to within a single timestep. For example, for datasets without language labels, `pad_mask_dict[\"language_instruction\"]` is set to `False`. For datasets without a wrist camera, `pad_mask_dict[\"image_wrist\"]` is set to `False`. For convenience, if a key is missing from the observation dict, it is equivalent to setting `pad_mask_dict` to `False` for that key.\n#### Does `model.sample_actions([...])` return the full trajectory to solve a task?\nOcto was pretrained with an action chunking size of 4, meaning it predicts the next 4 actions at once. You can choose to execute all these actions before sampling new ones, or only execute the first action before sampling new ones (also known as receding horizon control). You can also do something more advanced like [temporal ensembling](octo/utils/gym_wrappers.py).\n\n## Updates for Version 1.5\n- Improved cross-attention between visual and language tokens by repeating language tokens at every timestep in the context window.\n- Augmented the language instructions in the data with rephrasings from GPT-3.5.\n- Bug fixes:\n  - Turned off dropout in the diffusion head due to incompatibility with layer norm.\n  - Fixed an off-by-one error with the attention mask.\n  - Fixed an issue where different image augmentations did not get fresh random seeds.\n\n## Citation\n\n```\n@inproceedings{octo_2023,\n    title={Octo: An Open-Source Generalist Robot Policy},\n    author = {{Octo Model Team} and Dibya Ghosh and Homer Walke and Karl Pertsch and Kevin Black and Oier Mees and Sudeep Dasari and Joey Hejna and Charles Xu and Jianlan Luo and Tobias Kreiman and {You Liang} Tan and Pannag Sanketi and Quan Vuong and Ted Xiao and Dorsa Sadigh and Chelsea Finn and Sergey Levine},\n    booktitle = {Proceedings of Robotics: Science and Systems},\n    address  = {Delft, Netherlands},\n    year = {2024},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Focto-models%2Focto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Focto-models%2Focto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Focto-models%2Focto/lists"}