{"id":19401039,"url":"https://github.com/google-research/language-table","last_synced_at":"2025-10-04T10:22:57.159Z","repository":{"id":64176794,"uuid":"561880527","full_name":"google-research/language-table","owner":"google-research","description":"Suite of human-collected datasets and a multi-task continuous control benchmark for open vocabulary visuolinguomotor learning.","archived":false,"fork":false,"pushed_at":"2025-03-21T22:53:33.000Z","size":1734,"stargazers_count":313,"open_issues_count":31,"forks_count":26,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-12T06:05:44.019Z","etag":null,"topics":["robotics","robotics-simulation"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-04T17:47:33.000Z","updated_at":"2025-04-08T08:22:16.000Z","dependencies_parsed_at":"2025-01-08T10:00:36.531Z","dependency_job_id":"267b1d9a-a720-4d9b-a58d-ad7427d43118","html_url":"https://github.com/google-research/language-table","commit_stats":{"total_commits":21,"total_committers":8,"mean_commits":2.625,"dds":0.6666666666666667,"last_synced_commit":"ee3104061f35e1261a2e28cc2a29ea84e7156661"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flanguage-table","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flanguage-table/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flanguage-table/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flanguage-table/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/language-table/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254509476,"owners_count":22082891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["robotics","robotics-simulation"],"created_at":"2024-11-10T11:16:48.892Z","updated_at":"2025-10-04T10:22:52.121Z","avatar_url":"https://github.com/google-research.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Language Table\n\nLanguage-Table is a suite of human-collected datasets and a multi-task continuous control benchmark for open vocabulary visuolinguomotor learning.\n\n![](./docs/real.jpeg)      |  ![](./docs/sim.jpeg)\n:-------------------------:|:-------------------------:|\n\n## Installation\n\nInstallation with `pip`. `requirements.txt` contains dependencies for running\nthe environment and simple dataset examples.\n\n```\npython3 -m venv ./ltvenv\nsource ./ltvenv/bin/activate\npip install -r ./requirements.txt\nexport PYTHONPATH=${PWD}:$PYTHONPATH\n```\n\nFor running the full train script, install using `requirements_static.txt`, as\nthis contains pinned versions for running the full train script.\n\n```\npython3 -m venv ./ltvenvtrain\nsource ./ltvenvtrain/bin/activate\npip install --no-deps -r ./requirements_static.txt\nexport PYTHONPATH=${PWD}:$PYTHONPATH\n```\n## Quickstart\n\n### Examples\n#### Scripts\nRun and edit the following examples:\n\nLoad the environment and run 5 random steps:\n\n```\npython3 language_table/examples/environment_example.py\n```\n\nLoad dataset and print first 5 elements:\n\n```\npython3 language_table/examples/dataset_example.py\n```\n\n#### Train\n\n```\nsource ./ltvenvtrain/bin/activate\nmkdir -p /tmp/language_table_train/\nPROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python  python language_table/train/main.py --config=./language_table/train/configs/language_table_sim_local.py --workdir=/tmp/language_table_train/\n```\n\n#### Colab\nSee the [colab](https://colab.research.google.com/github/google-research/language-table/blob/main/language_table/examples/language_table_tutorial.ipynb) for a more complete tutorial.\n\n### Data\n```\nimport tensorflow_datasets as tfds\ndata_directory = 'gs://gresearch/robotics/language_table/0.0.1/'\ndataset = tfds.builder_from_directory(data_directory).as_dataset()\n```\n\n### Environment\n```\nfrom language_table.environments import blocks\nfrom language_table.environments import language_table\nfrom language_table.environments.rewards import block2block\n\nenv = language_table.LanguageTable(\n  block_mode=blocks.LanguageTableBlockVariants.BLOCK_8,\n  reward_factory=block2block.BlockToBlockReward,\n  control_frequency=10.0,\n)\nobs = env.reset()\n```\n\n## Datasets\n\n### Descriptions\n\n* **Real Robot**\n  * **language_table**: 442,226 episodes of real robot relabeled data.\n* **Simulation (human)**\n  * **language_table_sim**: 181,020 episodes of simulation relabeled data.\n  * **language_table_blocktoblock_sim**: 8,000 episodes of single task \"block to block\" data.\n  * **language_table_blocktoblock_4block_sim**: 8,298 episodes of single task \"block to block\" data in the 4 block configuration.\n* **Simulation (oracle)**\n  * **language_table_blocktoblock_oracle_sim**: 200,000 episodes of single task \"block to block\" data from an oracle scripted agent.\n  * **language_table_blocktoblockrelative_oracle_sim**: 200,000 episodes of single task \"block-to-block-relative\" data from an oracle scripted agent.\n  * **language_table_blocktoabsolute_oracle_sim**: 200,000 episodes of single task \"block to absolute location\" data from an oracle scripted agent.\n  * **language_table_blocktorelative_oracle_sim**: 200,000 episodes of single task \"block to relative location\" data from an oracle scripted agent.\n  * **language_table_separate_oracle_sim**: 200,000 episodes of single task \"separate blocks\" data from an oracle scripted agent.\n\n### Summary Table\n\nDataset | Real/sim | Controlled by | Language-labeled by | # episodes\n--------| --------- | ------------- | ----------------- | --------: \nlanguage_table | real | human | human | 442,226\nlanguage_table_sim | sim | human | human | 181,020\nlanguage_table_blocktoblock_sim | sim | human | scripted | 8,000\nlanguage_table_blocktoblock_4block_sim |  sim | human | scripted | 8,298\nlanguage_table_blocktoblock_oracle_sim | sim | oracle | scripted | 200,000\nlanguage_table_blocktoblockrelative_oracle_sim | sim | oracle | scripted | 200,000\nlanguage_table_blocktoabsolute_oracle_sim | sim | oracle | scripted | 200,000\nlanguage_table_blocktorelative_oracle_sim | sim | oracle | scripted | 200,000\nlanguage_table_separate_oracle_sim | sim | oracle | scripted | 200,000\n\n### Paths\n\nDataset | Data Location\n--------| --------------\nlanguage_table | [gs://gresearch/robotics/language_table](https://console.cloud.google.com/storage/browser/gresearch/robotics/language_table/0.0.1/)\nlanguage_table_sim | [gs://gresearch/robotics/language_table_sim](https://console.cloud.google.com/storage/browser/gresearch/robotics/language_table_sim/0.0.1/)\nlanguage_table_blocktoblock_sim | [gs://gresearch/robotics/language_table_blocktoblock_sim](https://console.cloud.google.com/storage/browser/gresearch/robotics/language_table_blocktoblock_sim/0.0.1/)\nlanguage_table_blocktoblock_4block_sim | [gs://gresearch/robotics/language_table_blocktoblock_4block_sim](https://console.cloud.google.com/storage/browser/gresearch/robotics/language_table_blocktoblock_4block_sim/0.0.1/)\nlanguage_table_blocktoblock_oracle_sim | [gs://gresearch/robotics/language_table_blocktoblock_oracle_sim](https://console.cloud.google.com/storage/browser/gresearch/robotics/language_table_blocktoblock_oracle_sim/0.0.1/)\nlanguage_table_blocktoblockrelative_oracle_sim | [gs://gresearch/robotics/language_table_blocktoblockrelative_oracle_sim](https://console.cloud.google.com/storage/browser/gresearch/robotics/language_table_blocktoblockrelative_oracle_sim/0.0.1/)\nlanguage_table_blocktoabsolute_oracle_sim | [gs://gresearch/robotics/language_table_blocktoabsolute_oracle_sim](https://console.cloud.google.com/storage/browser/gresearch/robotics/language_table_blocktoabsolute_oracle_sim/0.0.1/)\nlanguage_table_blocktorelative_oracle_sim | [gs://gresearch/robotics/language_table_blocktorelative_oracle_sim](https://console.cloud.google.com/storage/browser/gresearch/robotics/language_table_blocktorelative_oracle_sim/0.0.1/)\nlanguage_table_separate_oracle_sim | [gs://gresearch/robotics/language_table_separate_oracle_sim](https://console.cloud.google.com/storage/browser/gresearch/robotics/language_table_separate_oracle_sim/0.0.1/)\n\n## Checkpoints\n\nName | Config | Checkpoint Location\n-----| -------| -------------------\nBC+ResNet Sim| language_table/train/configs/language_table_resnet_sim_local.py | [gs://gresearch/robotics/language_table_checkpoints/bc_resnet_sim_checkpoint_955000](https://storage.googleapis.com/gresearch/robotics/language_table_checkpoints/bc_resnet_sim_checkpoint_955000)\n\n## Interactive Language: Talking to Robots in Real Time\n[Project Website](https://interactive-language.github.io/)\u0026nbsp;\u0026nbsp;•\u0026nbsp;\u0026nbsp;[PDF](https://arxiv.org/pdf/2210.06407.pdf)\n\n*Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, Pete Florence*\n\n**Abstract.** We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. \"make a smiley face out of blocks\". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots.\n\n## Note\n\nThis is not an officially supported Google product.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Flanguage-table","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Flanguage-table","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Flanguage-table/lists"}