{"id":31755722,"url":"https://github.com/hk-zh/language-conditioned-robot-manipulation-models","last_synced_at":"2025-10-09T19:01:52.848Z","repository":{"id":215350977,"uuid":"738703604","full_name":"hk-zh/language-conditioned-robot-manipulation-models","owner":"hk-zh","description":"https://arxiv.org/abs/2312.10807","archived":false,"fork":false,"pushed_at":"2024-12-01T11:42:43.000Z","size":1344,"stargazers_count":51,"open_issues_count":1,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-12-01T12:35:35.603Z","etag":null,"topics":["foundation-models","imitation-learning","language-conditioned-learning","large-languge-models","neural-symbolic","reinforcement-learning","robot-manipulation","visual-language-models"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hk-zh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-03T21:09:05.000Z","updated_at":"2024-12-01T11:42:47.000Z","dependencies_parsed_at":"2024-01-04T01:07:05.332Z","dependency_job_id":"9d857a43-2755-4c7c-9c5c-0b6d06c4ecdb","html_url":"https://github.com/hk-zh/language-conditioned-robot-manipulation-models","commit_stats":null,"previous_names":["hk-zh/language-conditioned-robot-manipulation-models"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hk-zh/language-conditioned-robot-manipulation-models","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hk-zh%2Flanguage-conditioned-robot-manipulation-models","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hk-zh%2Flanguage-conditioned-robot-manipulation-models/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hk-zh%2Flanguage-conditioned-robot-manipulation-models/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hk-zh%2Flanguage-conditioned-robot-manipulation-models/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hk-zh","download_url":"https://codeload.github.com/hk-zh/language-conditioned-robot-manipulation-models/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hk-zh%2Flanguage-conditioned-robot-manipulation-models/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001903,"owners_count":26083232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["foundation-models","imitation-learning","language-conditioned-learning","large-languge-models","neural-symbolic","reinforcement-learning","robot-manipulation","visual-language-models"],"created_at":"2025-10-09T19:00:30.550Z","updated_at":"2025-10-09T19:01:52.835Z","avatar_url":"https://github.com/hk-zh.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bridging Language and Action: Awesome Language-conditioned Robot Manipulation Models [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome)\n\n![alt text](graphs/overview.png)\n\n## News\n[November 30 2024] Extended Survey paper is available!\n\n[October 02 2024]  Cutting edge papers in 2024 are avaliable!!!\n## Table of the Content\n\n- [Survey Paper](#survey)\n- [Language-conditioned Reinforcement Learning](#language-conditioned-reinforcement-learning)\n- [Language-conditioned Imitation Learning](#language-conditioned-imitation-learning)\n  - [Behaviour Cloning](#behaviour-cloning)\n  - [Inverse Reinforcement Learning](#inverse-reinforcement-learning)\n- [Diffusion Policy](#diffusion-policy)\n- [Neuralsymbolic](#neuralsymbolic)\n- [Enpowered by LLMs](#empowered-by-llms)\n  - [Planning](#planning)\n  - [Reasoning](#reasoning)\n- [Enpowered by VLMs](#empowered-by-vlms)\n  - [Vision-Language-Action Models (VLAMs)](#vision-language-action-models-vlams)\n- [Comparative Analysis](#comparative-analysis)\n  - [Simulators](#simulators)\n  - [Benchmarks](#benchmarks)\n  - [Models](#models)\n\n## Survey\n\nThis paper is basically based on the survey paper:\n\n**[Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation](https://arxiv.org/abs/2312.10807)**\n\u003cbr /\u003e\nHongkuan Zhou, \nXiangtong Yao, \nOier Mees, \nYuan Meng, \nTed Xiao,\nYonatan Bisk, \nJean Oh,\nEdward Johns, \nMohit Shridhar, \nDhruv Shah, \nJesse Thomason, \nKai Huang, \nJoyce Chai,\nZhenshan Bing, \nAlois Knoll\n\u003cbr /\u003e\n\n## Language-conditioned Reinforcement Learning\n### Games\n- From language to goals: Inverse reinforcement learning for vision-based instruction following [[paper]](https://openreview.net/forum?id=r1lq1hRqYQ)\n- Grounding english commands to reward function [[paper]](https://www.roboticsproceedings.org/rss11/p18.pdf)\n- Learning to understand goal specifications by modelling reward [[paper]](https://openreview.net/pdf?id=H1xsSjC9Ym)\n- Beating atari with natural language guided reinforcement learning [[paper]](https://arxiv.org/abs/1704.05539) [[code]](https://github.com/ishan00/beating-atari-with-natural-language-guided-rl)\n- Using natural language for reward shaping in reinforcement learning [[paper]](https://www.ijcai.org/proceedings/2019/331)\n### Navigation \n- Gated-attention architectures for task-oriented language grounding [[paper]](https://cdn.aaai.org/ojs/11832/11832-13-15360-1-2-20201228.pdf) [[code]](https://github.com/devendrachaplot/DeepRL-Grounding)\n- Mapping instructions and visual observations to actions with reinforcement learning [[paper]](https://aclanthology.org/D17-1106/) \n- Modular multitask reinforcement learning with policy sketches [[paper]](https://dl.acm.org/doi/10.5555/3305381.3305399)\n- Representation learning for grounded spatial reasoning [[paper]](https://aclanthology.org/Q18-1004/)\n### Manipulation \n- Lancon-learn: Learning with language to enable generalization in multi-task manipulation [[paper]](https://ieeexplore.ieee.org/document/9667188) [[code]](https://github.com/hartikainen/metaworld/tree/reward-tweaks-rebase)\n- Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards [[paper]](https://proceedings.mlr.press/v155/goyal21a.html)[[code]](https://github.com/prasoongoyal/PixL2R)\n- Learning from symmetry: Meta-reinforcement learning with symmetrical behaviors and language instructions [[paper]](https://arxiv.org/abs/2209.10656)[[website]](https://tumi6robot.wixsite.com/symmetry/) \n- Meta-reinforcement learning via language instructions [[paper]](https://arxiv.org/abs/2209.04924)[[code]](https://github.com/yaoxt3/MILLION)[[website]](https://tumi6robot.wixsite.com/million)\n- Learning language-conditioned robot behavior from offline data and crowd-sourced annotation [[paper]](https://proceedings.mlr.press/v164/nair22a/nair22a.pdf) \n- Concept2robot: Learning manipulation concepts from instructions and human demonstrations [[paper]](https://www.roboticsproceedings.org/rss16/p082.pdf) \n\n## Language-conditioned Imitation Learning\n### Behaviour Cloning\n- Language conditioned imitation learning over unstructured data [[paper]](https://arxiv.org/abs/2005.07648) [[code]]() [[website]](https://language-play.github.io/)\n- Bc-z: Zero-shot task generalization with robotic imitation learning [[paper]](https://arxiv.org/abs/2202.02005) \n- What matters in language-conditioned robotic imitation learning over unstructured data [[paper]](https://arxiv.org/abs/2204.06252) [[code]](https://github.com/lukashermann/hulc)[[website]](http://hulc.cs.uni-freiburg.de/) \n- Grounding language with visual affordances over unstructured data [[paper]](https://arxiv.org/abs/2210.01911) [[code]](https://github.com/mees/hulc2)[[website]](http://hulc2.cs.uni-freiburg.de/)\n- Language-conditioned imitation learning with base skill priors under unstructured data [[paper]](https://arxiv.org/abs/2305.19075) [[code]](https://github.com/hk-zh/spil) [[website]](https://hk-zh.github.io/spil/)\n- Pay attention!- robustifying a deep visuomotor policy through task-focused visual attention [[paper]](https://arxiv.org/abs/1809.10093)\n- Language-conditioned imitation learning for robot manipulation tasks [[paper]](https://arxiv.org/abs/2010.12083)\n- Multimodal Diffusion Transformer for Learning from Play [[paper]](https://openreview.net/pdf?id=nvtxqMGpn1)\n- Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models [[paper]](https://arxiv.org/pdf/2310.10639) [[code]](https://github.com/kvablack/susie) [[website]](https://rail-berkeley.github.io/susie/)\n- PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play [[paper]](https://arxiv.org/pdf/2312.04549) [[website]](https://play-fusion.github.io/)\n- ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation [[paper]](https://openreview.net/pdf?id=W0zgY2mBTA8) [[code]](https://github.com/zhouxian/act3d-chained-diffuser) [[website]](https://github.com/zhouxian/act3d-chained-diffuser)\n- GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields [[paper]](https://arxiv.org/pdf/2308.16891) [[code]](https://github.com/YanjieZe/GNFactor) [[website]](https://yanjieze.com/GNFactor/)\n- DNAct: Diffusion Guided Multi-Task 3D Policy Learning [[paper]](https://arxiv.org/pdf/2403.04115) [[website]](https://dnact.github.io/)\n- 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations [[paper]](https://arxiv.org/pdf/2402.10885) [[code]](https://github.com/nickgkan/3d_diffuser_actor) [[website]](https://3d-diffuser-actor.github.io/)\n- Vision-Language Foundation Models as Effective Robot Imitators [[paper]](https://arxiv.org/pdf/2311.01378)\n- OpenVLA:An Open-Source Vision-Language-Action Model [[paper]](https://arxiv.org/pdf/2406.09246) [[code]](https://github.com/openvla/openvla) [[website]](https://openvla.github.io/)\n- Open X-Embodiment: Robotic Learning Datasets and RT-X Models [[paper]](https://arxiv.org/pdf/2310.08864)\n- 3D-VLA: A 3D Vision-Language-Action Generative World Model [[paper]](https://openreview.net/pdf?id=EZcFK8HupF) [[code]](https://github.com/UMass-Foundation-Model/3D-VLA) [[website]](https://vis-www.cs.umass.edu/3dvla/)\n- Octo: An Open-Source Generalist Robot Policy [[paper]](https://arxiv.org/pdf/2405.12213) [[code]](https://github.com/octo-models/octo) [[website]](https://octo-models.github.io/)\n\n### Inverse Reinforcement Learning\n- Grounding english commands to reward function [[paper]](https://www.roboticsproceedings.org/rss11/p18.pdf)\n- From language to goals: Inverse reinforcement learning for vision-based instruction following [[paper]](https://arxiv.org/abs/1902.07742) \n\n## Diffusion Policy\n- Multimodal Diffusion Transformer for Learning from Play [[paper]](https://arxiv.org/abs/2407.05996)\n- Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models [[paper]](https://arxiv.org/abs/2310.10639) [[code]](https://github.com/kvablack/susie) [[website]](https://rail-berkeley.github.io/susie/)\n- PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play [[paper]](https://arxiv.org/abs/2312.04549) [[website]](https://play-fusion.github.io/)\n- ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation [[paper]](https://openreview.net/pdf?id=W0zgY2mBTA8) [[code]](https://github.com/zhouxian/act3d-chained-diffuser) [[website]](https://chained-diffuser.github.io/)\n- DNAct: Diffusion Guided Multi-Task 3D Policy Learning [[paper]](https://arxiv.org/pdf/2403.04115) [[website]](https://dnact.github.io/)\n- 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations [[paper]](https://arxiv.org/abs/2402.10885)\n\n## Neuralsymbolic\n### Learning for Reasoning\n- Hierarchical understanding in robotic manipulation: A knowledge-based framework [[paper]](https://www.mdpi.com/2076-0825/13/1/28)\n- Semantic Grasping Via a Knowledge Graph of Robotic Manipulation: A Graph Representation Learning Approach [[paper]](https://ieeexplore.ieee.org/iel7/7083369/7339444/09830861.pdf)\n- Knowledge Acquisition and Completion for Long-Term Human-Robot Interactions using Knowledge Graph Embedding [[paper]](https://arxiv.org/pdf/2301.06834)\n### Reasoning for Learning\n- Tell me dave: Context-sensitive grounding of natural language to manipulation instructions [[paper]](https://www.semanticscholar.org/paper/Tell-me-Dave%3A-Context-sensitive-grounding-of-to-Misra-Sung/8cb52a0424992807dceeaf2af740364b2e80c438)\n- Neuro-symbolic procedural planning with commonsense prompting [[paper]](https://arxiv.org/abs/2206.02928)\n- Reinforcement Learning Based Navigation with Semantic Knowledge of Indoor Environments [[paper]](https://ieeexplore.ieee.org/abstract/document/8919366/?casa_token=7x7LciTVSGYAAAAA:Ou51YDO9Zz6Ozk_7XTjvhdlW2IL5gOv8g9XK5tlrTOLvE2bRsuZvD2E7MRSCyIZ4c2zm-EvDJSI)\n- Learning Neuro-Symbolic Skills for Bilevel Planning [[paper]](Learning Neuro-Symbolic Skills for Bilevel Planning)\n### Learning-Reasoning\n- Learning Neuro-symbolic Programs for Language Guided Robot Manipulation [[paper]](https://arxiv.org/abs/2211.06652) [[code]](https://github.com/dair-iitd/nsrmp) [[website]](https://nsrmp.github.io/)\n- Long-term robot manipulation task planning with scene graph and semantic knowledge [[paper]](https://www.emerald.com/insight/content/doi/10.1108/RIA-09-2022-0226/full/html)\n## Empowered by LLMs\n### Planning\n- Sayplan: Grounding large language models using 3d scene graphs for scalable task planning [[paper]](https://arxiv.org/abs/2307.06135)\n- Language models as zero-shot planners: Extracting actionable knowledge for embodied agents [[paper]](https://arxiv.org/abs/2201.07207)\n- Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents [[paper]](https://arxiv.org/abs/2302.01560)\n- Progprompt: Generating situated robot task plans using large language models [[paper]](https://arxiv.org/abs/2209.11302)\n- Robots that ask for help: Uncertainty alignment for large language model planners [[paper]](https://arxiv.org/abs/2307.01928)\n- Task and motion planning with large language models for object rearrangement [[paper]](https://arxiv.org/abs/2303.06247)\n- Do as i can, not as i say: Grounding language in robotic affordances [[paper]](https://arxiv.org/abs/2204.01691)\n- The 2014 international planning competition: Progress and trends [[paper]](https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2571)\n- Robot task planning via deep reinforcement learning: a tabletop object sorting application [[paper]](https://ieeexplore.ieee.org/document/8914278)\n- Robot task planning and situation handling in open worlds [[paper]](https://arxiv.org/abs/2210.01287) [[code]](https://github.com/yding25/GPT-Planner) [[website]](https://cowplanning.github.io/)\n- Embodied Task Planning with Large Language Models [[paper]](https://arxiv.org/abs/2307.01848) [[code]](https://github.com/Gary3410/TaPA) [[website]](https://gary3410.github.io/TaPA/)\n- Text2motion: From natural language instructions to feasible plans [[paper]](https://arxiv.org/abs/2303.12153) [[website]](https://sites.google.com/stanford.edu/text2motion)\n- Large language models as commonsense knowledge for large-scale task planning [[paper]](https://arxiv.org/abs/2305.14078) [[code]](https://github.com/1989Ryan/llm-mcts) [[website]](https://llm-mcts.github.io/)\n- Alphablock: Embodied finetuning for vision-language reasoning in robot manipulation [[paper]](https://arxiv.org/abs/2305.18898) \n- Learning to reason over scene graphs: a case study of finetuning gpt-2 into a robot language model for grounded task planning [[paper]](https://www.frontiersin.org/articles/10.3389/frobt.2023.1221739/full) [[code]](https://github.com/dnandha/RobLM)\n- Scaling up and distilling down: Language-guided robot skill acquisition [[paper]](https://arxiv.org/abs/2307.14535)[[code]](https://github.com/real-stanford/scalingup) [[website]](https://www.cs.columbia.edu/~huy/scalingup/)\n- Stap: Sequencing task-agnostic policies [[paper]](https://ieeexplore.ieee.org/document/10160220) [[code]](https://github.com/agiachris/STAP)[[website]](https://sites.google.com/stanford.edu/stap/home)\n- Inner monologue: Embodied reasoning through planning with language models [[paper]](https://arxiv.org/abs/2207.05608) [[website]](https://innermonologue.github.io/) \n### Reasoning\n- Rearrangement:A challenge for embodied ai [[paper]](https://arxiv.org/abs/2011.01975)\n- The threedworld transport challenge: A visually guided task and motion planning benchmark for physically realistic embodied ai [[paper]](https://ieeexplore.ieee.org/document/9812329) \n- Tidy up my room: Multi-agent cooperation for service tasks in smart environments [[paper]](https://dl.acm.org/doi/abs/10.3233/AIS-190524) \n- A quantifiable stratification strategy for tidy-up in service robotics [[paper]](https://ieeexplore.ieee.org/document/9542842)\n- Tidybot: Personalized robot assistance with large language models [[paper]](https://arxiv.org/abs/2305.05658)\n- Housekeep: Tidying virtual households using commonsense reasoning [[paper]](https://arxiv.org/abs/2205.10712)\n- Building cooperative embodied agents modularly with large language models [[paper]](https://arxiv.org/abs/2307.02485)\n- Socratic models: Composing zero-shot multimodal reasoning with language [[paper]](https://arxiv.org/abs/2204.00598)\n- Voyager: An open-ended embodied agent with large language models [[paper]](https://arxiv.org/abs/2305.16291)\n- Translating natural language to planning goals with large-language models [[paper]](https://arxiv.org/abs/2302.05128)\n## Empowered by VLMs\n- Cliport: What and where pathways for robotic manipulation [[paper]](https://arxiv.org/abs/2109.12098) [[code]](https://github.com/cliport/cliport) [[website]](https://cliport.github.io/)\n- Transporter networks: Rearranging the visual world for robotic manipulation [[paper]](https://proceedings.mlr.press/v155/zeng21a/zeng21a.pdf) [[code]](https://github.com/google-research/ravens) [[website]](https://transporternets.github.io/)\n- Simple but effective: Clip embeddings for embodied ai [[paper]](https://openaccess.thecvf.com/content/CVPR2022/papers/Khandelwal_Simple_but_Effective_CLIP_Embeddings_for_Embodied_AI_CVPR_2022_paper.pdf) \n- Instruct2act: Mapping multi-modality instructions to robotic actions with large language model [[paper]](https://arxiv.org/abs/2305.11176) [[code]](https://github.com/OpenGVLab/Instruct2Act)\n- Latte: Language trajectory transformer [[paper]](https://arxiv.org/abs/2208.02918) [[code]](https://github.com/arthurfenderbucker/LaTTe-Language-Trajectory-TransformEr)\n- Embodied Task Planning with Large Language Models [[paper]](https://arxiv.org/abs/2307.01848) [[code]](https://github.com/Gary3410/TaPA) [[website]](https://gary3410.github.io/TaPA/)\n- Palm-e: An embodied multimodal language model [[paper]](https://arxiv.org/abs/2303.03378)  [[website]](https://palm-e.github.io/)\n- Socratic models: Composing zero-shot multimodal reasoning with language [[paper]](https://arxiv.org/abs/2204.00598)\n- Pretrained language models as visual planners for human assistance [[paper]](https://openaccess.thecvf.com/content/ICCV2023/papers/Patel_Pretrained_Language_Models_as_Visual_Planners_for_Human_Assistance_ICCV_2023_paper.pdf) [[code]](https://github.com/facebookresearch/vlamp)\n- Open-world object manipulation using pre-trained vision-language models [[paper]](https://arxiv.org/abs/2303.00905) [[website]](https://robot-moo.github.io/)\n- Robotic skill acquisition via instruction augmentation with vision-language models [[paper]](https://arxiv.org/abs/2211.11736) [[website]](https://instructionaugmentation.github.io/)\n- Language reward modulation for pretraining reinforcement learning [[paper]](https://arxiv.org/abs/2308.12270) [[code]](https://github.com/ademiadeniji/lamp)\n- Vision-language models as success detectors [[paper]](https://proceedings.mlr.press/v232/du23b.html)\n### Vision Language Action Models (VLAMs)\n- A Generalist Agent [[paper]](https://openreview.net/pdf?id=1ikK0kHjvj)\n- RT-1: Robotics Transformer for Real-World Control at Scale [[paper]](https://arxiv.org/pdf/2212.06817) [[code]](https://github.com/google-research/robotics_transformer) [[website]](https://robotics-transformer1.github.io/)\n- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [[paper]](https://proceedings.mlr.press/v229/zitkovich23a/zitkovich23a.pdf)\n- Vision-Language Foundation Models as Effective Robot Imitators [[paper]](https://arxiv.org/pdf/2311.01378)\n- OpenVLA:An Open-Source Vision-Language-Action Model [[paper]](https://arxiv.org/pdf/2406.09246) [[code]](https://github.com/openvla/openvla) [[website]](https://openvla.github.io/)\n- Open X-Embodiment: Robotic Learning Datasets and RT-X Models [[paper]](https://arxiv.org/pdf/2310.08864)\n- 3D-VLA: A 3D Vision-Language-Action Generative World Model [[paper]](https://openreview.net/pdf?id=EZcFK8HupF) [[code]](https://github.com/UMass-Foundation-Model/3D-VLA) [[website]](https://vis-www.cs.umass.edu/3dvla/)\n\n## Comparative Analysis\n### Simulators\n| Simulator | Description |\n| - | - |\n| [PyBullet](https://pybullet.org/wordpress/) | \u003cdiv style=\"width: 350pt\"\u003eWith its origins rooted in the Bullet physics engine, PyBullet transcends the boundaries of conventional simulation platforms, offering a wealth of tools and resources for tasks ranging from robot manipulation and locomotion to computer-aided design analysis.\u003c/div\u003e | \u003cdiv style=\"width: 250pt\"\u003e Shao et al., Mees et al.  leverage pybullet to build a table-top environment to conduct object manipulations tasks. \u003c/div\u003e|\n| [MuJoCo](https://mujoco.org/) | \u003cdiv style=\"width: 350pt\"\u003e MuJoCo, short for \"Multi-Joint dynamics with Contact\", originates from the vision of creating a physics engine tailored for simulating articulated and deformable bodies. It has evolved into an essential tool for exploring diverse domains, from robot locomotion and manipulation to human movement and control. \u003c/div\u003e|\n| [CoppeliaSim](https://www.coppeliarobotics.com/) | \u003cdiv style=\"width: 350pt\"\u003e CoppeliaSim is formerly known as V-REP (Virtual Robot Experimentation Platform). It offers a comprehensive environment for simulating and prototyping robotic systems, enabling users to create, analyze, and optimize a wide spectrum of robotic applications. Its origins as an educational tool have evolved into a full-fledged simulation framework, revered for its versatility and user-friendly interface. \u003c/div\u003e|\n| [NVIDIA Omniverse](https://www.nvidia.com/en-us/omniverse/) | \u003cdiv style=\"width: 350pt\"\u003e NVIDIA Omniverse offers real-time physics simulation and lifelike rendering, creating a virtual environment for comprehensive testing and fine-tuning of robotic manipulation algorithms and control strategies, all prior to their actual deployment in the physical realm. \u003c/div\u003e|\n| [Unity](https://unity.com/) | \u003cdiv style=\"width: 350pt\"\u003e Unity is a cross-platform game engine developed by Unity Technologies. Renowned for its user-friendly interface and powerful capabilities, Unity has become a cornerstone in the worlds of video games, augmented reality (AR), virtual reality (VR), and also simulations. \u003c/div\u003e| \n\n### Benchmarks\n\u003ctable\u003e\n    \u003ctr align=\"center\"\u003e\n        \u003cth rowspan=\"2\" \u003eBenchmark\u003c/th\u003e\n        \u003cth rowspan=\"2\"\u003eSimulation Engine\u003c/th\u003e\n        \u003cth rowspan=\"2\"\u003eManipulator\u003c/th\u003e\n        \u003ctd colspan=\"3\"\u003e \u003cb\u003e Observation\u003c/td\u003e\n        \u003cth rowspan=\"2\"\u003eTool used\u003c/th\u003e\n        \u003cth rowspan=\"2\"\u003eMulti-agents\u003c/th\u003e\n        \u003cth rowspan=\"2\"\u003eLong-horizon\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003cth\u003eRGB\u003c/th\u003e\n        \u003cth\u003eDepth\u003c/th\u003e\n        \u003cth\u003eMasks\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr align=\"center\"\u003e\n        \u003ctd\u003e\u003ca href=\"http://calvin.cs.uni-freiburg.de/\"\u003eCALVIN\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003ePyBullet\u003c/td\u003e\n        \u003ctd\u003eFranka Panda\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr align=\"center\"\u003e\n        \u003ctd\u003e\u003ca href=\"https://meta-world.github.io/\"\u003eMeta-world\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eMuJoCo\u003c/td\u003e\n        \u003ctd\u003eSawyer\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr align=\"center\"\u003e\n        \u003ctd\u003e\u003ca href=\"https://arxiv.org/abs/2308.00937\"\u003eLEMMA\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eNVIDIA Omniverse\u003c/td\u003e\n        \u003ctd\u003eUR10 \u0026 UR5\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr align=\"center\"\u003e\n        \u003ctd\u003e\u003ca href=\"https://github.com/stepjam/RLBench\"\u003eRLbench\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eCoppeliaSim\u003c/td\u003e\n        \u003ctd\u003eFranka Panda\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr align=\"center\"\u003e\n        \u003ctd\u003e\u003ca href=\"https://github.com/vimalabs/VIMABench\"\u003eVIMAbench\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003ePybullet\u003c/td\u003e\n        \u003ctd\u003eUR5\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr align=\"center\"\u003e\n        \u003ctd\u003e\u003ca href=\"https://cisnlp.github.io/lohoravens-webpage/\"\u003eLoHoRavens\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003ePybullet\u003c/td\u003e\n        \u003ctd\u003eUR5\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr align=\"center\"\u003e\n        \u003ctd\u003e\u003ca href=\"https://arnold-benchmark.github.io/\"\u003eARNOLD\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eNVIDIA Isaac Gym\u003c/td\u003e\n        \u003ctd\u003eFranka Panda\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n\n### Models\n| Model | Year | Benchmark | Simulation Engine | Language Module| Perception Module | Real World Experiment | LLM | Reinforcement Learning | Imitation Learning |\n| ------ | ------ | :-----------: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n|  [DREAMCELL](https://arxiv.org/abs/1903.08309) |  2019  |        #         | -       | LSTM | *   | ❌ | ❌ | ❌ | ✅ |\n|  [PixL2R](https://proceedings.mlr.press/v155/goyal21a.html) |  2020  |    Meta-World    | MuJoCo  | LSTM | CNN | ❌ | ❌ | ✅ | ❌ |\n| [Concept2Robot](https://www.roboticsproceedings.org/rss16/p082.pdf) | 2020 | # | PyBullet | BERT | ResNet-18 | ❌ | ❌ | ❌ | ✅ |\n| [LanguagePolicy](https://proceedings.neurips.cc/paper/2020/hash/9909794d52985cbc5d95c26e31125d1a-Abstract.html) | 2020 | # | CoppeliaSim | GLoVe | Faster RCNN | ❌ | ❌ | ❌ | ✅ |\n| [LOReL](https://proceedings.mlr.press/v164/nair22a.html)| 2021 | Meta-World | MuJoCo | distillBERT | CNN | ✅ | ❌ | ❌ | ✅ |\n| [CARE](https://proceedings.mlr.press/v139/sodhani21a.html) | 2021 | Meta-World | MuJoCo | RoBERTa | * | ❌ | ✅ | ✅ | ❌ |\n| [MCIL](https://arxiv.org/abs/2005.07648) | 2021 | # | MuJoCo | MUSE | CNN | ❌ | ❌ | ❌ | ✅ |\n| [BC-Z](https://arxiv.org/abs/2202.02005) | 2021 | # | - | MUSE | ResNet18 | ✅ | ❌ | ❌ | ✅ |\n| [CLIPort](https://proceedings.mlr.press/v164/shridhar22a.html) | 2021 | # | Pybullet | CLIP | CLIP/ResNet | ✅ | ❌ | ❌ | ✅ |\n| [LanCon-Learn](https://ieeexplore.ieee.org/document/9667188) | 2022 | Meta-World | MuJoCo | GLoVe | * | ❌ | ❌ | ✅ | ✅ | \n| [MILLON](https://arxiv.org/abs/2209.04924) | 2022 | Meta-World| MuJoCo | GLoVe | * | ✅ | ❌ | ✅ | ❌ | \n| [PaLM-SayCan](https://arxiv.org/abs/2204.01691) | 2022 | # | - | PaLM | ViLD | ✅ | ✅ | ✅ | ✅ |\n| [ATLA](https://arxiv.org/abs/2206.13074) | 2022 | # | PyBullet | BERT-Tiny | CNN | ❌ | ✅ | ✅ | ❌ |\n| [HULC](https://arxiv.org/abs/2204.06252) | 2022 | CALVIN | Pybullet | MiniLM-L3-v2 | CNN | ❌ | ❌ | ❌ | ✅ |\n| [PerAct](https://arxiv.org/abs/2209.05451) | 2022 | RLbench | CoppelaSim | CLIP | ViT | ✅ | ❌ | ❌ | ✅ |\n| [RT-1](https://arxiv.org/abs/2212.06817) | 2022 | # | - | USE | EfficientNet-B3 |  ✅ | ✅ | ❌ | ❌ |\n| [LATTE](https://arxiv.org/abs/2208.02918) | 2023 | # | CoppeliaSim | distillBERT, CLIP | CLIP | ✅ | ❌ |  ❌ | ❌ |\n| [DIAL](https://arxiv.org/abs/2211.11736) | 2022 | # | - | CLIP | CLIP |  ✅ | ✅ | ❌ | ✅ |\n| [R3M](https://arxiv.org/abs/2203.12601) | 2022 | # | - | distillBERT | ResNet | ✅ | ❌ | ❌ | ✅ |\n| [Inner Monologue](https://arxiv.org/abs/2207.05608) | 2022 | # | - | CLIP | CLIP | ✅ | ✅ | ❌ | ❌ |\n| [NLMap](https://ieeexplore.ieee.org/document/10161534) | 2023 | # | - | CLIP | ViLD | ✅ | ✅ | ❌ | ✅ |\n| [Code as Policies](https://ieeexplore.ieee.org/document/10160591) | 2023 | # | - | GPT3, Codex | ViLD | ✅ | ✅ | ❌ | ❌ | \n| [PROGPROMPT](https://arxiv.org/abs/2209.11302) |  2023 | Virtualhome | Unity3D | GPT-3 | * | ✅ | ✅ | ❌ | ❌ |\n| [Language2Reward](https://arxiv.org/abs/2306.08647) | 2023 | # | MuJoCo MPC | GPT-4 | * | ✅ | ✅ | ✅ | ❌ |\n| [LfS](https://arxiv.org/abs/2209.10656) | 2023 | Meta-World | MuJoCo | Cons. Parser | * | ✅ | ❌ | ✅ | ❌ |\n| [HULC++](https://arxiv.org/abs/2210.01911)| 2023 | CALVIN | PyBullet | MiniLM-L3-v2 | CNN | ✅ | ❌ | ❌ | ✅ |\n| [LEMMA](https://arxiv.org/abs/2308.00937) | 2023 | LEMMA | NVIDIA Omniverse | CLIP | CLIP | ❌ | ❌ | ❌ | ✅ |\n| [SPIL](https://arxiv.org/abs/2305.19075)| 2023 | CALVIN | PyBullet | MiniLM-L3-v2 | CNN | ✅ | ❌ | ❌ | ✅ |\n| [PaLM-E](https://proceedings.mlr.press/v202/driess23a.html) | 2023 | # | PyBullet | PaLM | ViT | ✅ | ✅ | ❌ | ✅ | \n| [LAMP](https://arxiv.org/abs/2308.12270) | 2023 | RLbench | CoppelaSim  | ChatGPT | R3M | ❌ | ✅ | ✅ | ❌ |\n| [MOO](https://arxiv.org/abs/2303.00905) | 2023 | # | - | OWL-ViT | OWL-ViT | ✅ | ❌ | ❌ | ✅ |\n| [Instruction2Act](https://arxiv.org/abs/2305.11176) | 2023 | VIMAbench | PyBullet | ChatGPT | CLIP | ❌ | ✅ |  ❌ | ❌ |\n| [VoxPoser](https://arxiv.org/abs/2307.05973) | 2023 | # | SAPIEN | CPT-4 | OWL-ViT  | ✅ | ✅ | ❌ | ❌| \n| [SuccessVQA](https://arxiv.org/abs/2303.07280) | 2023 | # | IA Playroom | Flamingo | Flamingo | ✅ | ✅ | ❌ | ❌| \n| [VIMA](https://arxiv.org/abs/2210.03094) | 2023 | VIMAbench | PyBullet | T5 model | ViT | ✅ | ✅ | ❌ | ✅| \n| [TidyBot](https://arxiv.org/abs/2305.05658) | 2023 | # | - | GPT-3 | CLIP | ✅ | ✅ | ❌ | ❌| \n| [Text2Motion](https://arxiv.org/abs/2303.12153) | 2023 | # | - | GPT-3, Codex | * | ✅ | ✅ | ✅ | ❌| \n| [LLM-GROP](https://arxiv.org/abs/2303.06247) | 2023 | # | Gazebo | GPT-3 | * | ✅ | ✅ | ❌ | ❌| \n| [Scaling Up](https://arxiv.org/abs/2307.14535) | 2023 | # | MuJoCo | CLIP, GPT-3 | ResNet-18 | ✅ | ✅ | ❌ | ✅ | \n| [Socratic Models](https://openreview.net/pdf?id=kdHpWogtX6Y) | 2023 | # | - | RoBERTa, GPT-3 | CLIP | ✅ | ✅ | ❌ | ❌| \n| [SayPlan](https://arxiv.org/abs/2307.06135) | 2023 | # | - | GPT-4 | * | ✅ | ✅ | ❌ | ❌ | \n| [RT-2](https://arxiv.org/abs/2307.15818) | 2023 | # | - | PaLI-X, PaLM-E | PaLI-X, PaLM-E | ✅ | ✅ | ❌ | ❌ | \n| [KNOWNO](https://arxiv.org/abs/2307.01928) | 2023 | # | PyBullet | PaLM-2L | * | ✅ | ✅ | ❌ | ❌ | \n| [Diffusion Policy](https://arxiv.org/abs/2303.04137) |2023| Push-T | MuJoCo| - | CNN |  ✅ | ❌ | ❌ | ✅ |\n| [MDT](https://arxiv.org/html/2407.05996v1) | 2023 | CALVIN | PyBullet | CLIP | CLIP | ❌ | ❌ | ✅ | ✅ |\n| [Scaling Up](https://arxiv.org/abs/2307.14535) | 2023 | # | MuJoCo| CLIP | CLIP | ✅ | ❌ | ❌ | ✅ |\n| [Playfussion](https://arxiv.org/abs/2312.04549) | 2023 | CALVIN | PyBullet | Sentence-BERT | ResNet-18 | ✅ | ❌ | ❌ | ✅ |\n| [ChainedDiffuer](https://proceedings.mlr.press/v229/xian23a.html) | 2023 | RLbench | CoppelaSim | CLIP | CLIP | ✅ | ❌ | ❌ | ✅ |\n| [GNFactor](https://arxiv.org/abs/2308.16891) | 2023 | RLbench | CoppelaSIm | CLIP | NeRF | ✅ | ❌ | ❌ | ✅ |\n| [DNAct](https://arxiv.org/abs/2403.04115) | 2024 | RLbench | CoppelaSim | CLIP | NeRF, PointNext | ✅ | ❌ | ❌ | ✅ |\n| [3D Diffuser Actor](https://arxiv.org/abs/2402.10885) | 2024 | CALVIN | PyBullet | CLIP | CLIP | ✅ | ❌ | ❌ | ✅ |\n| [RoboFlamingo](https://arxiv.org/abs/2311.01378) | 2024 | CALVIN | PyBullet | OpenFlamingo | OpenFlamingo | ❌ | ✅ | ❌ | ✅ |\n| [OpenVLA](https://arxiv.org/abs/2406.09246) | 2024 | Open X-Embodiment | - | Llama 2 7B | DinoV2 \u0026 SigLIP |  ✅ | ✅ | ❌ | ✅ | \n| [RT-X](https://arxiv.org/abs/2310.08864) | 2024 | Open X-Embodiment | - | PaLi-X/PaLM-E | PaLi-X/PaLM-E | ✅ | ✅ | ❌ | ✅ | \n| [PIVOT](https://openreview.net/forum?id=051jaf8MQy) | 2024 | Open X-Embodiment | - | GPT-4/Gemini | GPT-4/Gemini | ✅ | ✅ | ❌ | ❌ | \n| [3D-VLA](https://openreview.net/forum?id=EZcFK8HupF) | 2024 | RL-Bench \u0026 CALVIN | CoppeliaSim \u0026 PyBullet | 3D-LLM | 3D-LLM | ❌ | ✅ | ❌ | ✅ |\n| [Octo](https://arxiv.org/abs/2405.12213) | 2024 | Open X-Embodiment | - | T5 | CNN | ✅ | ✅ | ❌ | ✅ | \n| [ECoT](https://arxiv.org/abs/2407.08693) | 2024 | BridgeData V2 | - | Llama 2 7B | DinoV2 \u0026 SigLIP | ✅ | ✅ | ❌ | ✅ | \n\n## Citation\nIf you find this survey useful please consider cite:\n```bibtex\n@article{zhou2023language,\n  author       = {Hongkuan Zhou and\n                  Xiangtong Yao and\n                  Oier Mees and\n                  Yuan Meng and\n                  Ted Xiao and\n                  Yonatan Bisk and\n                  Jean Oh and\n                  Edward Johns and\n                  Mohit Shridhar and\n                  Dhruv Shah and\n                  Jesse Thomason and\n                  Kai Huang and\n                  Joyce Chai and\n                  Zhenshan Bing and\n                  Alois Knoll},\n  title        = {Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation},\n  journal      = {CoRR},\n  volume       = {abs/2312.10807},\n  year         = {2023},\n  url          = {https://doi.org/10.48550/arXiv.2312.10807}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhk-zh%2Flanguage-conditioned-robot-manipulation-models","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhk-zh%2Flanguage-conditioned-robot-manipulation-models","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhk-zh%2Flanguage-conditioned-robot-manipulation-models/lists"}