{"id":16562801,"url":"https://github.com/charmve/puppygo","last_synced_at":"2025-06-12T19:33:21.457Z","repository":{"id":192718154,"uuid":"687263594","full_name":"Charmve/PuppyGo","owner":"Charmve","description":"vision language model and large language model powered embodied robot ","archived":false,"fork":false,"pushed_at":"2023-09-18T03:45:23.000Z","size":12314,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-15T09:05:25.589Z","etag":null,"topics":["chatgpt","deepmind","embodied-agent","palm2","rt-2","voxposer"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Charmve.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-05T02:05:57.000Z","updated_at":"2024-01-16T12:29:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"cf2277c6-aba5-480f-8b54-1a6e199edf12","html_url":"https://github.com/Charmve/PuppyGo","commit_stats":{"total_commits":5,"total_committers":2,"mean_commits":2.5,"dds":"0.19999999999999996","last_synced_commit":"e878c06a9d3600b336fab87dba83e1e7d7b34de3"},"previous_names":["charmve/puppygo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Charmve%2FPuppyGo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Charmve%2FPuppyGo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Charmve%2FPuppyGo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Charmve%2FPuppyGo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Charmve","download_url":"https://codeload.github.com/Charmve/PuppyGo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238737927,"owners_count":19522276,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","deepmind","embodied-agent","palm2","rt-2","voxposer"],"created_at":"2024-10-11T20:37:13.026Z","updated_at":"2025-02-13T21:31:15.703Z","avatar_url":"https://github.com/Charmve.png","language":null,"funding_links":["https://github.com/sponsors/Charmve?frequency=one-time\u0026sponsor=Charmve"],"categories":[],"sub_categories":[],"readme":"# PuppyGo\n\nVision language model and large language model powered embodied agent.\n\n\u003cdiv\u003e\n  \u003cp\u003e\n    \u003cimg src=\"src/voxposer.gif\" width=430\u003e\n    \u003cimg src=\"src/puppygo.jpg\" width=320\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n## Here’s what I did:\n\n- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents\n- extracts affordances and constraints from large language models and vision-language models to compose 3D value maps, which are used by motion planners to zero-shot synthesize trajectories for everyday manipulation tasks.\n- combine with e2e large model trainning framework, like UniAD;\n\n\n## This Package Is Sponsorware 💰💰💰\n\n[![](https://img.shields.io/static/v1?label=Sponsor\u0026message=%E2%9D%A4\u0026logo=GitHub\u0026color=%23fe8e86)](https://github.com/sponsors/Charmve?frequency=one-time\u0026sponsor=Charmve) https://github.com/sponsors/Charmve?frequency=one-time\u0026sponsor=Charmve\n\nThis repo was only available to my sponsors on GitHub Sponsors until I reached 15 sponsors.\n\nLearn more about **Sponsorware** at [github.com/sponsorware/docs](https://github.com/sponsorware/docs) 💰.\n\n\n![image](https://github.com/Charmve/PuppyGo/assets/29084184/3577e267-53a0-4904-909f-6733995e892e)\n\n\n\u003cbr\u003e\n\n## Execution under Disturbances\n\nBecause the language model output stays the same throughout the task, we can cache its output and re-evaluate the generated code using closed-loop visual feedback, which enables fast replanning using MPC. This enables VoxPoser to be robust to online disturbances.\n      \n\u003cdiv class=\"columns\"\u003e\n  \u003cdiv class=\"column has-text-centered\"\u003e\n    \u003cvideo id=\"dist1\" controls=\"\" muted=\"\" autoplay=\"\" loop=\"\" width=\"99%\"\u003e\n      \u003csource src=\"src/sort-trash-to-tray-dist.mp4\" type=\"video/mp4\"\u003e\n    \u003c/video\u003e\n    \u003cp style=\"text-align:center\"\u003e\n      \"Sort the paper trash into the blue tray.\"\n    \u003c/p\u003e\n  \u003c/div\u003e\n\n  \u003cdiv class=\"column has-text-centered\"\u003e\n    \u003cvideo id=\"dist2\" controls=\"\" muted=\"\" autoplay=\"\" loop=\"\" width=\"99%\"\u003e\n      \u003csource src=\"src/close-top-drawer-dist.mp4\" type=\"video/mp4\"\u003e\n    \u003c/video\u003e\n    \u003cp style=\"text-align:center\"\u003e\n      \"Close the top drawer.\"\n    \u003c/p\u003e\n  \u003c/div\u003e\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcharmve%2Fpuppygo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcharmve%2Fpuppygo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcharmve%2Fpuppygo/lists"}