{"id":19156255,"url":"https://github.com/kyegomez/autort","last_synced_at":"2025-04-09T23:21:27.393Z","repository":{"id":216150265,"uuid":"740582416","full_name":"kyegomez/AutoRT","owner":"kyegomez","description":"Implementation of AutoRT: \"AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents\"","archived":false,"fork":false,"pushed_at":"2024-11-11T07:01:19.000Z","size":2608,"stargazers_count":39,"open_issues_count":1,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-02T22:07:23.071Z","etag":null,"topics":["ai","artificial-intelligence","attention-is-all-you-need","attention-mechanism","gpt4","machine-learning","ml","multi-modal","multimodal-learning","robotics","robots","ros","swarm","swarms"],"latest_commit_sha":null,"homepage":"https://discord.gg/GYbXvDGevY","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyegomez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["kyegomez"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2024-01-08T16:31:15.000Z","updated_at":"2025-03-12T19:59:30.000Z","dependencies_parsed_at":"2024-11-09T08:35:49.453Z","dependency_job_id":"ce76686d-7975-45f7-8853-3099b57d447f","html_url":"https://github.com/kyegomez/AutoRT","commit_stats":null,"previous_names":["kyegomez/autort"],"tags_count":0,"template":false,"template_full_name":"kyegomez/Python-Package-Template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FAutoRT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FAutoRT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FAutoRT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FAutoRT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyegomez","download_url":"https://codeload.github.com/kyegomez/AutoRT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248126366,"owners_count":21051910,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","attention-is-all-you-need","attention-mechanism","gpt4","machine-learning","ml","multi-modal","multimodal-learning","robotics","robots","ros","swarm","swarms"],"created_at":"2024-11-09T08:33:52.199Z","updated_at":"2025-04-09T23:21:27.359Z","avatar_url":"https://github.com/kyegomez.png","language":"Python","funding_links":["https://github.com/sponsors/kyegomez"],"categories":[],"sub_categories":[],"readme":"[![Multi-Modality](agorabanner.png)](https://discord.gg/qUtxnK2NMf)\n\n# AutoRT\n![AutoRTImage](autort.png)\nImplementation of AutoRT: \"AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents\". This repo will implement the multi agent system that transforms a scene into a list of ranked and priortized tasks for an robotic action model to execute. This is an very effective setup that I personally believe is the future for swarming robotic foundation models!\n\nThis project will be implemented using Swarms, for the various llms and use the official RT-1 as the robotic action model.\n\n[PAPER LINK](https://auto-rt.github.io/static/pdf/AutoRT.pdf)\n\n## Install\n`$ pip3 install autort-swarms`\n\n\n## Usage\n\n\n### AutoRTAgent\nA single AutoRT agent that: analyzes a scene using visual COT -\u003e generate tasks -\u003e filter tasks -\u003e execute it with a robotic transformer.\n```python\n# Import necessary modules\nimport os\nfrom autort import AutoRTSwarm, AutoRTAgent\n\n# Set the OpenAI API key\nopenai_api_key = os.getenv(\"OPENAI_API_KEY\")\n\n# Define a list of AutoRTAgent instances\nagents = [\n    AutoRTAgent(openai_api_key, max_tokens=1000),\n    AutoRTAgent(openai_api_key, max_tokens=1000),\n]\n\n# Create an instance of AutoRTSwarm with the agents and datastore\nautort_swarm = AutoRTSwarm(agents)\n\n# Run the AutoRTSwarm with the given inputs\nautort_swarm.run(\n    \"There is a bottle on the table.\",\n    \"https://i.imgur.com/2qY9f8U.png\",\n)\n```\n\n\n### AutoRTSwarm\nA team of AutoRT agents where you can plug in and play any number of `AutoRTAgents` with customization. First, the task will be routed to each agent and then all of their outputs will be saved.\n```python\n# Import necessary modules\nimport os\nfrom autort import AutoRTSwarm, AutoRTAgent\n\n# Set the OpenAI API key\nopenai_api_key = os.getenv(\"OPENAI_API_KEY\")\n\n# Define a list of AutoRTAgent instances\nagents = [\n    AutoRTAgent(openai_api_key, max_tokens=1000),\n    AutoRTAgent(openai_api_key, max_tokens=1000),\n]\n\n# Create an instance of AutoRTSwarm with the agents and datastore\nautort_swarm = AutoRTSwarm(agents)\n\n# Run the AutoRTSwarm with the given inputs\nautort_swarm.run(\n    \"There is a bottle on the table.\",\n    \"https://i.imgur.com/2qY9f8U.png\",\n)\n```\n\n## Citation\n```bibtex\n@inproceedings{\n    anonymous2023autort,\n    title={Auto{RT}: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents},\n    author={Anonymous},\n    booktitle={Submitted to The Twelfth International Conference on Learning Representations},\n    year={2023},\n    url={https://openreview.net/forum?id=xVlcbh0poD},\n    note={under review}\n}\n\n```\n\n\n# License\nMIT\n\n\n\n# Todo\n- [ ] Implement a run method into `AutoRTSwarm` that runs all the agents with APIs.\n- [ ] Make it able to send commands to a certain agent using the swarm network.\n- [ ] Send a task to all agents in the swarm network\n- [ ] ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2Fautort","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyegomez%2Fautort","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2Fautort/lists"}