https://github.com/hu-po/o

⚙️ Zero-Shot Autonomous Robots
https://github.com/hu-po/o

gpt-4 replicate-api robotics

Last synced: 7 months ago
JSON representation

⚙️ Zero-Shot Autonomous Robots

• Host: GitHub
• URL: https://github.com/hu-po/o
• Owner: hu-po
• License: mit
• Created: 2023-11-20T19:21:10.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-03-29T14:18:49.000Z (almost 2 years ago)
• Last Synced: 2024-03-29T15:30:10.535Z (almost 2 years ago)
• Topics: gpt-4, replicate-api, robotics
• Language: Python
• Homepage:
• Size: 1.95 MB
• Stars: 69
• Watchers: 3
• Forks: 11
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Contributing: docs/contributing.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# o

**o** stands for Zer**o**-Sh**o**t Aut**o**n**o**m**o**us R**o**b**o**ts.

![o](docs/cover.jpeg)

This repo uses model APIs to create a Zero-Shot Autonomous Robot. Individual robot behaviors are wrapped in asynchronous nodes (python) which are launched via scripts (bash). It's kind of like a more minimalist and simpler ROS. Four main types of models are used:

- **LLM** (Language Language Model) a `text2text` model used for planning, reasoning, dialogue, and more!

- **VLM** (Vision Language Model) - a `image2text` model used for scene understanding, object detection, and more!

- **TTS** (Text-to-Speech) a `text2audio` model used for speech synthesis so the robot can talk.

- **STT** (Speech-to-Text) a `audio2text` model used for speech recognition so the robot can listen.

To get started [follow the setup guide](docs/setup.md).

The `models` module contains code for different model apis. For example `models/rep.py` is for the open source Replicate API, and `models/gpt.py` is for the OpenAI API. [More info on models](docs/models.md).

The `robots` module contains code for different robots. For example `robots/nex.py` is for the HiWonder AiNex Humanoid. [More info on robots](docs/robots.md).

The `nodes` module contains code for different nodes. For example `nodes/look.py` contains the loop used vision with a Vision Language Model. [More info on nodes](docs/nodes.md).

The `params` module contains code for different parameters. For example `params/default.sh` will load environment variables (params) that contain default values. [More info on params](docs/params.md).

If you are interested in contributing, please read the [contributing guide](docs/contributing.md).

## Video

[![YouTube Video](https://img.youtube.com/vi/bN9_ml4f05M/0.jpg)](https://www.youtube.com/watch?v=bN9_ml4f05M)

## Citation

```

@misc{zero-shot-robot-2023,

  title={Zero-Shot Autonomous Robots},

  author={Hugo Ponte},

  year={2023},

  url={https://github.com/hu-po/o}

}

```