https://github.com/kyegomez/hlt

Implementation of the transformer from the paper: "Real-World Humanoid Locomotion with Reinforcement Learning"
https://github.com/kyegomez/hlt

ai artificial-intelligence attention attention-is-all-you-need ml multi-modal robotics transformers

Last synced: 4 days ago
JSON representation

Implementation of the transformer from the paper: "Real-World Humanoid Locomotion with Reinforcement Learning"

• Host: GitHub
• URL: https://github.com/kyegomez/hlt
• Owner: kyegomez
• License: mit
• Created: 2024-02-01T23:14:10.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2024-10-07T22:47:15.000Z (about 1 month ago)
• Last Synced: 2024-10-13T02:09:04.949Z (about 1 month ago)
• Topics: ai, artificial-intelligence, attention, attention-is-all-you-need, ml, multi-modal, robotics, transformers
• Language: Python
• Homepage: https://discord.gg/GYbXvDGevY
• Size: 2.17 MB
• Stars: 32
• Watchers: 4
• Forks: 5
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

README

        
[![Multi-Modality](agorabanner.png)](https://discord.gg/qUtxnK2NMf)

# Humanoid Locomotion Transformer

This is an implementation of the robotic transformer for humanoid robots from the premier paper from berkely: "Real-World Humanoid Locomotion with Reinforcement Learning". Here we implement the state policy model which is an MLP/FFN and a Transformer model that intakes both observation and action tokens to output the next action sequence.

## Install

`pip install hlt-torch`

## Usage

```python

import torch

from hlt_torch.model import HLT

# Import the necessary libraries

# Create an instance of the HLT model

model = HLT(

    num_classes=10,

    dim_conv_stem=64,

    dim=512,

    dim_head=64,

    depth=(4, 4, 4),

    window_size=8,

    mbconv_expansion_rate=4,

    mbconv_shrinkage_rate=2,

    dropout=0.1,

    num_actions=11,

    hl_depth=4,

    hl_heads=8,

    hl_dim_head=64,

    cond_drop_prob=0.2,

)

# Generate some dummy input tensors

video = torch.randn(

    1, 3, 16, 112, 112

)  # Shape: (batch_size, num_channels, num_frames, height, width)

instructions = torch.randn(

    1, 10, 512

)  # Shape: (batch_size, num_instructions, embedding_dim)

# Perform a forward pass through the model

output = model(video, instructions)

# Print the output tensor

print(output)

```

# License

MIT