https://github.com/achronus/velora
A Liquid Reinforcement Learning (RL) framework
https://github.com/achronus/velora
gymnasium liquid-neural-networks model-free mujoco oop python pytorch reinforcement-learning
Last synced: 6 months ago
JSON representation
A Liquid Reinforcement Learning (RL) framework
- Host: GitHub
- URL: https://github.com/achronus/velora
- Owner: Achronus
- License: mit
- Created: 2024-11-04T15:01:10.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2025-04-05T14:41:25.000Z (6 months ago)
- Last Synced: 2025-04-05T15:25:23.924Z (6 months ago)
- Topics: gymnasium, liquid-neural-networks, model-free, mujoco, oop, python, pytorch, reinforcement-learning
- Language: Python
- Homepage: https://velora.achronus.dev/
- Size: 2.72 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://codecov.io/gh/Achronus/velora)
Found on:
- [PyPi](https://pypi.org/project/velora)
- [GitHub](https://github.com/Achronus/velora)
# Velora
**Velora** is a lightweight and extensible framework built on top of powerful libraries like [Gymnasium](https://gymnasium.farama.org/) and [PyTorch](https://pytorch.org/), specializing in a unique approach to Deep Reinforcement Learning (RL) algorithms, a paradigm we call *Liquid RL*.
Instead of Fully-connected Networks, Velora combines [Liquid Neural Networks](https://arxiv.org/abs/2006.04439) (LNNs) with [Neural Circuit Policies](https://arxiv.org/abs/1803.08554) (NCPs), specifically [Ordinary Neural Circuits](https://proceedings.mlr.press/v119/hasani20a.html) (ONCs).
These two components have interesting benefits:
- LNNs are a powerful RNN architecture that learns system dynamics, not just data patterns.
- NCPs focus on sparsely connected neurons with distinct functions, mimicking biological behaviour.
From what we've seen, these networks are powerful, small-scale architectures that excel in model explainability, making them perfect for control tasks.
Velora offers Liquidfied PyTorch-based implementations of RL algorithms, designed to be intuitive, easy to use, and customizable.
In other frameworks, we've seen a trend of heavy abstraction in favour of minimal lines of code. Our approach aims to offer a best of both worlds, abstracting code away but making the details explainable on the backend, while giving you the freedom to customize as needed.
## Installation
To get started, simply install it through [pip](https://pypi.org/) using one of the options below.
### GPU Enabled
For [PyTorch](https://pytorch.org/get-started/locally/) with CUDA (recommended):
```bash
pip install torch torchvision velora --extra-index-url https://download.pytorch.org/whl/cu124
```
### CPU Only
Or, for [PyTorch](https://pytorch.org/get-started/locally/) with CPU only:
```bash
pip install torch torchvision velora
```
## Example Usage
Here's a simple example that should work 'as is':
```python
from functools import partial
from velora.models import LiquidDDPG
from velora.gym import wrap_gym_env
from velora.utils import set_device, set_seed
import gymnasium as gym
from gymnasium.wrappers import NormalizeObservation, NormalizeReward, ClipReward
# Setup reproducibility and PyTorch device
seed = 64
set_seed(seed)
device = set_device()
# Add extra wrappers to our environment
env = wrap_gym_env("InvertedPendulum-v5", [
partial(NormalizeObservation, epsilon=1e-8),
partial(NormalizeReward, gamma=0.99, epsilon=1e-8),
partial(ClipReward, max_reward=10.0),
# RecordEpisodeStatistics, # Applied automatically!
# partial(NumpyToTorch, device=device), # Applied automatically!
])
# Or, use the standard gym API (recommended for this env)
env = gym.make("InvertedPendulum-v5")
# Set core variables
state_dim = env.observation_space.shape[0] # in features
n_neurons = 20 # decision/hidden nodes
action_dim = env.action_space.shape[0] # out features
buffer_size = 100_000
batch_size = 128
# Train a model
model = LiquidDDPG(
state_dim,
n_neurons,
action_dim,
buffer_size=buffer_size,
device=device,
)
model.train(env, batch_size, n_episodes=300)
```
Currently, the framework only supports [Gymnasium](https://gymnasium.farama.org/) environments and is planned to expand to [PettingZoo](https://pettingzoo.farama.org/index.html) for Multi-agent (MARL) tasks.
## API Structure
The frameworks API is designed to be simple and intuitive. We've broken into two main categories: `core` and `extras`.
### Core
The primary building blocks you'll use regularly.
```python
from velora.models import [algorithm]
from velora.callbacks import [callback]
```
### Extras
Utility methods that you may use occasionally.
```python
from velora.gym import [method]
from velora.utils import [method]
```
## Customization
Customization is at the heart of Velora but requires a deeper understanding of the API.
You can read more about it in the [documentation tutorials](https://velora.achronus.dev/learn/customize).
## Active Development
🚧 View the [Roadmap](https://velora.achronus.dev/starting/roadmap) 🚧
**Velora** is a tool that is continuously being developed. There's still a lot to do to make it a fully functioning framework, such as detailed API documentation, and more RL algorithms.
Our goal is to provide a quality open-source product that works 'out-of-the-box' that everyone can experiment with, and then gradually fix unexpected bugs and introduce more features on the road to a `v1` release.