Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pockerman/rlenvscpp
Environments and various utilities for reinforcement learning with C++
https://github.com/pockerman/rlenvscpp
cpp gymnasium-environment openai-gym reinforcement-learning reinforcement-learning-environments
Last synced: 3 days ago
JSON representation
Environments and various utilities for reinforcement learning with C++
- Host: GitHub
- URL: https://github.com/pockerman/rlenvscpp
- Owner: pockerman
- Created: 2021-07-21T12:15:33.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2024-12-31T21:20:31.000Z (8 days ago)
- Last Synced: 2024-12-31T21:27:16.489Z (8 days ago)
- Topics: cpp, gymnasium-environment, openai-gym, reinforcement-learning, reinforcement-learning-environments
- Language: C++
- Homepage:
- Size: 678 KB
- Stars: 12
- Watchers: 0
- Forks: 0
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
[](https://github.com/pockerman/rlenvs_from_cpp/actions/workflows/build.yml)
# rlenvscpp
```rlenvscpp``` is an effort to provide implementations and wrappers of environments suitable for training reinforcement learning agents
using C++. In addition, the library provides various utilities such as experiment tracking,
representing trajectories via waypoints and simple implementation of popular dynamics such as
quadrotor dynamics.
## Environments
Currently, ```rlenvscpp``` provides the following environments:
| Environment | Use REST | Example |
| :---------------- | :----------: | :----: |
| FrozenLake 4x4 map | Yes | example_1 |
| FrozenLake 8x8 map | Yes | TODO |
| Blackjack | Yes | example_1 |
| CliffWalking | Yes | example_1 |
| CartPole | Yes | TODO |
| MountainCar | Yes | TODO |
| Taxi | Yes | example_1 |
| Pendulum | Yes | example_6 |
| Acrobot | Yes | TODO |
| GymWalk | Yes | TODO |
| gym-pybullet-drones | TODO | TODO |
| GridWorld | No | example_5 |
| Connect2 | No | example_7 |
The Gymnasium (former OpenAI-Gym) environments utilise a REST API to communicate requests to/from the
environment and ```rlenvscpp```.
Some environments have a vector implementation meaning multiple instances of the same
environment. Currently, ```rlenvscpp``` provides the following vector environments:
| Environment | Use REST | Example |
| :---------------- | :----------: | :----: |
| AcrobotV | Yes | example_8 |
Various RL algorithms using the environments can be found at cuberl.
### How to use
The following is an example how to use the
```FrozenLake``` environment from Gymnasium.
```cpp
#include "rlenvs/rlenvs_types_v2.h"
#include "rlenvs/envs/gymnasium/toy_text/frozen_lake_env.h"
#include "rlenvs/envs/api_server/apiserver.h"
#include
#include
#include
#include
namespace example_1{
const std::string SERVER_URL = "http://0.0.0.0:8001/api";
using rlenvscpp::envs::gymnasium::FrozenLake;
using rlenvscpp::envs::RESTApiServerWrapper;
void test_frozen_lake(const RESTApiServerWrapper& server){
FrozenLake<4> env(server);
std::cout<<"Environame URL: "< options;
options.insert({"is_slippery", false});
env.make("v1", options);
std::cout<<"Is environment created? "<());
std::cout<<"Reward on reset: "<Environment API and Semantics specification.
For more details see the ```rlenvscpp``` environment specification document.
The general use case is to build the library and link it with your driver code to access its functionality.
The environments specified as using REST in the tables above, that is all ```Gymnasium```, ```gym_pybullet_drones``` and ```GymWalk```
environments are accessed via a client/server pattern. Namely, they are exposed via an API developed using
FastAPI.
You need to fire up the FastAPI server, see dependencies, before using the environments in your code.
To do so
```
./start_uvicorn.sh
```
By default the ```uvicorn``` server listents on port 8001. Change this if needed. You can access the OpenAPI specification at
```
http://0.0.0.0:8001/docs
```
Note that currently the implementation is not thread/process safe i.e. if multiple threads/processes access the environment
a global instance of the environment is manipulated. Thus no session based environment exists.
However, you can create copies of the same environment and access this via its dedicate index.
If just one thread/process touches this specific environment you should be ok.
Notice that the FastAPI server only uses a single process to manage all the environments.
In addition, if you need multiple instances of the same environment you can also use one
of the exissting vectorised environments (see table above).
Finally, you can choose to launch several instances of ```uvirocrn``` (listening on different ports).
However in this case you need to implement all the interactions logic yourself as currently no implementation exists to handle such a scenario.
## Dynamics
Apart from the exposed environments, ```rlenvscpp``` exposes classes that
describe the dynamics of some popular rigid bodies:
| Dynamics | Example |
| :---------------- | :----------------------------------------------------------: |
| Differential drive | example_9 |
| Quadrotor | example_10 |
| Bicycle vehicle | TODO |
## Miscellaneous
| Item | Example |
| :---------------- | :----------------------------------------------------------: |
| Environment trajectory | example_3 |
| WaypointTrajectory | example_11 |
| TensorboardServer | example_12 |
## Dependencies
The library has the following general dependencies
- A compiler that supports C++20 e.g. g++-11
- Boost C++
- CMake >= 3.10
- Gtest (if configured with tests)
- Eigen3
Using the Gymnasium environments requires Gymnasium installed on your machine.
In addition, you need to install
- FastAPI
- Uvicorn
- Pydantic
By installing the requirement under ```requirements.txt``` should set your Python environment up correctly.
In addition, the library also incorporates, see ```(src/extern)```, the following libraries
There are extra dependencies if you want to generate the documentation. Namely,
- Doxygen
- Sphinx
- sphinx_rtd_theme
- breathe
- m2r2
## Installation
The usual CMake based installation process is used. Namely
```
mkdir build && cd build && cmake ..
make install
```
You can toggle the following variables
- CMAKE_BUILD_TYPE (default is RELEASE)
- ENABLE_TESTS_FLAG (default is OFF)
- ENABLE_EXAMPLES_FLAG (default is OFF)
- ENABLE_DOC_FLAG (default is OFF)
For example enbling the examples
```
cmake -DENABLE_EXAMPLES_FLAG=ON ..
make install
```
### Run the tests
You can execute all the tests by running the helper script ```execute_tests.sh```.
### Issues
#### Could not find ```boost_system```
It is likely that you are missing the boost_system library with your local Boost installation. This may be the case
is you installed boost via a package manager. On a Ubuntu machine the following should resolve the issue
```
sudo apt-get update -y
sudo apt-get install -y libboost-system-dev
```
#### FastAPI throws 422 Unpocessable entity
Typically, this is a problem with how the client (400-range error) specified the data
to be sent to the server.