https://github.com/trust-ai/SafeBench
A Benchmark for Evaluating Autonomous Vehicles in Safety-critical Scenarios
https://github.com/trust-ai/SafeBench
Last synced: 2 months ago
JSON representation
A Benchmark for Evaluating Autonomous Vehicles in Safety-critical Scenarios
- Host: GitHub
- URL: https://github.com/trust-ai/SafeBench
- Owner: trust-ai
- License: mit
- Created: 2022-06-16T15:48:45.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-02-23T18:45:57.000Z (about 1 year ago)
- Last Synced: 2024-05-22T00:05:58.385Z (12 months ago)
- Language: Python
- Homepage: https://safebench.github.io
- Size: 81.7 MB
- Stars: 84
- Watchers: 1
- Forks: 20
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-safety-critical-ai - `trust-ai/SafeBench` - critical scenarios (<a id="tools"></a>🛠️ Tools / Model Evaluation)
- awesome-safety-critical-ai - `trust-ai/SafeBench` - critical scenarios (<a id="tools"></a>🛠️ Tools / Model Evaluation)
README
SafeBench: A Benchmark for Evaluating Autonomous Vehicles in Safety-critical Scenarios
[](https://safebench.readthedocs.io)
[](https://safebench.github.io)
[](https://arxiv.org/pdf/2206.09682.pdf)
[](#License)
| Perception Evaluation | Control Evaluation |
| :-------------------: | :----------------: |
|  |  |
## Installation
**Recommended system: Ubuntu 20.04 or 22.04**
### 1. Local Installation
Click to expand
Step 1: Setup conda environment
```bash
conda create -n safebench python=3.8
conda activate safebench
```
Step 2: Clone this git repo in an appropriate folder
```bash
git clone [email protected]:trust-ai/SafeBench.git
```
Step 3: Enter the repo root folder and install the packages:
```bash
cd SafeBench
pip install -r requirements.txt
pip install -e .
```
Step 4: Download our [CARLA_0.9.13](https://drive.google.com/file/d/139vLRgXP90Zk6Q_du9cRdOLx7GJIw_0v/view?usp=sharing) and extract it to your folder.
Step 5: Run `sudo apt install libomp5` as per this [git issue](https://github.com/carla-simulator/carla/issues/4498).
Step 6: Add the python API of CARLA to the ```PYTHONPATH``` environment variable. You can add the following commands to your `~/.bashrc`:
```bash
export CARLA_ROOT={path/to/your/carla}
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla/dist/carla-0.9.13-py3.8-linux-x86_64.egg
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla/agents
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI
```
### 2. Docker Installation (Beta)
Click to expand
We also provide a docker image with CARLA and SafeBench installed. Use the following command to launch a docker container:
```bash
bash docker/run_docker.sh
```
The CARLA simulator is installed at `/home/safebench/carla` and SafeBench is installed at `/home/safebench/SafeBench`.
## Usage
### 1. Desktop Users
Click to expand
Enter the CARLA root folder, launch the CARLA server and run our platform with
```bash
# Launch CARLA
./CarlaUE4.sh -prefernvidia -windowed -carla-port=2000
# Launch SafeBench in another terminal
python scripts/run.py --agent_cfg basic.yaml --scenario_cfg standard.yaml --mode eval
```
### 2. Remote Server Users
Click to expand
Enter the CARLA root folder, launch the CARLA server with headless mode, and run our platform with
```bash
# Launch CARLA
./CarlaUE4.sh -prefernvidia -RenderOffScreen -carla-port=2000
# Launch SafeBench in another terminal
SDL_VIDEODRIVER="dummy" python scripts/run.py --agent_cfg basic.yaml --scenario_cfg standard.yaml --mode eval
```
(Optional) You can also visualize the pygame window using [TurboVNC](https://sourceforge.net/projects/turbovnc/files/).
First, launch CARLA with headless mode, and run our platform on a virtual display.
```bash
# Launch CARLA
./CarlaUE4.sh -prefernvidia -RenderOffScreen -carla-port=2000
# Run a remote VNC-Xserver. This will create a virtual display "8".
/opt/TurboVNC/bin/vncserver :8 -noxstartup
# Launch SafeBench on the virtual display
DISPLAY=:8 python scripts/run.py --agent_cfg basic.yaml --scenario_cfg standard.yaml --mode eval
```
You can use the TurboVNC client on your local machine to connect to the virtual display.
```bash
# Use the built-in SSH client of TurboVNC Viewer
/opt/TurboVNC/bin/vncviewer -via user@host localhost:n
# Or you can manually forward connections to the remote server by
ssh -L fp:localhost:5900+n user@host
# Open another terminal on local machine
/opt/TurboVNC/bin/vncviewer localhost::fp
```
where `user@host` is your remote server, `fp` is a free TCP port on the local machine, and `n` is the display port specified when you started the VNC server on the remote server ("8" in our example).
### 3. Visualization with CarlaViz
Click to expand
CarlaViz is a convenient visualization tool for CARLA developed by a former member [mjxu96](https://github.com/mjxu96) of our team. To use CarlaViz, please open another terminal and follow the intructions:
```bash
# pull docker image from docker hub
docker pull mjxu96/carlaviz:0.9.13
# run docker container of CarlaViz
cd Safebench/scripts
sh start_carlaviz.sh
```
Then, you can open the CarlaViz window at http://localhost:8080. You can also remotely access the CarlaViz window by forwarding the port 8080 to your local machine.
### 4. Scenic users
Click to expand
If you want to use scenic to control the surrounding adversarial agents, and use RL to control the ego, then first install scenic as follows:
```bash
# Download Scenic repository
git clone https://github.com/BerkeleyLearnVerify/Scenic.git
cd Scenic
python -m pip install -e .
```
Then you can create a directory in ```safebench/scenario/scenario_data/scenic_data```, e.g., ```Carla_Challenge```, and put your scenic files in that directory (the relative map path defined in scenic file should be ```../maps/*.xodr```).
Next, set the param ```scenic_dir``` in ```safebench/scenario/config/scenic.yaml``` with the directory where you store the scenic files, e.g., ```safebench/scenario/scenario_data/scenic_data/Carla_Challenge```, and our code will automatically load all scenic files in that directory.
For selecting the most adversarial scenes, the param ```sample_num``` within the ```scenic.yaml``` serves to determine the number of scenes sampled for each scenic file and the param ```select_num``` is used to specify the number of the most adversarial scenes to be selected from among the sample_num scenes:
```bash
python scripts/run.py --agent_cfg sac.yaml --scenario_cfg scenic.yaml --num_scenario 1 --mode train_scenario
```
Now you can test the ego with these selected adversarial scenes:
```bash
python scripts/run.py --agent_cfg sac.yaml --scenario_cfg scenic.yaml --num_scenario 1 --mode eval
```
Or if you want to Launch it on the virtual display:
```bash
DISPLAY=:8 python scripts/run.py --agent_cfg sac.yaml --scenario_cfg scenic.yaml --num_scenario 1 --mode train_scenario
DISPLAY=:8 python scripts/run.py --agent_cfg sac.yaml --scenario_cfg scenic.yaml --num_scenario 1 --mode eval
```
## Running Arguments
| Argument | Choice | Usage |
| :----: | :----: | :---- |
| `mode` | `{train_agent, train_scenario, eval}` | We provide three modes for training agent, training scenario, and evaluation. |
| `agent_cfg` | str | path to the configuration file of agent. |
| `scenario_cfg` | str | path to the configuration file of scenario. |
| `max_episode_step` | int | Number of episode used for training agents and scenario. |
| `num_scenario` | `{1, 2, 3, 4}` | We support running multiple scenarios in parallel. Current map allows at most 4 scenarios. |
| `save_video` | store_true | We support saving videos during the evaluation mode. |
| `auto_ego` | store_true | Overwrite the action of ego agent with auto-polit |
| `port` | int | Port used by Carla, default 2000 |