https://github.com/KTH-RPL/DriveCmd_LLM
[WAVC'24 Workshop] Human-Centric Autonomous Systems With LLMs for User Command Reasoning
https://github.com/KTH-RPL/DriveCmd_LLM
Last synced: 3 months ago
JSON representation
[WAVC'24 Workshop] Human-Centric Autonomous Systems With LLMs for User Command Reasoning
- Host: GitHub
- URL: https://github.com/KTH-RPL/DriveCmd_LLM
- Owner: KTH-RPL
- Created: 2023-10-16T10:11:43.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2024-07-10T11:06:59.000Z (11 months ago)
- Last Synced: 2024-08-01T05:14:36.499Z (10 months ago)
- Language: Python
- Homepage:
- Size: 538 KB
- Stars: 12
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- Awesome-LLM4AD - DriveCmd
README
Human-Centric Autonomous Systems With LLMs for User Command Reasoning
---
[](https://arxiv.org/abs/2311.08206)
Task: In-Cabin User Command Understanding (UCU), [workshop in WACV2024](https://llvm-ad.github.io/challenges/)
Here is our solution code. Please check the report for more detail.
## llama & codellama
Here we introduced how to setup and the way to downloaded their model:
1. Send request to their form [Apply here in Meta official page](https://ai.meta.com/llama/) and you will receive an email with some details.
2. Dependencies: `sudo apt install wget ucommon-utils`
3. run `./scripts/llama/download.sh` to download Llama models.
4. `mamba create --name llc python=3.8 && mamba activate llc && pip install -r requirements.txt`
5. Run the example:
```bash
torchrun --nproc_per_node 1 scripts/main_codellama.py
```
## GPT API
copy your OPENAI_API_KEY and save it in `.env`.
`OPENAI_API_KEY='xxxx'`
install below
```
pip install openai
pip install -U python-dotenv
```
Run the example:
```bash
python scripts/main_gpt.py --provide_few_shots True --step_by_step True
```
## Evaluation
You will have a result `.json` file finally. Then run the `eval.py` For each task-level accuracy:
```bash
python scripts/eval.py -g assets/ucu.csv -e assets/result/test.json
```
Here is the demo output:
```
Evaluating assets/result/gpt-4_best.json ...
| Task | Accuracy |
|---------------------+------------|
| Perception | 0.931756 |
| In-cabin monitoring | 0.748863 |
| Localization | 0.915378 |
| Vehicle control | 0.88626 |
| Entertainment | 0.944495 |
| Personal data | 0.859873 |
| Network access | 0.919927 |
| Traffic laws | 0.915378 |
| Overall | 0.890241 |
```
### LLVM_AD Official Leaderboard
Here is official evaluate with `-o`:
```bash
python3 scripts/llvm_ad/official_eval.py -o -g assets/ucu.csv -e assets/result/gpt4_best.csv
```
We attach the raw `.json` output files (with explainations and output), and its corresponding `.cvs` files (binary output only) under `assets/result` folder.
Here is demo output:
```
Since the input file is .json, we save the prediction to .csv file at assets/result/gpt-4_best.csv
Evaluating assets/result/gpt-4_best.json ...
Following is the evaluation result in official way:
Command-level acc: 0.38034576888080074
Question-level acc: 0.8902411282984531
```
## Acknowledgements
This work was funded by Vinnova, Sweden (research grant). The computations were enabled by the supercomputing resource Berzelius provided by National Supercomputer Centre at Linköping University and the Knut and Alice Wallenberg foundation, Sweden.
This implementation is based on codes from several repositories. Thanks for these authors who kindly open-sourcing their work to the community. Please see our paper reference part to get more information.
❤️: [llvm-ad](https://llvm-ad.github.io/), [llama2](https://github.com/facebookresearch/llama/tree/main)
### Cite Our Paper
```bash
@inproceedings{yang2024human,
title={Human-centric autonomous systems with llms for user command reasoning},
author={Yang, Yi and Zhang, Qingwen and Li, Ci and Marta, Daniel Sim{\~o}es and Batool, Nazre and Folkesson, John},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages={988--994},
year={2024}
}
```