https://github.com/umuthopeyildirim/doom-mistral
Mistral7B playing DOOM
https://github.com/umuthopeyildirim/doom-mistral
doom fireworksai mistral
Last synced: 10 months ago
JSON representation
Mistral7B playing DOOM
- Host: GitHub
- URL: https://github.com/umuthopeyildirim/doom-mistral
- Owner: umuthopeyildirim
- Created: 2024-03-23T20:54:35.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-07-15T19:00:45.000Z (about 2 years ago)
- Last Synced: 2025-01-06T09:08:24.576Z (over 1 year ago)
- Topics: doom, fireworksai, mistral
- Language: C++
- Homepage:
- Size: 21.1 MB
- Stars: 123
- Watchers: 5
- Forks: 7
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# DOOM Mistral
)
)
This repository is home to the DOOM Mistral project, crafted during the CV [Mistral AI Hackathon](https://cerebralvalley.notion.site/Mistral-AI-Hackathon-Event-Details-Hackers-ee95c2545eda4ce1ae82bd5910a4a3ae) in San Francisco, on March 23-24 2024. Our team (Bhav, Umut, and Paul) developed a deep learning model capable of playing DOOM through visual input alone, utilizing the [ViZDoom](https://vizdoom.farama.org/#) engine, a prominent tool in visual reinforcement learning. We transformed each game frame into a 32x64 grid, representing game elements in each cell, enabling the LLM to interpret the game via ASCII text. For a glimpse into how the model views the game, refer to this video (note: contains strobing effects):
https://twitter.com/i/status/1772075251297550457
We generated training data by playing DOOM, then fine-tuned a LoRA model on `Mistral-7B`, achieving reasonable results. While not an expert, the model can navigate the map and engage enemies when they appear.
During the inference-loop, the model takes in a prompt and a grid representing the current game screen and produces actions which are then asyncronously fed as input into the game.
Watch the model in action here:
https://twitter.com/i/status/1772166935410532709
You can find our Weights & Biases dashboard [here](https://wandb.ai/anubhavashok93/doom-mistral).
## Setting Up DOOM Mistral
To initialize, create a virtual environment and install the ViZDoom dependencies:
```bash
pip install -r requirements.txt
```
Rename `.env.example` to `.env` and input your `FIREWORKS_API_KEY`.
To start the game, execute:
```bash
python llm_game.py
```
## Generating Training Data
Capture gameplay for training with:
`python user_game.py`
Access the training dataset at [HuggingFace 🤗](https://huggingface.co/datasets/CV-Mistral-Hackathon/doom-mistral-final), download it, and place it in the root directory.
Upload your dataset to Fireworks.ai for model training:
```bash
firectl create dataset doom-mistral doom_mistral.jsonl
```
## Model Training and Usage
Training settings are listed in `train_settings.yaml`. To train, use:
```bash
firectl create fine-tuning-job --settings-file train_settings.yaml --display-name "DOOM-Mistral"
```
Access the trained model on fireworks.ai, named `doom-mistral`.
To play the game with any LLM, modify `model_id` in line 16, then run:
`python llm_game.py`
### Understanding the Grid
- E: Enemy
- P: Player
- B: Bullet
- W: Wall
- F: Floor
- A: ArmorBonus
- Z: Zombieman
- H: HealthBonus
- S: Stimpack
## Star History
[](https://star-history.com/#umuthopeyildirim/DOOM-Mistral&Date)
## Licensing
ViZDoom's original code is under the MIT license. ZDoom incorporates various sources, each with [distinct licensing](http://zdoom.org/wiki/license).