{"id":17601302,"url":"https://github.com/umuthopeyildirim/doom-mistral","last_synced_at":"2025-09-21T06:32:48.813Z","repository":{"id":229516313,"uuid":"776559515","full_name":"umuthopeyildirim/DOOM-Mistral","owner":"umuthopeyildirim","description":"Mistral7B playing DOOM","archived":false,"fork":false,"pushed_at":"2024-07-15T19:00:45.000Z","size":22078,"stargazers_count":123,"open_issues_count":1,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-01-06T09:08:24.576Z","etag":null,"topics":["doom","fireworksai","mistral"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/umuthopeyildirim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-23T20:54:35.000Z","updated_at":"2024-12-31T16:56:30.000Z","dependencies_parsed_at":"2024-11-12T16:38:24.956Z","dependency_job_id":null,"html_url":"https://github.com/umuthopeyildirim/DOOM-Mistral","commit_stats":null,"previous_names":["umuthopeyildirim/gamecopilot","umuthopeyildirim/doom-mistral"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umuthopeyildirim%2FDOOM-Mistral","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umuthopeyildirim%2FDOOM-Mistral/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umuthopeyildirim%2FDOOM-Mistral/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umuthopeyildirim%2FDOOM-Mistral/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/umuthopeyildirim","download_url":"https://codeload.github.com/umuthopeyildirim/DOOM-Mistral/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233723966,"owners_count":18720110,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["doom","fireworksai","mistral"],"created_at":"2024-10-22T12:17:26.312Z","updated_at":"2025-09-21T06:32:40.471Z","avatar_url":"https://github.com/umuthopeyildirim.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DOOM Mistral\n\u003cp align=\"center\"\u003e\n\u003cimg src = https://github.com/umuthopeyildirim/DOOM-Mistral/assets/90144938/0259a48f-c51d-4223-9ffe-f626dfcb2c73) width=\"18%\" height=\"9%\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src = https://github.com/umuthopeyildirim/DOOM-Mistral/assets/90144938/e82098e9-2401-48d9-b99d-1e78f2cc8ba7)\u003e\n\u003c/p\u003e\n\n\nThis repository is home to the DOOM Mistral project, crafted during the CV [Mistral AI Hackathon](https://cerebralvalley.notion.site/Mistral-AI-Hackathon-Event-Details-Hackers-ee95c2545eda4ce1ae82bd5910a4a3ae) in San Francisco, on March 23-24 2024. Our team (Bhav, Umut, and Paul) developed a deep learning model capable of playing DOOM through visual input alone, utilizing the [ViZDoom](https://vizdoom.farama.org/#) engine, a prominent tool in visual reinforcement learning. We transformed each game frame into a 32x64 grid, representing game elements in each cell, enabling the LLM to interpret the game via ASCII text. For a glimpse into how the model views the game, refer to this video (note: contains strobing effects):\n\nhttps://twitter.com/i/status/1772075251297550457\n\nWe generated training data by playing DOOM, then fine-tuned a LoRA model on `Mistral-7B`, achieving reasonable results. While not an expert, the model can navigate the map and engage enemies when they appear.\n\nDuring the inference-loop, the model takes in a prompt and a grid representing the current game screen and produces actions which are then asyncronously fed as input into the game.\n\n![DOOM Mistral Visualization](./docs/arch.jpeg)\n\nWatch the model in action here:\n\nhttps://twitter.com/i/status/1772166935410532709\n\nYou can find our Weights \u0026 Biases dashboard [here](https://wandb.ai/anubhavashok93/doom-mistral).\n\n## Setting Up DOOM Mistral\n\nTo initialize, create a virtual environment and install the ViZDoom dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\nRename `.env.example` to `.env` and input your `FIREWORKS_API_KEY`.\n\nTo start the game, execute:\n\n```bash\npython llm_game.py\n```\n\n## Generating Training Data\n\nCapture gameplay for training with:\n`python user_game.py`\n\nAccess the training dataset at [HuggingFace 🤗](https://huggingface.co/datasets/CV-Mistral-Hackathon/doom-mistral-final), download it, and place it in the root directory.\n\nUpload your dataset to Fireworks.ai for model training:\n\n```bash\nfirectl create dataset doom-mistral doom_mistral.jsonl\n```\n\n## Model Training and Usage\n\nTraining settings are listed in `train_settings.yaml`. To train, use:\n\n```bash\nfirectl create fine-tuning-job --settings-file train_settings.yaml --display-name \"DOOM-Mistral\"\n```\n\nAccess the trained model on fireworks.ai, named `doom-mistral`.\n\nTo play the game with any LLM, modify `model_id` in line 16, then run:\n`python llm_game.py`\n\n### Understanding the Grid\n\n- E: Enemy\n- P: Player\n- B: Bullet\n- W: Wall\n- F: Floor\n- A: ArmorBonus\n- Z: Zombieman\n- H: HealthBonus\n- S: Stimpack\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=umuthopeyildirim/DOOM-Mistral\u0026type=Date)](https://star-history.com/#umuthopeyildirim/DOOM-Mistral\u0026Date)\n\n## Licensing\n\nViZDoom's original code is under the MIT license. ZDoom incorporates various sources, each with [distinct licensing](http://zdoom.org/wiki/license).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumuthopeyildirim%2Fdoom-mistral","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fumuthopeyildirim%2Fdoom-mistral","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumuthopeyildirim%2Fdoom-mistral/lists"}