{"id":13645195,"url":"https://github.com/minosvasilias/godot-dodo","last_synced_at":"2025-10-10T11:37:47.028Z","repository":{"id":154085939,"uuid":"631692814","full_name":"minosvasilias/godot-dodo","owner":"minosvasilias","description":"Finetuning large language models for GDScript generation.","archived":false,"fork":false,"pushed_at":"2023-05-26T21:09:02.000Z","size":8398,"stargazers_count":556,"open_issues_count":7,"forks_count":26,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-10-02T06:29:32.625Z","etag":null,"topics":["ai","finetuning","gdscript","godot","llama"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/minosvasilias.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-04-23T20:16:06.000Z","updated_at":"2025-09-25T02:53:40.000Z","dependencies_parsed_at":"2024-01-14T09:59:58.740Z","dependency_job_id":"5ca984b6-6657-4307-9f42-953fb537a92e","html_url":"https://github.com/minosvasilias/godot-dodo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/minosvasilias/godot-dodo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minosvasilias%2Fgodot-dodo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minosvasilias%2Fgodot-dodo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minosvasilias%2Fgodot-dodo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minosvasilias%2Fgodot-dodo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/minosvasilias","download_url":"https://codeload.github.com/minosvasilias/godot-dodo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minosvasilias%2Fgodot-dodo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278136222,"owners_count":25935915,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-03T02:00:06.070Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","finetuning","gdscript","godot","llama"],"created_at":"2024-08-02T01:02:31.054Z","updated_at":"2025-10-10T11:37:47.011Z","avatar_url":"https://github.com/minosvasilias.png","language":"Python","funding_links":[],"categories":["Langchain","Python"],"sub_categories":[],"readme":"# godot-dodo\n\n![Godot-Dodo logo imagined by Midjourney v5](godot_dodo_logo.png)\n\nThe godot-dodo project presents a pipeline to finetune open source language models on human-created, language-specific code retrieved from GitHub.\n\nIn this case, the targeted language is GDScript, but the same methodology can be applied to other languages.\n\nThis repository includes the following:\n\n- Scripts to assemble the finetuning dataset\n- Pre-assembled, raw datasets (up to a size of 60k rows)\n- Scripts to finetune a model\n- Links to model weights\n- Performance report comparing finetuned models\n\n## Performance\n\n![Results](models/results.png)\n\n**For comprehensive results explaining the methodology used and a full list of all result, please refer to the full performance report [here](models/).**\n\nIn summary, `godot_dodo` models achieve significantly greater consistency than `gpt-4`/`gpt-3.5-turbo` when it comes to generating accurate GDScript syntax, and variants trained on code-specific base-models can even outperform them on complex instructions.\n\nThe major remaining weakpoint of this approach is the loss in appropriate verbosity when writing methods. Since human-written samples will often include references to objects initialized outside of the scope of the sample method, the model learns to do the same, resulting in cases where functionality relevant to the instruction is assumed to be already implemented. This can most likely be improved significantly by a more sophisticated dataset.\n\n## Concept\n\n### How?\n\nUnlike other, similar approaches to finetuning models such as stanford-alpaca, this approach does not use existing, larger language models for the output-values of the finetuning-dataset. All code used is human-created. Language models are instead only used to **label** each code snippet.\n\nAs such, we can assemble `comment:code` data-pairs in the style of [CodeSearchNet](https://github.com/github/CodeSearchNet), making use of powerful existing models to annotate high-quality human-created code.\n\n### Why?\n\nSome existing language models such as `gpt-4` are excellent coders. However, a lot of their ability is concentrated in only the most popular languages, such as Python or Javascript.\n\nLess widely used languages are underrepresented in the training data and experience a massive performance drop-off, where models routinely mistake syntax or hallucinate language features that do not exist.\n\nThis aims to provide much more robust language-specific models that can be used to reliably generate code that compiles on first try.\n\n## Demo\n\nTo try out the pre-trained models, you can use the [inference_demo.ipynb](demo/inference_demo.ipynb) notebook.\n\nIn order to use that notebook on Google Colab, follow [this link](https://colab.research.google.com/github/minosvasilias/godot-dodo/blob/main/demo/inference_demo.ipynb).\n\n## Dataset Generation\n\nDue to this approach relying on human-created data, we scrape GitHub repositories using the GitHub search API.\n\nUsing the `language:gdscript` search term, we retrieve a list of repositories including GDScript code.\n\nWe also use `license:mit` to limit the dataset to suitable repositories. **Only MIT-licensed code is used for training!**\n\nWe then clone each one and apply the following logic:\n\n- Find `project.godot` file\n- Detect whether project is made for `3.x` or `4.x` Godot engine versions\n- Iterate through all `.gd` files found in the repository\n- For each one, split file into individual functions\n- For each function found, ask existing LLM (`gpt-3.5-turbo`) for a detailed comment describing the functions purpose\n- Add `instruction:response` data pair to dataset\n\nNote that existing, human-written comments located above the code-block are not used for the `instruction` value. We are interested in consistent detail for comments, rather than trying to preserve some potentially higher-quality human-written ones.\n\nHuman comments within the code block however are preserved.\n\n### Run\n\nTo assemble a dataset yourself, follow these instructions:\n\n- Run `python data/generate_unlabeled_dataset.py`\n- Run `python data/label_dataset.py`\n\nPlease do note that you'll need GitHub and OpenAI API keys in order to use these scripts.\n\n## Pre-assembled datasets\n\nPre-assembled datasets included in this repository:\n\n- [godot_dodo_4x_60k](data/godot_dodo_4x_60k/)\n  - Assembled using `4.x` Godot projects - ~60k rows\n\nFurther datasets may be added in the future (particularly regarding `3.x` data)\n\n## Finetuning\n\nThe fine-tuning process closely mirrors the one introduced by [stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).\n\nTo reproduce a fine-tuned version of LLaMA, please follow the steps below.\n\n### Hardware Requirements\n\nIn order to effectively finetune a `llama-7b` or `llama-13b` model, it is highly recommended to use at least two `A100 80GB` GPUs. You may otherwise encounter out of memory errors or experience extremely long training times, and will need to adjust the training parameters.\n\nFor finetuning `godot_dodo_4x_60k_llama_13b`, eight `A100 80GB` GPUs were used.\n\nAnother important consideration is the protocol used for GPU communication. It is recommended to use `NVLink` setups rather than `PCIe`.\n\nShould you only have access to `PCIe` setups, please replace `full-shard` with `shard_grad_op` in the `torchrun` command. This may severely speed up your training runs at the cost of potentially higher memory usage.\n\n### Setup\n\nBefore finetuning, make sure to install all requirements using:\n\n```bash\npip install -r requirements.txt\n```\n\n### Run\n\nFor exact commands used for finetuning models, please refer to the individual model pages:\n\n- [models/godot_dodo_4x_60k_llama_7b](models/godot_dodo_4x_60k_llama_7b)\n- [models/godot_dodo_4x_60k_llama_13b](models/godot_dodo_4x_60k_llama_13b)\n\n## Inference\n\nTo test out your finetuned model, you can use the `eval.py` script. Simply run:\n\n```bash\npython finetune/eval.py --model_name_or_path PATH_TO_FINETUNED_MODEL/\n```\n\n## Publishing to Huggingface\n\nTo easily upload a finetuned model to Huggingface, you can use:\n\n```bash\npython finetune/push_to_hub.py --model_name_or_path PATH_TO_FINETUNED_MODEL/ --push_name HF_MODEL_NAME --auth_token HF_ACCESS_TOKEN\n```\n\n## Finetuned model weights\n\nLinks to model weights hosted on Huggingface are provided in the respective model pages:\n\n- [models/godot_dodo_4x_60k_llama_7b](models/godot_dodo_4x_60k_llama_7b)\n- [models/godot_dodo_4x_60k_llama_13b](models/godot_dodo_4x_60k_llama_13b)\n\n## Cost\n\nBelow the dollar-cost of assembling each available dataset and finetuning each model.\n\n### Datasets\n\n- [godot_dodo_4x_60k](data/godot_dodo_4x_60k/)\n  - `30$` (`gpt-3.5-turbo` API costs)\n\n### Finetuned Models\n\n- [models/godot_dodo_4x_60k_llama_7b](models/godot_dodo_4x_60k_llama_7b)\n  - `24$` (8x A100 80GB instance costs)\n- [models/godot_dodo_4x_60k_llama_13b](models/godot_dodo_4x_60k_llama_13b)\n  - `84$`(8x A100 80GB instance costs)\n\n## Use with godot-copilot\n\nUsage of finetuned models with [godot-copilot](https://github.com/minosvasilias/godot-copilot) for in-editor, fully local code generation may be supported in the future.\n\n## Acknowledegments\n\nThank you to all MIT-licensed Godot projects! This would not be possible without you.\n\nAll projects that were scraped during assembly of the included finetuning data are listed in the respective dataset folders in [data](data).\n\nAnother thank you goes to [fluidstack.io](https://fluidstack.io) for their reliable, cheap GPU instances that were used for finetuning these models.\n\n## Citation\n\nIf you wish to cite this project, please use:\n\n```\n@misc{godot-dodo,\n  author = {Markus Sobkowski},\n  title = {Godot-Dodo: Finetuned language models for GDScript generation},\n  year = {2023},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/minosvasilias/godot-dodo}},\n}\n```\n\nYou should also cite the [original LLaMA paper](https://arxiv.org/abs/2302.13971) as well as [stanford-alpaca](https://github.com/tatsu-lab/stanford_alpaca).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminosvasilias%2Fgodot-dodo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fminosvasilias%2Fgodot-dodo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminosvasilias%2Fgodot-dodo/lists"}