{"id":17902713,"url":"https://github.com/dhbrojas/gopilot","last_synced_at":"2025-03-23T18:31:27.660Z","repository":{"id":161962083,"uuid":"629378072","full_name":"dhbrojas/gopilot","owner":"dhbrojas","description":"Gopilot 🤖 is a (tiny) Large Language Model, trained on Go code, on a research budget","archived":false,"fork":false,"pushed_at":"2023-06-17T12:24:10.000Z","size":5402,"stargazers_count":34,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-18T22:35:52.471Z","etag":null,"topics":["code-generation","deep-learning","golang","llm","natural-language-processing","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dhbrojas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-18T07:36:02.000Z","updated_at":"2025-02-19T13:54:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"ec49ed5e-d569-4cc0-9d4e-8ac481e8d32b","html_url":"https://github.com/dhbrojas/gopilot","commit_stats":null,"previous_names":["dhbrojas/gopilot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dhbrojas%2Fgopilot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dhbrojas%2Fgopilot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dhbrojas%2Fgopilot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dhbrojas%2Fgopilot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dhbrojas","download_url":"https://codeload.github.com/dhbrojas/gopilot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245149253,"owners_count":20568860,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-generation","deep-learning","golang","llm","natural-language-processing","transformers"],"created_at":"2024-10-28T16:36:46.635Z","updated_at":"2025-03-23T18:31:27.651Z","avatar_url":"https://github.com/dhbrojas.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🤖 Gopilot - AI-Assitant for Golang\n\n![Tsinghua University](https://img.shields.io/badge/Tsinghua-University-purple) ![290M Parameters](https://img.shields.io/badge/Parameters-290M-blue) ![Fine-tuned](https://img.shields.io/badge/Fine--tuned-Yes-success) ![Loss](https://img.shields.io/badge/Loss-1.4-red) ![Task: Code Completion](https://img.shields.io/badge/Task-Code%20Completion-yellow) ![Open-Source](https://img.shields.io/badge/Open-Source-blue)\n\n\nGoPilot is a **290M parameters** language model trained exclusively on **Go code** using a **small research budget** (~100$).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/demo.gif\" width=\"75%\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003ci\u003eDemo of the Gopilot \u003ca href=\"https://github.com/rojas-diego/gopilot-vscode-ext\"\u003eVSCode Code Extension\u003c/a\u003e\u003c/i\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cb\u003e\u003ca href=\"https://github.com/rojas-diego\"\u003e⭐️ Diego Rojas\u003c/a\u003e\u003c/b\u003e \u003cb\u003e\u003ca style=\"margin-left:16px\" href=\"https://github.com/LeowWB\"\u003e⭐️ Wenbin Leow\u003c/a\u003e\u003c/b\u003e \u003cb\u003e\u003ca style=\"margin-left:16px\" href=\"https://github.com/jy477274\"\u003e⭐️ Jayden Macdonald\u003c/a\u003e\u003c/b\u003e  \u003cb\u003e\u003ca style=\"margin-left:16px\" href=\"https://github.com/Stanislas0\"\u003e⭐️ Qinkai Zheng\u003c/a\u003e\u003c/b\u003e\u003c/p\u003e\n\n## Overview\n\nGopilot is a GPT-style Transformer model trained on **20B tokens** on a **single RTX4090 for less than a week** using the Go split of [The Stack Dedup v1.2](https://www.google.com/search?client=safari\u0026rls=en\u0026q=the+stack+dedup+v1.2\u0026ie=UTF-8\u0026oe=UTF-8) dataset. It comes in two flavours: a HuggingFace tokenizer based model and a model based on a custom Go tokenizer that we developed.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/hugging-face-loss-curve.png\" width=\"50%\"\u003e\n\u003c/p\u003e\n\nThe pre-training and fine-tuning weights are made available [here](https://gopilot.s3.ap-east-1.amazonaws.com).\n\n```\naws s3 ls s3://gopilot/checkpoints/ --region ap-east-1\n```\n\n## Installation\n\nYou need to have `conda` and `go` installed on your machine. You can install the necessary dependencies using `conda` and the provided `environment_cpu.yml` (choose `environment_cuda.yml` when running CUDA). Dependencies may not be up to date, hence, using the official [Docker image](https://hub.docker.com/r/rojasdiego/gopilot) is preferred.\n\nBuild the Go tokenizer binary:\n\n```bash\n# Linux, MacOS\ngo build -o tokenizer/libgotok.so -buildmode=c-shared ./tokenizer/libgotok.go\n# Windows\ngo build -o tokenizer/libgotok.dll -buildmode=c-shared ./tokenizer/libgotok.go\n```\n\n## Usage\n\nA CUDA Docker image is made available [here](https://hub.docker.com/r/rojasdiego/gopilot).\n\n### Pre-Training\n\nThe pre-training script trains the model for the specified token budget. Expects a pre-tokenized dataset.\n\n```bash\npython train.py \\\n        --model-cf model/config/gopilot-290M.yml \\\n        --tokenizer hugging-face \\\n        --tokenizer-cf tokenizer/config/hugging-face.json \\\n        --s3-dataset-prefix \u003cprefix-of-your-s3-dataset\u003e \\\n        --s3-bucket \u003cyour-s3-bucket\u003e \\\n        --gradient-accumulation-steps 64 \\\n        --optimizer sophiag \\\n        --batch-size 8 \\\n        --lr 0.0005 \\\n        --token-budget 10000000000 \\\n        --device cuda \\\n        --precision fp16 \\\n        --s3-checkpoints \\\n        --warmup 1000 \\\n        --neptune \\\n        --compile\n```\n\n### Fine-tuning\n\nYou can fine-tune Gopilot on any JSONL dataset composed of samples of the following form: `{\"sample\": \"package main\\nconst Value = 1...\"}`. We use a mix of pre-training samples, AI-generated samples to perform finetuning.\n\n```bash\npython finetune.py \\\n    --model-cf model/config/gopilot-290M.yml \\\n    --tokenizer-cf tokenizer/config/hugging-face.json \\\n    --tokenizer hugging-face \\\n    --in-model-weights checkpoints/hugging-face.pt \\\n    --out-model-weights checkpoints/hugging-face-ft.pt \\\n    --dataset-filepath all \\\n    --gradient-accumulation-steps 16 \\\n    --batch-size 8 \\\n    --dropout 0.1 \\\n    --weight-decay 0.1 \\\n    --lr 0.000025 \\\n    --num-epochs 10 \\\n    --precision fp16 \\\n    --neptune\n```\n\n### Evaluation\n\nThe evaluation script runs evaluation on the HumanEvalX benchmark. Our best model obtains **7.4% `pass@10`** and **77.1% compile@10**. Check out the `results` folder for more information.\n\n```bash\npython evaluate.py \\\n    --model-cf model/config/gopilot-290M.yml \\\n    --tokenizer-cf tokenizer/config/gopilot.json \\\n    --tokenizer gopilot \\\n    --model-weights /checkpoints/gopilot-ft.pt \\\n    --device cuda \\\n    --k 10 \\\n    --max-new-tokens 128 \\\n    --verbose\n```\n\n### Inference Server\n\nThe inference server is a simple HTTP server that hosts the model and exposes a `/complete` endpoint to submit samples to auto-complete. It's used by the VSCode extension to provide completions.\n\n```bash\npython inference_server.py \\\n    --model-cf model/config/gopilot-290M.yml \\\n    --tokenizer-cf tokenizer/config/gopilot.json \\\n    --tokenizer gopilot \\\n    --device mps \\\n    --checkpoint-path .cache/checkpoints/gopilot-ft.pt\n```\n\n### VSCode Extension\n\nCheck out the Gopilot VSCode extension [here](https://github.com/rojas-diego/gopilot-vscode-ext). Works with the inference server.\n\n## Acknowledgements \u0026 Notes\n\n- **Thank you to Qinkai Zheng** for providing guidance and the hardware resources.\n- We did not check for leakage when performing HumanEvalX evaluation. **Do not include these results in research**.\n- This project was made during the course of **Deep Learning (80240743-0) at Tsinghua University**.\n- While fun to play around with, we do not recommend using a model of this size for code completion in your editor. It's a school project!\n- Feel free to use the code, tweak the checkpoints, and all!\n\n## Future Work\n\n- [ ] Release the model weights on HuggingFace\n- [ ] Quantize the model weights for fast inference\n- [ ] Interactive online demo\n- [ ] Try on other languages such as Rust or C++\n- [ ] Experiment with different tokenization strategies\n- [ ] Train for longer on more data\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdhbrojas%2Fgopilot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdhbrojas%2Fgopilot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdhbrojas%2Fgopilot/lists"}