{"id":13410346,"url":"https://github.com/ravenscroftj/turbopilot","last_synced_at":"2025-09-27T10:31:29.939Z","repository":{"id":152464700,"uuid":"625612711","full_name":"ravenscroftj/turbopilot","owner":"ravenscroftj","description":"Turbopilot is an open source large-language-model based code completion engine that runs locally on CPU","archived":true,"fork":false,"pushed_at":"2023-09-30T08:16:59.000Z","size":2667,"stargazers_count":3832,"open_issues_count":18,"forks_count":127,"subscribers_count":43,"default_branch":"main","last_synced_at":"2024-09-21T17:12:10.709Z","etag":null,"topics":["code-completion","cpp","language-model","machine-learning"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ravenscroftj.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-04-09T16:46:33.000Z","updated_at":"2024-09-21T11:59:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"64486068-d1b9-4d1e-a6ff-5988e12a5b02","html_url":"https://github.com/ravenscroftj/turbopilot","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ravenscroftj%2Fturbopilot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ravenscroftj%2Fturbopilot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ravenscroftj%2Fturbopilot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ravenscroftj%2Fturbopilot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ravenscroftj","download_url":"https://codeload.github.com/ravenscroftj/turbopilot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219871974,"owners_count":16554475,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-completion","cpp","language-model","machine-learning"],"created_at":"2024-07-30T20:01:06.351Z","updated_at":"2025-09-27T10:31:24.514Z","avatar_url":"https://github.com/ravenscroftj.png","language":"C++","funding_links":[],"categories":["Coding","Code","Tools for Self-Hosting","Python","C++","\u003ca name=\"cpp\"\u003e\u003c/a\u003eC++","Uncategorized","Artificial Intelligence","A01_文本生成_文本对话","Specialized Tools","🌟 编辑推荐","AI Tools","Repos","Projects","1. Local Agents","Language Models \u0026 Engines"],"sub_categories":["Coding Assistants","Developer tools","Development","Other text generators","Uncategorized","Android Launcher","大语言对话模型及数据","IDE Extensions","编程辅助工具","👨‍💻 Developer Tools","Coding Agents \u0026 IDE Extensions","Other IDEs"],"readme":"# TurboPilot 🚀\n\n## Turbopilot is deprecated/archived as of 30/9/23. There are other mature solutions that meet the community's needs better. Please read [my blog post](https://brainsteam.co.uk/posts/2023/09/30/turbopilot-obit/) about my decision to down tools and for recommended alternatives.\n\n\n-----------------------------------\n\n[![Mastodon Follow](https://img.shields.io/mastodon/follow/000117012?domain=https%3A%2F%2Ffosstodon.org%2F\u0026style=social)](https://fosstodon.org/@jamesravey) ![BSD Licensed](https://img.shields.io/github/license/ravenscroftj/turbopilot) ![Time Spent](https://img.shields.io/endpoint?url=https://wakapi.nopro.be/api/compat/shields/v1/jamesravey/all_time/label%3Aturbopilot)\n\n\nTurboPilot is a self-hosted [copilot](https://github.com/features/copilot) clone which uses the library behind [llama.cpp](https://github.com/ggerganov/llama.cpp) to run the [6 Billion Parameter Salesforce Codegen model](https://github.com/salesforce/CodeGen) in 4GiB of RAM. It is heavily based and inspired by on the [fauxpilot](https://github.com/fauxpilot/fauxpilot) project.\n\n***NB: This is a proof of concept right now rather than a stable tool. Autocompletion is quite slow in this version of the project. Feel free to play with it, but your mileage may vary.***\n\n![a screen recording of turbopilot running through fauxpilot plugin](assets/vscode-status.gif)\n\n**✨ Now Supports [StableCode 3B Instruct](https://huggingface.co/stabilityai/stablecode-instruct-alpha-3b)** simply use [TheBloke's Quantized GGML models](https://huggingface.co/TheBloke/stablecode-instruct-alpha-3b-GGML) and set `-m stablecode`.\n\n**✨ New: Refactored + Simplified**: The source code has been improved to make it easier to extend and add new models to Turbopilot. The system now supports multiple flavours of model\n\n**✨ New: Wizardcoder, Starcoder, Santacoder support** - Turbopilot now supports state of the art local code completion models which provide more programming languages and \"fill in the middle\" support.\n\n## 🤝 Contributing\n\nPRs to this project and the corresponding [GGML fork](https://github.com/ravenscroftj/ggml) are very welcome.\n\nMake a fork, make your changes and then open a [PR](https://github.com/ravenscroftj/turbopilot/pulls).\n\n\n## 👋 Getting Started\n\nThe easiest way to try the project out is to grab the pre-processed models and then run the server in docker.\n\n\n### Getting The Models\n\nYou have 2 options for getting the model\n\n#### Option A: Direct Download - Easy, Quickstart\n\nYou can download the pre-converted, pre-quantized models from Huggingface.\n\nFor low RAM users (4-8 GiB), I recommend [StableCode](https://huggingface.co/TheBloke/stablecode-instruct-alpha-3b-GGML) and for high power users (16+ GiB RAM, discrete GPU or apple silicon) I recomnmend [WizardCoder](https://huggingface.co/TheBloke/WizardCoder-15B-1.0-GGML/resolve/main/WizardCoder-15B-1.0.ggmlv3.q4_0.bin).\n\nTurbopilot still supports the first generation codegen models from `v0.0.5` and earlier builds. Although old models do need to be requantized.\n\nYou can find a full catalogue of models in [MODELS.md](MODELS.md).\n\n\n#### Option B: Convert The Models Yourself - Hard, More Flexible\n\nFollow [this guide](https://github.com/ravenscroftj/turbopilot/wiki/Converting-and-Quantizing-The-Models) if you want to experiment with quantizing the models yourself.\n\n### ⚙️ Running TurboPilot Server\n\nDownload the [latest binary](https://github.com/ravenscroftj/turbopilot/releases) and extract it to the root project folder. If a binary is not provided for your OS or you'd prefer to build it yourself follow the [build instructions](BUILD.md)\n\nRun:\n\n```bash\n./turbopilot -m starcoder -f ./models/santacoder-q4_0.bin\n```\n\nThe application should start a server on port `18080`, you can change this with the `-p` option but this is the default port that vscode-fauxpilot tries to connect to so you probably want to leave this alone unless you are sure you know what you're doing.\n\nIf you have a multi-core system you can control how many CPUs are used with the `-t` option - for example, on my AMD Ryzen 5000 which has 6 cores/12 threads I use:\n\n```bash\n./codegen-serve -t 6 -m starcoder -f ./models/santacoder-q4_0.bin\n```\n\nTo run the legacy codegen models. Just change the model type flag `-m` to `codegen` instead.\n\n**NOTE: Turbopilot 0.1.0 and newer re-quantize your codegen models old models from v0.0.5 and older. I am working on providing updated quantized codegen models**\n\n### 📦 Running From Docker\n\nYou can also run Turbopilot from the pre-built docker image supplied [here](https://github.com/users/ravenscroftj/packages/container/package/turbopilot)\n\nYou will still need to download the models separately, then you can run:\n\n```bash\ndocker run --rm -it \\\n  -v ./models:/models \\\n  -e THREADS=6 \\\n  -e MODEL_TYPE=starcoder \\\n  -e MODEL=\"/models/santacoder-q4_0.bin\" \\\n  -p 18080:18080 \\\n  ghcr.io/ravenscroftj/turbopilot:latest\n```\n\n#### Docker and CUDA\n\nAs of release v0.0.5 turbocode now supports CUDA inference. In order to run the cuda-enabled container you will need to have [nvidia-docker](https://github.com/NVIDIA/nvidia-docker) enabled, use the cuda tagged versions and pass in `--gpus=all` to docker with access to your GPU like so:\n\n```bash\ndocker run --gpus=all --rm -it \\\n  -v ./models:/models \\\n  -e THREADS=6 \\\n  -e MODEL_TYPE=starcoder \\\n  -e MODEL=\"/models/santacoder-q4_0.bin\" \\\n  -e GPU_LAYERS=32 \\\n  -p 18080:18080 \\\n  ghcr.io/ravenscroftj/turbopilot:v0.2.0-cuda11-7\n```\n\nIf you have a big enough GPU then setting `GPU_LAYERS` will allow turbopilot to fully offload computation onto your GPU rather than copying data backwards and forwards, dramatically speeding up inference. \n\nSwap `ghcr.io/ravenscroftj/turbopilot:v0.1.0-cuda11` for `ghcr.io/ravenscroftj/turbopilot:v0.2.0-cuda12-0` or `ghcr.io/ravenscroftj/turbopilot:v0.2.0-cuda12-2` if you are using CUDA 12.0 or 12.2 respectively.\n\nYou will need CUDA 11 or CUDA 12 later to run this container. You should be able to see `/app/turbopilot` listed when you run `nvidia-smi`.\n\n\n#### Executable and CUDA\n\nAs of v0.0.5 a CUDA version of the linux executable is available - it requires that libcublas 11 be installed on the machine - I might build ubuntu debs at some point but for now running in docker may be more convenient if you want to use a CUDA GPU.\n\nYou can use GPU offloading via the `--ngl` option.\n\n### 🌐 Using the API\n\n#### Support for the official Copilot Plugin\n\nSupport for the official VS Code copilot plugin is underway (See ticket #11). The API should now be broadly compatible with OpenAI.\n\n#### Using the API with FauxPilot Plugin\n\n\nTo use the API from VSCode, I recommend the vscode-fauxpilot plugin. Once you install it, you will need to change a few settings in your settings.json file.\n\n- Open settings (CTRL/CMD + SHIFT + P) and select `Preferences: Open User Settings (JSON)`\n- Add the following values:\n\n```json\n{\n    ... // other settings\n\n    \"fauxpilot.enabled\": true,\n    \"fauxpilot.server\": \"http://localhost:18080/v1/engines\",\n}\n```\n\nNow you can enable fauxpilot with `CTRL + SHIFT + P` and select `Enable Fauxpilot`\n\nThe plugin will send API calls to the running `codegen-serve` process when you make a keystroke. It will then wait for each request to complete before sending further requests.\n\n#### Calling the API Directly\n\nYou can make requests to `http://localhost:18080/v1/engines/codegen/completions` which will behave just like the same Copilot endpoint.\n\nFor example:\n\n```bash\ncurl --request POST \\\n  --url http://localhost:18080/v1/engines/codegen/completions \\\n  --header 'Content-Type: application/json' \\\n  --data '{\n \"model\": \"codegen\",\n \"prompt\": \"def main():\",\n \"max_tokens\": 100\n}'\n```\n\nShould get you something like this:\n\n```json\n{\n \"choices\": [\n  {\n   \"logprobs\": null,\n   \"index\": 0,\n   \"finish_reason\": \"length\",\n   \"text\": \"\\n  \\\"\\\"\\\"Main entry point for this script.\\\"\\\"\\\"\\n  logging.getLogger().setLevel(logging.INFO)\\n  logging.basicConfig(format=('%(levelname)s: %(message)s'))\\n\\n  parser = argparse.ArgumentParser(\\n      description=__doc__,\\n      formatter_class=argparse.RawDescriptionHelpFormatter,\\n      epilog=__doc__)\\n  \"\n  }\n ],\n \"created\": 1681113078,\n \"usage\": {\n  \"total_tokens\": 105,\n  \"prompt_tokens\": 3,\n  \"completion_tokens\": 102\n },\n \"object\": \"text_completion\",\n \"model\": \"codegen\",\n \"id\": \"01d7a11b-f87c-4261-8c03-8c78cbe4b067\"\n}\n```\n\n## 👉 Known Limitations\n\n- Currently Turbopilot only supports one GPU device at a time (it will not try to make use of multiple devices).\n\n## 👏 Acknowledgements\n\n- This project would not have been possible without [Georgi Gerganov's work on GGML and llama.cpp](https://github.com/ggerganov/ggml)\n- It was completely inspired by [fauxpilot](https://github.com/fauxpilot/fauxpilot) which I did experiment with for a little while but wanted to try to make the models work without a GPU\n- The frontend of the project is powered by [Venthe's vscode-fauxpilot plugin](https://github.com/Venthe/vscode-fauxpilot)\n- The project uses the [Salesforce Codegen](https://github.com/salesforce/CodeGen) models.\n- Thanks to [Moyix](https://huggingface.co/moyix) for his work on converting the Salesforce models to run in a GPT-J architecture. Not only does this [confer some speed benefits](https://gist.github.com/moyix/7896575befbe1b99162ccfec8d135566) but it also made it much easier for me to port the models to GGML using the [existing gpt-j example code](https://github.com/ggerganov/ggml/tree/master/examples/gpt-j)\n- The model server uses [CrowCPP](https://crowcpp.org/master/) to serve suggestions.\n- Check out the [original scientific paper](https://arxiv.org/pdf/2203.13474.pdf) for CodeGen for more info.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fravenscroftj%2Fturbopilot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fravenscroftj%2Fturbopilot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fravenscroftj%2Fturbopilot/lists"}