https://github.com/xNul/code-llama-for-vscode

Use Code Llama with Visual Studio Code and the Continue extension. A local LLM alternative to GitHub Copilot.
https://github.com/xNul/code-llama-for-vscode

assistant code code-llama codellama continue continuedev copilot llama llama2 llamacpp llm local meta ollama studio visual vscode

Last synced: 27 days ago
JSON representation

Use Code Llama with Visual Studio Code and the Continue extension. A local LLM alternative to GitHub Copilot.

• Host: GitHub
• URL: https://github.com/xNul/code-llama-for-vscode
• Owner: xNul
• License: mit
• Created: 2023-08-25T03:48:12.000Z (11 months ago)
• Default Branch: main
• Last Pushed: 2023-08-27T07:18:52.000Z (10 months ago)
• Last Synced: 2024-02-24T13:34:17.497Z (4 months ago)
• Topics: assistant, code, code-llama, codellama, continue, continuedev, copilot, llama, llama2, llamacpp, llm, local, meta, ollama, studio, visual, vscode
• Language: Python
• Homepage:
• Size: 6.84 KB
• Stars: 463
• Watchers: 6
• Forks: 24
• Open Issues: 8
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Lists

• awesome-stars - xNul/code-llama-for-vscode - Use Code Llama with Visual Studio Code and the Continue extension. A local LLM alternative to GitHub Copilot. (Python)
• awesome-stars - xNul/code-llama-for-vscode - Use Code Llama with Visual Studio Code and the Continue extension. A local LLM alternative to GitHub Copilot. (vscode)

README

        
# Code Llama for VSCode

An API which mocks [llama.cpp](https://github.com/ggerganov/llama.cpp) to enable support for Code Llama with the

[Continue Visual Studio Code extension](https://continue.dev/).

As of the time of writing and to my knowledge, this is the only way to use Code Llama with VSCode locally without having

to sign up or get an API key for a service. The only exception to this is Continue with [Ollama](https://ollama.ai/), but

Ollama doesn't support Windows or Linux. On the other hand, Code Llama for VSCode is completely cross-platform and will

run wherever Meta's own [codellama](https://github.com/facebookresearch/codellama) code will run.

Now let's get started!

### Setup

Prerequisites:

- [Download and run one of the Code Llama Instruct models](https://github.com/facebookresearch/codellama)

- [Install the Continue VSCode extension](https://marketplace.visualstudio.com/items?itemName=Continue.continue)

After you are able to use both independently, we will glue them together with Code Llama for VSCode.

Steps:

1. Move `llamacpp_mock_api.py` to your [`codellama`](https://github.com/facebookresearch/codellama) folder and install Flask to your environment with `pip install flask`.

2. Run `llamacpp_mock_api.py` with your [Code Llama Instruct torchrun command](https://github.com/facebookresearch/codellama#fine-tuned-instruction-models). For example:

```

torchrun --nproc_per_node 1 llamacpp_mock_api.py \

    --ckpt_dir CodeLlama-7b-Instruct/ \

    --tokenizer_path CodeLlama-7b-Instruct/tokenizer.model \

    --max_seq_len 512 --max_batch_size 4

```

3. Type `/config` in VSCode with Continue and make changes to `config.py` so it looks like [this](https://continue.dev/docs/customization#local-models-with-ggml).

Restart VSCode or reload the Continue extension and you should now be able to use Code Llama for VSCode!

**TODO: Response streaming**