Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/kurnevsky/llama-cpp.el
A client for llama-cpp server
https://github.com/kurnevsky/llama-cpp.el
ai emacs llama llm
Last synced: 4 months ago
JSON representation
A client for llama-cpp server
- Host: GitHub
- URL: https://github.com/kurnevsky/llama-cpp.el
- Owner: kurnevsky
- License: gpl-3.0
- Created: 2023-09-07T07:41:03.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2023-12-28T20:43:47.000Z (about 1 year ago)
- Last Synced: 2024-04-16T23:05:56.067Z (10 months ago)
- Topics: ai, emacs, llama, llm
- Language: Emacs Lisp
- Homepage:
- Size: 90.8 KB
- Stars: 18
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Llama-cpp in Emacs
*The readme is mostly written by CodeLLama.* :satisfied:
Llama package for Emacs provides a client for the
[llama-cpp](https://github.com/ggerganov/llama.cpp)
[server](https://github.com/ggerganov/llama.cpp/tree/master/examples/server).
It allows you to ask llama for code completion and perform tasks within
specified regions of the buffer.
## Table of Contents
1. [Installation](#installation)
2. [Usage](#usage)
3. [Customization](#customization)
4. [Models](#models)
5. [License](#license)
## Installation
To install this package, add it to your Emacs configuration:
```elisp
(use-package llama-cpp
:ensure t)
```
Make sure to have [llama-cpp](https://github.com/ggerganov/llama.cpp)
[server](https://github.com/ggerganov/llama.cpp/tree/master/examples/server)
running and accessible from the host specified in `llama-cpp-host` variable (default
is "localhost"). The server should be accessible via HTTP on the port specified
in `llama-cpp-port` (default is 8080).
## Usage
This package provides `llama-cpp-complete` as a public API for other packages to
use.
To cancel a running llama process, use `M-x llama-cpp-cancel`.
### Chat
You can start a Llama chat session using the command `M-x llama-cpp-chat-start`.
After providing the necessary input, you can execute `M-x llama-cpp-chat-answer`
to request a response from Llama.
To complete text from the llama buffer, use `M-x llama-cpp-chat-complete`.
### Code
`M-x llama-cpp-code-region-task` asks for a description of your task and then calls
the llama to provide an answer. The task context is taken from the region you select
when calling this function.
## Customization
The following customization options are available:
* `llama-cpp-host`: the host of the llama-cpp server (default is "localhost").
* `llama-cpp-port`: the port of the llama-cpp server (default is 8080).
* `llama-cpp-params`: parameters for the llama-cpp /completion request.
* `llama-cpp-chat-prompt`: sets the chat prompt to start the chat with.
* `llama-cpp-chat-input-prefix`: string to prefix user inputs with.
* `llama-cpp-chat-input-suffix`: string to suffix after user inputs with.
* `llama-cpp-code-lang-modes`: an alist mapping major modes to their language names.
* `llama-cpp-code-region-prompt`: a prompt template for code region tasks.
## Models
You will need to set `llama-cpp-chat-prompt-prefix`, `llama-cpp-chat-input-prefix`
and `llama-cpp-chat-input-suffix` according to the model you use. Some examples are
listed below.
### [dolphin-2.2.1-mistral-7B](https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF)
```elisp
(setq llama-cpp-chat-prompt-prefix "<|im_start|>system
"
llama-cpp-chat-input-prefix "<|im_end|>
<|im_start|>user
"
llama-cpp-chat-input-suffix "<|im_end|>
<|im_start|>assistant
")
```
### [Phind-CodeLlama-34B-v2](https://huggingface.co/TheBloke/Phind-CodeLlama-34B-v2-GGUF)
```elisp
(setq llama-cpp-chat-prompt-prefix "### System Prompt
"
llama-cpp-chat-input-prefix "
### User Message
"
llama-cpp-chat-input-suffix "
### Assistant
")
```
### [WizardLM-1.0-Uncensored-CodeLlama-34B](https://huggingface.co/TheBloke/WizardLM-1.0-Uncensored-CodeLlama-34B-GGUF)
```elisp
(setq llama-cpp-chat-prompt-prefix ""
llama-cpp-chat-input-prefix "
USER: "
llama-cpp-chat-input-suffix "
ASSISTANT: ")
```
### [Samantha-1.11-70B](https://huggingface.co/TheBloke/Samantha-1.11-70B-GGUF)
```elisp
(setq llama-cpp-chat-prompt ""
llama-cpp-chat-input-prefix "
USER: "
llama-cpp-chat-input-suffix "
ASSISTANT: ")
```
## License
This package is distributed under the GNU General Public License version 3 or
later. Please refer to the
[official repository](https://github.com/kurnevsky/llama-cpp.el) for more information.