Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/linonetwo/langchain-alpaca
Run Alpaca LLM in LangChain
https://github.com/linonetwo/langchain-alpaca
alpaca chatgpt llama llm local-first offline-first self-hosted
Last synced: 3 months ago
JSON representation
Run Alpaca LLM in LangChain
- Host: GitHub
- URL: https://github.com/linonetwo/langchain-alpaca
- Owner: linonetwo
- License: mit
- Created: 2023-03-18T10:56:05.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-12-14T20:33:28.000Z (11 months ago)
- Last Synced: 2024-08-08T19:04:51.782Z (3 months ago)
- Topics: alpaca, chatgpt, llama, llm, local-first, offline-first, self-hosted
- Language: TypeScript
- Homepage: https://www.npmjs.com/package/langchain-alpaca
- Size: 89.8 KB
- Stars: 218
- Watchers: 5
- Forks: 6
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# LangChain-Alpaca
Run alpaca LLM fully locally in langchain.
```shell
pnpm i langchain-alpaca
```
You can store following example as `loadLLM.mjs` , run it with google/zx by `DEBUG=langchain-alpaca:* zx './loadLLM.mjs'`
```ts
/* eslint-disable no-undef */
import { AlpacaCppChat, getPhysicalCore } from 'langchain-alpaca'
import path from 'node:path'
console.time('LoadAlpaca')
const alpaca = new AlpacaCppChat({
// replace this with your local model path.
modelParameters: { model: '/Users/linonetwo/Desktop/model/LanguageModel/ggml-alpaca-7b-q4.bin', threads: getPhysicalCore() - 1 },
})
const response = await alpaca.generate(['Say "hello world"']).catch((error) => console.error(error))
console.timeEnd('LoadAlpaca')
console.log(`response`, response, JSON.stringify(response))
// close the node-tty session to free the memory used by alpaca.cpp. You can query alpaca as much as you want before closing it.
alpaca.closeSession()
```
See `example/*.mjs` for more examples. Run with env `DEBUG=langchain-alpaca:*` will show internal debug details, useful when you found this LLM not responding to input.
Read [doc of LangChainJS](https://hwchase17.github.io/langchainjs/docs/overview/) to learn how to build a fully localized free AI workflow for you.
### Prebuild Binary
By default, langchain-alpaca bring prebuild binry with it. But it will still try to build one when `postinstall` , which should be very fast, and produce a somehow faster version of binary. This is optional, and will fail in silence, because it still work with prebuild binary.
If you are using windows, and want to make postinstall work, make sure download and install CMake: https://cmake.org/download/ as said in https://github.com/antimatter15/alpaca.cpp#windows-setup .
## Parameter of AlpacaCppChat
```ts
export interface AlpacaCppModelParameters {
/** run in interactive mode
* (This also means to stream the results in langchain)
*/
interactive?: boolean
/** run in interactive mode and poll user input at startup */
interactiveStart?: boolean
/** in interactive mode, poll user input upon seeing PROMPT */
reversePrompt?: string | null
/** colorise output to distinguish prompt and user input from generations */
color?: boolean
/** RNG seed (default: -1) */
seed?: number
/** number of threads to use during computation (default: %d) */
threads?: number
/** prompt to start generation with (default: random) */
prompt?: string | null
/** prompt file to start generation */
file?: string | null
/** number of tokens to predict (default: %d) */
n_predict?: number
/** top-k sampling (default: %d) */
top_k?: number
/** top-p sampling (default: %.1f) */
top_p?: number
/** last n tokens to consider for penalize (default: %d) */
repeat_last_n?: number
/** penalize repeat sequence of tokens (default: %.1f) */
repeat_penalty?: number
/** size of the prompt context (default: %d) */
ctx_size?: number
/** temperature (default: %.1f) */
temp?: number
/** batch size for prompt processing (default: %d) */
batch_size?: number
/** model path, absolute or relative location of `ggml-alpaca-7b-q4.bin` model file (default: %s) */
model?: string
}
export interface AlpacaCppChatParameters {
/**
* Working directory of dist file. Default to `path.join(path.dirname(require.resolve('langchain-alpaca')), 'binary')`.
* If you are using esm, try set this to node_modules/langchain-alpaca/dist/binary
*
cwd: string
/**
* Name of alpaca.cpp binary
*/
cmd: string
shell: string
}
```
Use params like this:
```ts
new AlpacaCppChat({ fields?: BaseLLMParams & Partial & { modelParameters?: Partial })
```
Where `BaseLLMParams` is from `langchain` core package.
## Development
During dev, you can put your model (or `ln -s` it) in the `model/ggml-alpaca-7b-q4.bin`.
And run the `zx example/loadLLM.mjs` to test it.