{"id":13431813,"url":"https://github.com/cocktailpeanut/dalai","last_synced_at":"2025-05-13T20:17:36.393Z","repository":{"id":142275556,"uuid":"613075789","full_name":"cocktailpeanut/dalai","owner":"cocktailpeanut","description":"The simplest way to run LLaMA on your local machine","archived":false,"fork":false,"pushed_at":"2024-06-18T20:29:46.000Z","size":12240,"stargazers_count":13081,"open_issues_count":337,"forks_count":1398,"subscribers_count":148,"default_branch":"main","last_synced_at":"2025-05-11T00:32:06.946Z","etag":null,"topics":["ai","llama","llm"],"latest_commit_sha":null,"homepage":"https://cocktailpeanut.github.io/dalai","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cocktailpeanut.png","metadata":{"files":{"readme":"docs/README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-12T20:07:32.000Z","updated_at":"2025-05-10T22:38:40.000Z","dependencies_parsed_at":null,"dependency_job_id":"8f15428d-ca44-4a50-8734-9aefacd59b14","html_url":"https://github.com/cocktailpeanut/dalai","commit_stats":{"total_commits":127,"total_committers":21,"mean_commits":"6.0476190476190474","dds":0.4173228346456693,"last_synced_commit":"e4f678efa274c08414227193b7c9607dd8f60db7"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cocktailpeanut%2Fdalai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cocktailpeanut%2Fdalai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cocktailpeanut%2Fdalai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cocktailpeanut%2Fdalai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cocktailpeanut","download_url":"https://codeload.github.com/cocktailpeanut/dalai/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253658076,"owners_count":21943423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","llama","llm"],"created_at":"2024-07-31T02:01:06.109Z","updated_at":"2025-05-13T20:17:33.656Z","avatar_url":"https://github.com/cocktailpeanut.png","language":"CSS","funding_links":[],"categories":["CSS","Tools","JavaScript","HarmonyOS","A01_文本生成_文本对话","Materials","Uncategorized","Langchain","llm","ai","Tools (GitHub)","📦 Legacy \u0026 Inactive Projects"],"sub_categories":["Windows Manager","大语言对话模型及数据","GitHub repositories","Uncategorized","LLMOps vs MLOps"],"readme":"# Dalai\n\nRun LLaMA and Alpaca on your computer.\n\n\u003ca href=\"https://github.com/cocktailpeanut/dalai\" class='inverse btn'\u003e\u003ci class=\"fa-brands fa-github\"\u003e\u003c/i\u003e GitHub\u003c/a\u003e\n\u003ca href=\"https://twitter.com/cocktailpeanut\" class='inverse btn'\u003e\u003ci class=\"fa-brands fa-twitter\"\u003e\u003c/i\u003e Twitter\u003c/a\u003e\n\u003ca href=\"https://discord.gg/WWfgrzzkCT\" class='inverse btn'\u003e\u003ci class=\"fa-brands fa-discord\"\u003e\u003c/i\u003e Discord\u003c/a\u003e\n\n---\n\n## JUST RUN THIS\n\n\u003cimg src=\"npx2.png\" class='round'\u003e\n\n## TO GET\n\nBoth alpaca and llama working on your computer!\n\n![alpaca.gif](alpaca.gif)\n\n---\n\n1. Powered by [llama.cpp](https://github.com/ggerganov/llama.cpp), [llama-dl CDN](https://github.com/shawwn/llama-dl), and [alpaca.cpp](https://github.com/antimatter15/alpaca.cpp)\n2. Hackable web app included\n3. Ships with JavaScript API\n4. Ships with [Socket.io](https://socket.io/) API\n\n---\n\n# Intro\n\n## 1. Cross platform\n\nDalai runs on all of the following operating systems:\n\n1. Linux\n2. Mac\n3. Windows\n\n## 2. Memory Requirements\n\nRuns on most modern computers. Unless your computer is very very old, it should work.\n\nAccording to [a llama.cpp discussion thread](https://github.com/ggerganov/llama.cpp/issues/13), here are the memory requirements:\n\n- 7B =\u003e ~4 GB\n- 13B =\u003e ~8 GB\n- 30B =\u003e ~16 GB\n- 65B =\u003e ~32 GB\n\n## 3. Disk Space Requirements\n\n### Alpaca\n\nCurrently 7B and 13B models are available via [alpaca.cpp](https://github.com/antimatter15/alpaca.cpp)\n\n#### 7B\n\nAlpaca comes fully quantized (compressed), and the only space you need for the 7B model is 4.21GB:\n\n![alpaca_7b.png](alpaca_7b.png)\n\n#### 13B\n\nAlpaca comes fully quantized (compressed), and the only space you need for the 13B model is 8.14GB:\n\n![alpaca_13b.png](alpaca_13b.png)\n\n\n### LLaMA\n\nYou need a lot of space for storing the models. **The model name must be one of: 7B, 13B, 30B, and 65B.**\n\nYou do NOT have to install all models, you can install one by one. Let's take a look at how much space each model takes up:\n\n\u003e NOTE\n\u003e\n\u003e The following numbers assume that you DO NOT touch the original model files and keep BOTH the original model files AND the quantized versions.\n\u003e\n\u003e You can optimize this if you delete the original models (which are much larger) after installation and keep only the quantized versions.\n\n#### 7B\n\n- Full: The model takes up 31.17GB\n- Quantized: 4.21GB\n\n![7b.png](7b.png)\n\n#### 13B\n\n- Full: The model takes up 60.21GB\n- Quantized: 4.07GB * 2 = 8.14GB\n\n![13b.png](13b.png)\n\n#### 30B\n\n- Full: The model takes up 150.48GB\n- Quantized: 5.09GB * 4 = 20.36GB\n\n![30b.png](30b.png)\n\n#### 65B\n\n- Full: The model takes up 432.64GB\n- Quantized: 5.11GB * 8 = 40.88GB\n\n![65b.png](65b.png)\n\n---\n\n# Quickstart\n\n## Docker compose\n\nRequires that you have docker installed and running.\n```\ndocker compose build\ndocker compose run dalai npx dalai alpaca install 7B # or a different model\ndocker compose up -d\n```\n\nThis will dave the models in the `./models` folder\n\nView the site at http://127.0.0.1:3000/\n\n## Mac\n\n### Step 1. Install node.js \u003e= 18\n\nIf your mac doesn't have node.js installed yet, make sure to install node.js \u003e= 18\n\n\u003ca href=\"https://nodejs.org/en/download/\" class='btn'\u003eInstall Node.js\u003c/a\u003e\n\n\n### Step 2.1. Install models\n\nCurrently supported engines are `llama` and `alpaca`.\n\n#### Add alpaca models\n\nTo download alpaca models, you can run:\n\n```\nnpx dalai alpaca install 7B\n```\n\n\n#### Add llama models\n\nTo download llama models, you can run:\n\n```\nnpx dalai llama install 7B\n```\n\n\nor to download multiple models:\n\n```\nnpx dalai llama install 7B 13B\n```\n\nNow go to step 3.\n\n### Step 2.2. Troubleshoot\n\nNormally you don't need this step, but if running the commands above don't do anything and immediately end, it means something went wrong because some of the required modules are not installed on your system.\n\nIn that case, try the following steps:\n\n#### 1. Install homebrew\n\nIn case homebrew is not installed on your computer, install it by running:\n\n```\n/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"\n```\n\n\u003e Or you can find the same instruction on the homebrew hompage: https://brew.sh/\n\n#### 2. Install dependencies\n\nOnce homebrew is installed, install these dependencies:\n\n```\nbrew install cmake\nbrew install pkg-config\n```\n\n#### 3. Update NPM\n\nJust to make sure we cover every vector, let's update NPM as well:\n\n```\nnpm install -g npm@latest\n```\n\nNow go back to step 2.1 and try running the `npx dalai` commands again.\n\n### Step 3. Run Web UI\n\nAfter everything has been installed, run the following command to launch the web UI server:\n\n```\nnpx dalai serve\n```\n\nand open http://localhost:3000 in your browser. Have fun!\n\n---\n\n## Windows\n\n### Step 1. Install Visual Studio\n\nOn windows, you need to install Visual Studio before installing Dalai.\n\nPress the button below to visit the Visual Studio downloads page and download:\n\n\u003ca href=\"https://visualstudio.microsoft.com/downloads/\" class='btn'\u003eDownload Microsoft Visual Studio\u003c/a\u003e\n\n**IMPORTANT!!!**\n\nWhen installing Visual Studio, make sure to check the 3 options as highlighted below:\n\n1. Python development\n2. Node.js development\n3. Desktop development with C++\n\n![vs.png](vs.png)\n\n---\n\n### Step 2.1. Install models\n\n\u003e **IMPORTANT**\n\u003e\n\u003e On Windows, make sure to run all commands in **cmd**.\n\u003e\n\u003e DO NOT run in **powershell**. Powershell has unnecessarily strict permissions and makes the script fail silently.\n\nCurrently supported engines are `llama` and `alpaca`.\n\n#### Install alpaca\n\nTo download alpaca models. Open your `cmd` application and enter:\n\n```\nnpx dalai alpaca install 7B\n```\n\n\n#### Add llama models\n\nTo download llama models. Open your `cmd` application and enter:\n\n```\nnpx dalai llama install 7B\n```\n\n\nor to download multiple models:\n\n```\nnpx dalai llama install 7B 13B\n```\n\n---\n\n### Step 2.2. Troubleshoot (optional)\n\nIn case above steps fail, try installing Node.js and Python separately.\n\nInstall Python:\n\n\u003ca href=\"https://www.python.org/ftp/python/3.10.10/python-3.10.10-embed-amd64.zip\" class='btn'\u003eDownload Python\u003c/a\u003e\n\nInstall Node.js \u003e= 18:\n\n\u003ca href=\"https://nodejs.org/en/download/\" class='btn'\u003eDownload Node.js\u003c/a\u003e\n\nAfter both have been installed, open powershell and type `python` to see if the application exists. And also type `node` to see if the application exists as well.\n\nOnce you've checked that they both exist, try again.\n\n\n### Step 3. Run Web UI\n\nAfter everything has been installed, run the following command to launch the web UI server (Make sure to run in `cmd` and not powershell!):\n\n```\nnpx dalai serve\n```\n\nand open http://localhost:3000 in your browser. Have fun!\n\n---\n\n\n## Linux\n\n### Step 1. Install Dependencies\n\nYou need to make sure you have the correct version of Python and Node.js installed.\n\n#### Step 1.1. Python \u003c= 3.10\n\n\u003ca href=\"https://pimylifeup.com/installing-python-on-linux/\" class='btn'\u003eDownload Python\u003c/a\u003e\n\n\u003e Make sure the version is 3.10 or lower (not 3.11)\nPython must be 3.10 or below (pytorch and other libraries are not supported yet on the latest)\n\n\n#### Step 1.2. Node.js \u003e= 18\n\n\u003ca href=\"https://nodejs.org/en/download/package-manager/\" class='btn'\u003eDownload node.js\u003c/a\u003e\n\n\u003e Make sure the version is 18 or higher\n\n\n\n---\n\n### Step 2.1. Install models\n\nCurrently supported engines are `llama` and `alpaca`.\n\n#### Add alpaca models\n\nTo download alpaca models, you can run:\n\n```\nnpx dalai alpaca install 7B\n```\n\n\n#### Add llama models\n\nTo download llama models, you can run:\n\n```\nnpx dalai llama install 7B\n```\n\n\nor to download multiple models:\n\n```\nnpx dalai llama install 7B 13B\n```\n\n### Step 2.2. Troubleshoot\n\nIn case the model install silently fails or hangs forever, try the following command, and try running the npx command again:\n\nOn ubuntu/debian/etc.:\n\n```\nsudo apt-get install build-essential python3-venv -y\n```\n\nOn fedora/etc.:\n\n```\ndnf install make automake gcc gcc-c++ kernel-devel python3-virtualenv -y\n```\n\n\n### Step 3. Run Web UI\n\nAfter everything has been installed, run the following command to launch the web UI server:\n\n```\nnpx dalai serve\n```\n\nand open http://localhost:3000 in your browser. Have fun!\n\n\n\n\n\n---\n\n\n# API\n\nDalai is also an NPM package:\n\n1. programmatically install\n2. locally make requests to the model\n3. run a dalai server (powered by socket.io)\n3. programmatically make requests to a remote dalai server (via socket.io)\n\nDalai is an NPM package. You can install it using:\n\n```\nnpm install dalai\n```\n\n---\n\n## 1. constructor()\n\n### Syntax\n\n```javascript\nconst dalai = new Dalai(home)\n```\n\n- `home`: (optional) manually specify the [llama.cpp](https://github.com/ggerganov/llama.cpp) folder\n\nBy default, Dalai automatically stores the entire `llama.cpp` repository under `~/llama.cpp`.\n\nHowever, often you may already have a `llama.cpp` repository somewhere else on your machine and want to just use that folder. In this case you can pass in the `home` attribute.\n\n### Examples\n\n#### Basic\n\nCreates a workspace  at `~/llama.cpp`\n\n```javascript\nconst dalai = new Dalai()\n```\n\n#### Custom path\n\nManually set the `llama.cpp` path:\n\n\n```javascript\nconst dalai = new Dalai(\"/Documents/llama.cpp\")\n```\n\n---\n\n## 2. request()\n\n### Syntax\n\n```javascript\ndalai.request(req, callback)\n```\n\n- `req`: a request object. made up of the following attributes:\n  - `prompt`: **(required)** The prompt string\n  - `model`: **(required)** The model type + model name to query. Takes the following form: `\u003cmodel_type\u003e.\u003cmodel_name\u003e`\n    - Example: `alpaca.7B`, `llama.13B`, ...\n  - `url`: only needed if connecting to a remote dalai server\n    - if unspecified, it uses the node.js API to directly run dalai locally\n    - if specified (for example `ws://localhost:3000`) it looks for a socket.io endpoint at the URL and connects to it.\n  - `threads`: The number of threads to use (The default is 8 if unspecified)\n  - `n_predict`: The number of tokens to return (The default is 128 if unspecified)\n  - `seed`: The seed. The default is -1 (none)\n  - `top_k`\n  - `top_p`\n  - `repeat_last_n`\n  - `repeat_penalty`\n  - `temp`: temperature\n  - `batch_size`: batch size\n  - `skip_end`: by default, every session ends with `\\n\\n\u003cend\u003e`, which can be used as a marker to know when the full response has returned. However sometimes you may not want this suffix. Set `skip_end: true` and the response will no longer end with `\\n\\n\u003cend\u003e`\n- `callback`: the streaming callback function that gets called every time the client gets any token response back from the model\n\n### Examples\n\n#### 1. Node.js\n\nUsing node.js, you just need to initialize a Dalai object with `new Dalai()` and then use it.\n\n```javascript\nconst Dalai = require('dalai')\nnew Dalai().request({\n  model: \"7B\",\n  prompt: \"The following is a conversation between a boy and a girl:\",\n}, (token) =\u003e {\n  process.stdout.write(token)\n})\n```\n\n#### 2. Non node.js (socket.io)\n\nTo make use of this in a browser or any other language, you can use thie socket.io API.\n\n##### Step 1. start a server\n\nFirst you need to run a Dalai socket server:\n\n```javascript\n// server.js\nconst Dalai = require('dalai')\nnew Dalai().serve(3000)     // port 3000\n```\n\n##### Step 2. connect to the server\n\nThen once the server is running, simply make requests to it by passing the `ws://localhost:3000` socket url when initializing the Dalai object:\n\n```javascript\nconst Dalai = require(\"dalai\")\nnew Dalai().request({\n  url: \"ws://localhost:3000\",\n  model: \"7B\",\n  prompt: \"The following is a conversation between a boy and a girl:\",\n}, (token) =\u003e {\n  console.log(\"token\", token)\n})\n```\n\n---\n\n## 3. serve()\n\n### Syntax\n\nStarts a socket.io server at `port`\n\n```javascript\ndalai.serve(port)\n```\n\n### Examples\n\n```javascript\nconst Dalai = require(\"dalai\")\nnew Dalai().serve(3000)\n```\n\n---\n\n## 4. http()\n\n### Syntax\n\nconnect with an existing `http` instance (The `http` npm package)\n\n```javascript\ndalai.http(http)\n```\n\n- `http`: The [http](https://nodejs.org/api/http.html) object\n\n### Examples\n\nThis is useful when you're trying to plug dalai into an existing node.js web app\n\n```javascript\nconst app = require('express')();\nconst http = require('http').Server(app);\ndalai.http(http)\nhttp.listen(3000, () =\u003e {\n  console.log(\"server started\")\n})\n```\n\n## 5. install()\n\n### Syntax\n\n```javascript\nawait dalai.install(model_type, model_name1, model_name2, ...)\n```\n\n- `model_type`: the name of the model. currently supports:\n  - \"alpaca\"\n  - \"llama\"\n- `model1`, `model2`, ...: the model names to install (\"7B\"`, \"13B\", \"30B\", \"65B\", etc)\n\n### Examples\n\nInstall Llama \"7B\" and \"13B\" models:\n\n\n```javascript\nconst Dalai = require(\"dalai\");\nconst dalai = new Dalai()\nawait dalai.install(\"llama\", \"7B\", \"13B\")\n```\n\nInstall alpaca 7B model:\n\n```javascript\nconst Dalai = require(\"dalai\");\nconst dalai = new Dalai()\nawait dalai.install(\"alpaca\", \"7B\")\n```\n\n---\n\n## 6. installed()\n\nreturns the array of installed models\n\n### Syntax\n\n```javascript\nconst models = await dalai.installed()\n```\n\n### Examples\n\n\n```javascript\nconst Dalai = require(\"dalai\");\nconst dalai = new Dalai()\nconst models = await dalai.installed()\nconsole.log(models)     // prints [\"7B\", \"13B\"]\n```\n\n\u003c!--\n\n---\n\n## 7. download()\n\nDownload models.\n\nThere are two download options:\n\n1. **LLaMA:** Download the original LLaMA model, convert it, and quantize (compress) it\n2. **LLaMA.zip:** Download the compressed version (generated from step 1 and published on HuggingFace)\n\n### Syntax\n\n```javascript\nawait dalai.download(model1, model2, model3, ...)\n```\n\n- `models`: the model names to install. Can be: \"7B\"`, \"13B\", \"30B\", \"65B\", \"7B.zip\", \"13B.zip\", \"30B.zip\", \"65B.zip\"\n  - \"7B\", \"13B\", \"30B\", \"65B\": download the raw model, convert, and quantize\n  - \"7B.zip\", \"13B.zip\", \"30B.zip\", \"65B.zip\": download the quantized model (no need to waste time downloading huge files)\n\n### Examples\n\nInstall the \"7B\" and \"13B\" models:\n\n\n```javascript\nconst Dalai = require(\"dalai\");\nconst dalai = new Dalai()\nawait dalai.install(\"7B\", \"13B\")\n```\n\n--\u003e\n\n---\n\n\n# FAQ\n\n## Using a different home folder\n\nBy default Dalai uses your home directory to store the entire repository (`~/dalai`). However sometimes you may want to store the archive elsewhere.\n\nIn this case you can call all CLI methods using the `--home` flag:\n\n### 1. Installing models to a custom path\n\n```\nnpx dalai llama install 7B --home ~/test_dir\n```\n\n### 2. Serving from the custom path\n\n```\nnpx dalai serve --home ~/test_dir\n```\n\n\n## Updating to the latest\n\nTo make sure you update to the latest, first find the latest version at https://www.npmjs.com/package/dalai\n\nLet's say the latest version is `0.3.0`. To update the dalai version, run:\n\n```\nnpx dalai@0.3.0 setup\n```\n\n\n## Staying up to date\n\nHave questions or feedback? Follow the project through the following outlets:\n\n\u003ca href=\"https://github.com/cocktailpeanut/dalai\" class='inverse btn'\u003e\u003ci class=\"fa-brands fa-github\"\u003e\u003c/i\u003e GitHub\u003c/a\u003e\n\u003ca href=\"https://twitter.com/cocktailpeanut\" class='inverse btn'\u003e\u003ci class=\"fa-brands fa-twitter\"\u003e\u003c/i\u003e Twitter\u003c/a\u003e\n\u003ca href=\"https://discord.gg/WWfgrzzkCT\" class='inverse btn'\u003e\u003ci class=\"fa-brands fa-discord\"\u003e\u003c/i\u003e Discord\u003c/a\u003e\n\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcocktailpeanut%2Fdalai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcocktailpeanut%2Fdalai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcocktailpeanut%2Fdalai/lists"}