{"id":15167000,"url":"https://github.com/fardjad/node-llmatic","last_synced_at":"2025-09-30T23:30:40.182Z","repository":{"id":168069738,"uuid":"643689466","full_name":"fardjad/node-llmatic","owner":"fardjad","description":"Use self-hosted LLMs with an OpenAI compatible API","archived":true,"fork":false,"pushed_at":"2024-04-01T11:37:45.000Z","size":16641,"stargazers_count":64,"open_issues_count":1,"forks_count":12,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-09-01T16:56:07.882Z","etag":null,"topics":["api","llama","llm","openai"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fardjad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-22T00:06:09.000Z","updated_at":"2025-03-12T09:11:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"9194f959-451c-4498-938d-aed80bd342fb","html_url":"https://github.com/fardjad/node-llmatic","commit_stats":{"total_commits":276,"total_committers":2,"mean_commits":138.0,"dds":0.1811594202898551,"last_synced_commit":"ff59eb04acced04224b5ae615c9a9578c6422a88"},"previous_names":["fardjad/node-llmatic"],"tags_count":230,"template":false,"template_full_name":null,"purl":"pkg:github/fardjad/node-llmatic","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fardjad%2Fnode-llmatic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fardjad%2Fnode-llmatic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fardjad%2Fnode-llmatic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fardjad%2Fnode-llmatic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fardjad","download_url":"https://codeload.github.com/fardjad/node-llmatic/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fardjad%2Fnode-llmatic/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274231507,"owners_count":25245600,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-08T02:00:09.813Z","response_time":121,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","llama","llm","openai"],"created_at":"2024-09-27T05:21:28.073Z","updated_at":"2025-09-30T23:30:39.105Z","avatar_url":"https://github.com/fardjad.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# LLMatic\n\n\u003cimg alt=\"LLMatic Logo\" width=\"300px\" height=\"300px\" src=\"/media/logo.png\"\u003e\n\nUse self-hosted LLMs with an OpenAI compatible API\n\n\u003cdiv class=\"paragraph\"\u003e\n\n\u003cspan class=\"image\"\u003e\u003ca href=\"https://www.npmjs.com/package/llmatic\" class=\"image\"\u003e\u003cimg src=\"https://img.shields.io/npm/v/llmatic\" alt=\"llmatic\" /\u003e\u003c/a\u003e\u003c/span\u003e \u003cspan class=\"image\"\u003e\u003ca href=\"https://www.npmjs.com/package/llmatic\" class=\"image\"\u003e\u003cimg src=\"https://img.shields.io/npm/dm/llmatic\" alt=\"llmatic\" /\u003e\u003c/a\u003e\u003c/span\u003e \u003cspan class=\"image\"\u003e\u003ca href=\"https://github.com/fardjad/node-llmatic/actions\" class=\"image\"\u003e\u003cimg src=\"https://img.shields.io/github/actions/workflow/status/fardjad/node-llmatic/test-and-release.yml?branch=master\" alt=\"test and release\" /\u003e\u003c/a\u003e\u003c/span\u003e\n\n\u003c/div\u003e\n\n\u003c/div\u003e\n\n\u003chr /\u003e\n\n## Project status\n\nThis project was the result of my curiousity and experimentation with OpenAI's API and I enjoyed building it. It is certainly not the first nor the last project of its kind. Given my limited time and resources, I'd like to pause the development of this project for now. I'll list some other similar projects below that can be used as alternatives:\n\n1. [Ollama](https://github.com/ollama/ollama/blob/main/docs/openai.md)\n2. [LLaMA.cpp HTTP Server](https://github.com/ggerganov/llama.cpp/tree/master/examples/server)\n3. [GPT4All Chat Server Mode](https://docs.gpt4all.io/gpt4all_chat.html#gpt4all-chat-server-mode)\n4. [FastChat](https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md)\n\n\n## Synopsis\n\nLLMatic can be used as a drop-in replacement for OpenAI's API [v1.2.0](https://github.com/openai/openai-openapi/blob/88f221442879061d9970ed453a65b973d226f15d/openapi.yaml) (see the\nsupported endpoints). By default, it uses [llama-node](https://github.com/Atome-FE/llama-node)\nwith [llama.cpp](https://github.com/ggerganov/llama.cpp) backend to run the models locally. However, you can easily create [your own adapter](#custom-adapters) to use any other model or service.\n\nSupported endpoints:\n\n- [x] /completions (stream and non-stream)\n- [x] /chat/completions (stream and non-stream)\n- [x] /embeddings\n- [x] /models\n\n## How to use\n\nIf you prefer a video tutorial, you can watch the following video for step-by-step instructions on how to use this project:\n\n\u003ca href=\"http://www.youtube.com/watch?feature=player_embedded\u0026v=V_baaAZMY44\" target=\"_blank\"\u003e\n\u003cimg src=\"https://img.youtube.com/vi/V_baaAZMY44/hqdefault.jpg\" alt=\"LLMatic\" style=\"min-height: 200px\" /\u003e\n\u003c/a\u003e\n\n### Requirements\n\n- Node.js \u003e=18.16\n- Unix-based OS (Linux, macOS, WSL, etc.)\n\n### Installation\n\nCreate an empty directory and run `npm init`:\n\n```bash\nexport LLMATIC_PROJECT_DIR=my-llmatic-project\nmkdir $LLMATIC_PROJECT_DIR\ncd $LLMATIC_PROJECT_DIR\nnpm init -y\n```\n\nInstall and configure LLMatic:\n\n```bash\nnpm add llmatic\n# Download a model and generate a config file\nnpx llmatic config\n```\n\nAdjust the config file to your needs and start the server:\n\n```bash\nnpx llmatic start\n```\n\nYou can run `llmatic --help` to see all available commands.\n\n### Usage with [chatbot-ui](https://github.com/mckaywrigley/chatbot-ui)\n\nClone the repo and install the dependencies:\n\n```bash\ngit clone https://github.com/mckaywrigley/chatbot-ui.git\ncd chatbot-ui\nnpm install\n```\n\nCreate a `.env.local` file:\n\n```bash\ncat \u003c\u003cEOF \u003e .env.local\n# For now, this is ignored by LLMatic\nDEFAULT_MODEL=Ignored\n\nNEXT_PUBLIC_DEFAULT_SYSTEM_PROMPT=A chat between a curious human (user) and an artificial intelligence assistant (assistant). The assistant gives helpful, detailed, and polite answers to the human's questions.\n\nuser: Hello!\nassistant: Hello! How may I help you today?\nuser: Please tell me the largest city in Europe.\nassistant: Sure. The largest city in Europe is Moscow, the capital of Russia.\n\nOPENAI_API_KEY=ANYTHING_WILL_DO\nOPENAI_API_HOST=http://localhost:3000\n\nGOOGLE_API_KEY=YOUR_API_KEY\nGOOGLE_CSE_ID=YOUR_ENGINE_ID\nEOF\n```\n\nRun the server:\n\n```bash\nnpm run dev -- --port 3001\n```\n\nDemo:\n\n![chatbot-ui Demo](/media/chatbot-ui.gif)\n\n### Usage with [LangChain](https://langchain.com)\n\nThere are two examples of using LLMatic with LangChain in the\n[`examples`](/examples) directory.\n\nTo run the Node.js example, first install the dependencies:\n\n```bash\ncd examples/node-langchain\nnpm install\n```\n\nThen run the main script:\n\n```bash\nnpm start\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eExpand this to see the sample output\u003c/summary\u003e\n\n```\n[chain/start] [1:chain:llm_chain] Entering Chain run with input: {\n  \"humanInput\": \"Rememeber that this is a demo of LLMatic with LangChain.\",\n  \"history\": \"\"\n}\n[llm/start] [1:chain:llm_chain \u003e 2:llm:openai] Entering LLM run with input: {\n  \"prompts\": [\n    \"A chat between a curious user and an artificial intelligence assistant.\\nThe assistant gives helpful, detailed, and polite answers to the user's questions.\\n\\n\\nHuman: Rememeber that this is a demo of LLMatic with LangChain.\\nAI:\"\n  ]\n}\n[llm/end] [1:chain:llm_chain \u003e 2:llm:openai] [5.92s] Exiting LLM run with output: {\n  \"generations\": [\n    [\n      {\n        \"text\": \" Yes, I understand. I am ready to assist you with your queries.\",\n        \"generationInfo\": {\n          \"finishReason\": \"stop\",\n          \"logprobs\": null\n        }\n      }\n    ]\n  ],\n  \"llmOutput\": {\n    \"tokenUsage\": {}\n  }\n}\n[chain/end] [1:chain:llm_chain] [5.92s] Exiting Chain run with output: {\n  \"text\": \" Yes, I understand. I am ready to assist you with your queries.\"\n}\n[chain/start] [1:chain:llm_chain] Entering Chain run with input: {\n  \"humanInput\": \"What did I ask you to remember?\",\n  \"history\": \"Human: Rememeber that this is a demo of LLMatic with LangChain.\\nAI:  Yes, I understand. I am ready to assist you with your queries.\"\n}\n[llm/start] [1:chain:llm_chain \u003e 2:llm:openai] Entering LLM run with input: {\n  \"prompts\": [\n    \"A chat between a curious user and an artificial intelligence assistant.\\nThe assistant gives helpful, detailed, and polite answers to the user's questions.\\n\\nHuman: Rememeber that this is a demo of LLMatic with LangChain.\\nAI:  Yes, I understand. I am ready to assist you with your queries.\\nHuman: What did I ask you to remember?\\nAI:\"\n  ]\n}\n[llm/end] [1:chain:llm_chain \u003e 2:llm:openai] [6.51s] Exiting LLM run with output: {\n  \"generations\": [\n    [\n      {\n        \"text\": \" You asked me to remember that this is a demo of LLMatic with LangChain.\",\n        \"generationInfo\": {\n          \"finishReason\": \"stop\",\n          \"logprobs\": null\n        }\n      }\n    ]\n  ],\n  \"llmOutput\": {\n    \"tokenUsage\": {}\n  }\n}\n[chain/end] [1:chain:llm_chain] [6.51s] Exiting Chain run with output: {\n  \"text\": \" You asked me to remember that this is a demo of LLMatic with LangChain.\"\n}\n```\n\n\u003c/details\u003e\n\n\u003chr\u003e\n\nTo run the Python example, first install the dependencies:\n\n```bash\ncd examples/python-langchain\npip3 install -r requirements.txt\n```\n\nThen run the main script:\n\n```bash\npython3 main.py\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eExpand this to see the sample output\u003c/summary\u003e\n\n```\n\u003e Entering new LLMChain chain...\nPrompt after formatting:\nA chat between a curious user and an artificial intelligence assistant.\nThe assistant gives helpful, detailed, and polite answers to the user's questions.\n\n\nHuman: Rememeber that this is a demo of LLMatic with LangChain.\nAI:\n\n\u003e Finished chain.\n Yes, I understand. I am ready to assist you with your queries.\n\n\n\u003e Entering new LLMChain chain...\nPrompt after formatting:\nA chat between a curious user and an artificial intelligence assistant.\nThe assistant gives helpful, detailed, and polite answers to the user's questions.\n\nHuman: Rememeber that this is a demo of LLMatic with LangChain.\nAI:  Yes, I understand. I am ready to assist you with your queries.\nHuman: What did I ask you to remember?\nAI:\n\n\u003e Finished chain.\n You asked me to remember that this is a demo of LLMatic with LangChain.\n```\n\n\u003c/details\u003e\n\n## Custom Adapters\n\nLLMatic is designed to be easily extensible. You can create your own adapters by extending the [`LlmAdapter`](/src/llm-adapter.ts) class. See [`examples/custom-adapter`](/examples/custom-adapter) for an example.\n\nTo start llmatic with a custom adapter, use the `--llm-adapter` flag:\n\n```bash\nllmatic start --llm-adapter ./custom-llm-adapter.ts\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffardjad%2Fnode-llmatic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffardjad%2Fnode-llmatic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffardjad%2Fnode-llmatic/lists"}