{"id":22160528,"url":"https://github.com/rozek/node-red-flow-stablelm-3b-4e1t","last_synced_at":"2026-01-05T07:50:37.451Z","repository":{"id":208393457,"uuid":"721517498","full_name":"rozek/node-red-flow-stablelm-3b-4e1t","owner":"rozek","description":"Node-RED Flows for the StableLM-3B-4E1T AI model","archived":false,"fork":false,"pushed_at":"2023-11-22T14:31:25.000Z","size":233,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-29T20:29:30.214Z","etag":null,"topics":["ai-chatbot","node-red","node-red-flow","stability-ai","stablelm"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rozek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-11-21T08:21:45.000Z","updated_at":"2023-11-21T08:50:56.000Z","dependencies_parsed_at":"2023-11-21T09:43:45.935Z","dependency_job_id":null,"html_url":"https://github.com/rozek/node-red-flow-stablelm-3b-4e1t","commit_stats":null,"previous_names":["rozek/node-red-flow-stablelm-3b-4e1t"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rozek%2Fnode-red-flow-stablelm-3b-4e1t","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rozek%2Fnode-red-flow-stablelm-3b-4e1t/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rozek%2Fnode-red-flow-stablelm-3b-4e1t/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rozek%2Fnode-red-flow-stablelm-3b-4e1t/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rozek","download_url":"https://codeload.github.com/rozek/node-red-flow-stablelm-3b-4e1t/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245294718,"owners_count":20591909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-chatbot","node-red","node-red-flow","stability-ai","stablelm"],"created_at":"2024-12-02T04:08:26.606Z","updated_at":"2026-01-05T07:50:37.413Z","avatar_url":"https://github.com/rozek.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# node-red-flow-stablelm-3b-4e1t\n\nNode-RED Flows for the StableLM-3B-4E1T AI model\n\nThis repository contains a function node for [Node-RED](https://nodered.org/) which can be used to run the [Stability AI](https://huggingface.co/stabilityai) [StableLM-3B-4E1T model](https://huggingface.co/stabilityai/stablelm-3b-4e1t) using [llama.cpp](https://github.com/rozek/llama.cpp) within a Node-RED flow. **Inference is done on the CPU** (without requiring any special harware) and still completes within a few seconds on a reasonably powerful computer.\n\n![StableLM-3B-4E1T Text Completion Flow](./StableLM-3B-4E1T-Completion-Flow.png)\n\nAdditionally, this repo also contains function nodes to tokenize a prompt or to calculate embeddings based on the StableLM-3B-4E1T model.\n\nHaving the inference, tokenization and embedding calculation as a self-contained function node gives you the possibility to create your own user interface or even use it as part of an autonomous agent.\n\n\u003e Nota bene: these flows do not contain the actual model. You will have to download your own copy directly from [Huggingface](https://huggingface.co/rozek/StableLM-3B-4E1T_GGUF) (use file [stablelm-3b-4e1t-Q8_K.bin](https://huggingface.co/rozek/StableLM-3B-4E1T_GGUF/blob/main/stablelm-3b-4e1t-Q8_K.bin)).\n\n\u003e Just a small note: if you like this work and plan to use it, consider \"starring\" this repository (you will find the \"Star\" button on the top right of this page), so that I know which of my repositories to take most care of.\n\n## Installation ##\n\nThis section shows you how to install Node.js, Node-RED, LLaMA.cpp and the flows from this repository - feel free to skip the steps for those components you already installed before.\n\n### Node.js ###\n\n\"_[Node.js](https://nodejs.org/en) is a cross-platform, open-source server environment that can run on Windows, Linux, Unix, macOS, and more. Node.js is a back-end JavaScript runtime environment, runs on the V8 JavaScript engine, and executes JavaScript code outside a web browser._\" (according to [Wikipedia](https://en.wikipedia.org/wiki/Node.js))\n\nStart by [installing Node.js](https://nodejs.org/en) as described on their web page.\n\n### Node-RED ###\n\n\"_[Node-RED](https://nodered.org/) is a flow-based, low-code development tool for visual programming developed originally by IBM..._\" (according to [Wikipedia](https://en.wikipedia.org/wiki/Node-RED))\n\nIf not already done, install Node-RED as described on their \"[Get Started](https://nodered.org/#get-started)\" page.\n\n### LLaMA.cpp ###\n\n[LLaMA.cpp](https://github.com/ggerganov/llama.cpp) is a port of Facebook's LLaMA model in C/C++ (don't be fooled by the statement that the \"_main goal of llama.cpp is to run the LLaMA model using 4-bit integer quantization on a MacBook_\" - LLaMA.cpp runs just as well under Windows and Linux, and with different quantizations)\n\n\u003e Note: please use my own fork of the original LLaMA.cpp as the code in there raises the context limit and contains additional functions for tokenization - however, if you are fine with a context length of up to 4096 tokens and don't want to tokenize any prompts, the original code will work as fine\n\nGet the code as follows:\n\n```\ngit https://github.com/rozek/llama.cpp\ncd llama.cpp\n```\n\nThen continue as described in the [LLaMA.cpp docs](https://github.com/rozek/llama.cpp#build)\n\nAfterwards, rename the executables\n\n* `main` to `llama`,\n* `tokenization` to `llama-tokens` and\n* `embedding` to `llama-embeddings`\n\nand copy them into the Node-RED \"User Directory\" (by default, this folder is located at `$HOME/.node-red`).\n\n### StableLM-3B-4E1T Flows ###\n\nIf not already done, download the [8-bit quantization of the StableLM-3B-4E1T model](https://huggingface.co/rozek/StableLM-3B-4E1T_GGUF/blob/main/stablelm-3b-4e1t-Q8_K.bin) into the same folder that already contains your executables.\n\n\u003e Nota bene: right now, the flows from this repository support the given model file only - if you prefer another one, you may simply change the model file name in the function nodes for text completion, tokenization and embeddings calculation.\n\nNow import the desired nodes and flows - if you want them all, just import file [StableLM-3B-4E1T-Flows.json](https://raw.githubusercontent.com/rozek/node-red-flow-stablelm-3b-4e1t/master/StableLM-3B-4E1T-Flows.json).\n\nIf you are new to Node-RED, [just follow the instructions from their docs](https://nodered.org/docs/user-guide/editor/workspace/import-export).\n\n## Configuration ##\n\nBy default, all StableLM-3B-4E1T function nodes expect their executables (and the model itself) in the folder `$HOME/.node-red` (where `$HOME` is the user's home directory)\n\nIf you prefer to store everything in a different folder, simply edit the \"configure Settings\" node from [StableLM-3B-4E1T-Flows.json](https://raw.githubusercontent.com/rozek/node-red-flow-stablelm-3b-4e1t/master/StableLM-3B-4E1T-Flows.json) and set `globals.UserDir` to the fully qualified path of the directory you chose.\n\n## Function Node Usage ##\n\nAll function nodes expect their parameters as properties of the msg object. The prompt itself (or the input text to tokenize or calculate embeddings from) is expected in `msg.payload` and will later be replaced by the function result.\n\nAll properties (except prompt or input text) are optional. If given, they should be strings (even if they contain numbers), this makes it simpler to extract them from an HTTP request.\n\n### Text Completion Node ###\n\nText completion supports the following properties:\n\n* `payload` - this is the actual prompt \n* `seed` - seed value for the internal pseudo random number generator (integer, default: -1, use random seed for \u003c= 0)\n* `threads` - number of threads to use during computation (integer ≧ 1, default: 4)\n* `context` - size of the prompt context (0...4096, default: 512)\n* `keep` - number of tokens to keep from the initial prompt (integer ≧ -1, default: 0, -1 = all)\n* `predict` - number of tokens to predict (integer ≧ -1, default: 128, -1 = infinity)\n* `topk` - top-k sampling limit (integer ≧ 1, default: 40)\n* `topp` - top-p sampling limit (0.0...1.0, default: 0.9)\n* `temperature` - temperature (0.0...2.0, default: 0.8)\n* `batches` - batch size for prompt processing (integer ≧ 1, default: 8)\n* `grammar` - [GBNF grammar](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md) to restrict text completion output\n\n### Tokenization Node ###\n\nTokenization supports the following properties:\n\n* `payload` - this is the actual input text \n* `threads` - number of threads to use during computation (integer ≧ 1, default: 4)\n* `context` - size of the prompt context (0...4096, default: 512)\n\n### Embeddings Node ###\n\nEmbeddings calculation supports the following properties:\n\n* `payload` - this is the actual input text  \n* `seed` - seed value for the internal pseudo random number generator (integer, default: -1, use random seed for \u003c= 0)\n* `threads` - number of threads to use during computation (integer ≧ 1, default: 4)\n* `context` - size of the prompt context (0...4096, default: 512)\n\n## HTTP Endpoint Usage ##\n\nBesides the sole function nodes for\n\n* [text completion](https://raw.githubusercontent.com/rozek/node-red-flow-stablelm-3b-4e1t/master/StableLM-3B-4E1T-Completion-Function.json),\n* [tokenization](https://raw.githubusercontent.com/rozek/node-red-flow-stablelm-3b-4e1t/master/StableLM-3B-4E1T-Tokenization-Function.json) and\n* [embeddings calculation](https://raw.githubusercontent.com/rozek/node-red-flow-stablelm-3b-4e1t/master/StableLM-3B-4E1T-Embeddings-Function.json)\n\nthis repository also contains example flows which answer incoming HTTP requests. The prompt itself and any desired parameters have to be passed as query parameters, the result of the called function will then be returned in the body of the HTTP response.\n\nFor inferencing, the following parameters are supported (most of them will be copied into a `msg` property of the same name):\n\n* `prompt` - will be copied into `msg.payload`\n* `seed` - will be copied into `msg.seed`\n* `threads` - will be copied into `msg.threads`\n* `context` - will be copied into `msg.context`\n* `keep` - will be copied into `msg.keep`\n* `predict` - will be copied into `msg.predict`\n* `topk` - will be copied into `msg.topk`\n* `topp` - will be copied into `msg.topp`\n* `temperature` - will be copied into `msg.temperature`\n* `batches` - will be copied into `msg.batches`\n* `grammar` - will be copied into `msg.grammar`\n\nTokenization and embeddings calculation endpoints support a subset of these parameters - as required by their respective function nodes.\n\n## Examples ##\n\nIf you have [cURL](https://curl.se/) installed (if not - but you want it - just [follow the instructions found in their docs](https://everything.curl.dev/get)) (and assuming that your Node-RED installation is listening at port 1880) you may use the following commands to \"smoke test\" the imported flows:\n\n### Text Completion ###\n\n```\ncurl \"http://127.0.0.1:1880/stablelm?prompt=who%20was%20Joseph%20Weizenbaum%3F\"\n```\n\n### Tokenization ###\n\n```\ncurl \"http://127.0.0.1:1880/stablelm-tokenization?prompt=who%20was%20Joseph%20Weizenbaum%3F\"\n```\n\n### Embeddings Calculation ###\n\n```\ncurl \"http://127.0.0.1:1880/stablelm-embeddings?prompt=who%20was%20Joseph%20Weizenbaum%3F\"\n```\n\n## License ##\n\n[MIT License](LICENSE.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frozek%2Fnode-red-flow-stablelm-3b-4e1t","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frozek%2Fnode-red-flow-stablelm-3b-4e1t","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frozek%2Fnode-red-flow-stablelm-3b-4e1t/lists"}