{"id":13910943,"url":"https://github.com/llm-edge/hal-9100","last_synced_at":"2025-07-18T10:32:00.681Z","repository":{"id":209101915,"uuid":"719119076","full_name":"llm-edge/hal-9100","owner":"llm-edge","description":"Edge full-stack LLM platform. Written in Rust","archived":false,"fork":false,"pushed_at":"2024-05-25T09:42:43.000Z","size":1661,"stargazers_count":380,"open_issues_count":35,"forks_count":31,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-07-04T13:31:05.148Z","etag":null,"topics":["ai","deep-learning","edge","large-language-models","llm","machine-learning"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/llm-edge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-15T13:50:19.000Z","updated_at":"2025-06-06T01:43:19.000Z","dependencies_parsed_at":"2024-04-17T02:17:57.446Z","dependency_job_id":"96052d12-6e48-4ffc-9b72-2e1c1fd1c0bb","html_url":"https://github.com/llm-edge/hal-9100","commit_stats":null,"previous_names":["stellar-amenities/assistants","stellar-amenities/hal-9100"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/llm-edge/hal-9100","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llm-edge%2Fhal-9100","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llm-edge%2Fhal-9100/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llm-edge%2Fhal-9100/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llm-edge%2Fhal-9100/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/llm-edge","download_url":"https://codeload.github.com/llm-edge/hal-9100/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llm-edge%2Fhal-9100/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265742309,"owners_count":23820825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","deep-learning","edge","large-language-models","llm","machine-learning"],"created_at":"2024-08-07T00:01:51.130Z","updated_at":"2025-07-18T10:32:00.137Z","avatar_url":"https://github.com/llm-edge.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"⚠️ README outdated ([undergoing large refactor](https://github.com/llm-edge/hal-9100/tree/0.1)) ⚠️\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg width=\"600\" alt=\"hal-9100\" src=\"https://github.com/llm-edge/hal-9100/assets/25003283/17c3792e-f191-48d7-9c77-7f39d8f94912\"\u003e\n  \u003ch1 align=\"center\"\u003e🤖 HAL-9100\u003c/h1\u003e\n  \u003ch2 align=\"center\"\u003eBuild AI Assistants that don't need internet. Using OpenAI SDK. For production.\u003c/h2\u003e\n  \u003ch4 align=\"center\"\u003e100% Private, 75% Cheaper \u0026 23x Faster Assistants.\u003c/h4\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href='https://codespaces.new/llm-edge/hal-9100?quickstart=1'\u003e\u003cimg src='https://github.com/codespaces/badge.svg' alt='Open in GitHub Codespaces' style='max-width: 100%;'\u003e\u003c/a\u003e\n    \u003cbr /\u003e\n    \u003ca href=\"https://discord.gg/pj5VRqqs84\"\u003e\u003cimg alt=\"Join Discord\" src=\"https://img.shields.io/discord/1066022656845025310?color=blue\u0026style=for-the-badge\"\u003e\u003c/a\u003e\n  \u003c/p\u003e\n\n\n\u003c/p\u003e\n\n\n-----\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://link.excalidraw.com/readonly/YSE7DNzB2LmEPfVdCqq3\"\u003e🖼️ Infra\u003c/a\u003e\n    \u003ca href=\"https://github.com/llm-edge/hal-9100/issues/new?assignees=\u0026labels=enhancement\"\u003e✨ Feature?\u003c/a\u003e\n    \u003ca href=\"https://github.com/llm-edge/hal-9100/issues/new?assignees=\u0026labels=bug\"\u003e❤️‍🩹 Bug?\u003c/a\u003e\n    \u003ca href=\"https://cal.com/louis030195/applied-ai\"\u003e📞 Help?\u003c/a\u003e\n\u003c/p\u003e\n\n-----\n\n\u003c!--\n# ⭐️ Latest News\n\n- [2024/01/19] 🔥 Added usage w ollama. Keep reading 👇.\n- [2024/01/19] 🔥 Action tool. [Let your Assistant make requests to APIs](https://github.com/llm-edge/hal-9100/tree/main/examples/hello-world-mlc-llm-mistral-nodejs-action).\n- [2023/12/19] 🔥 New example: Open source LLM with code interpreter. [Learn more](./examples/hello-world-code-interpreter-mixtral-nodejs/README.md).\n- [2023/12/08] 🔥 New example: Open source LLM with function calling. [Learn more](./examples/hello-world-intel-neural-chat-nodejs-function-calling/README.md).\n- [2023/11/29] 🔥 New example: Using mistral-7b, an open source LLM. [Check it out](./examples/hello-world-mistral-curl/README.md).\n--\u003e\n# ✨ Key Features\n- [x] **Code Interpreter**: Generate and runs Python code in a sandboxed environment autonomously. (beta)\n- [x] **Knowledge Retrieval**: Retrieves external knowledge or documents autonomously.\n- [x] **Function Calling**: Defines and executes custom functions autonomously.\n- [x] **Actions**: Execute requests to external APIs autonomously.\n- [x] **Files**: Supports a range of file formats.\n- [x] **OpenAI compatible**: Works with OpenAI (Assistants) SDK\n\u003c!--\n- [x] **Enterprise production-ready**: \n  - [x] observability (metrics, errors, traces, logs, etc.)\n  - [x] scalability (serverless, caching, autoscaling, etc.)\n  - [x] security (encryption, authentication, authorization, etc.)\n--\u003e\n# 😃 Who is it for?\n\n\u003cimg width=\"800\" alt=\"hal-9100-2\" src=\"https://github.com/llm-edge/hal-9100/assets/25003283/5a393d61-7a1d-4e06-8932-f822b18015ba\"\u003e\n\n- You want to increase customization (e.g. use your own models, extend the API, etc.)\n- You work in a data-sensitive environment (healthcare, IoT, military, law, etc.)\n- Your product does have poor or no internet access (military, IoT, edge, extreme environment, etc.)\n- (not our main focus) You operate on a large scale and want to reduce your costs\n- (not our main focus) You operate on a large scale and want to increase your speed\n\n# 🤖 Our definition of Software 3.0\n\nFirst, our definition of **Software 3.0**, as it is a loaded term:\nSoftware 3.0 is the bridge connecting the cognitive capabilities of Large Language Models with the practical needs of human digital activity. It is a comprehensive approach that allows LLMs to:\n1. perform the same activity (or better) on the digital world than humans\n2. generally, allow the user to [perform more operations without conscious effort](https://third.software/)\n\n# 📏 Principles\n\nHAL-9100 is in continuous development, with the aim of always offering better infrastructure for **Edge Software 3.0**. To achieve this, it is based on several principles that define its functionality and scope.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eLess prompt is more\u003c/strong\u003e\u003c/summary\u003e\n\u003cp\u003e\n\nAs few prompts as possible should be hard-coded into the infrastructure, just enough to bridge the gap between **Software 1.0** and **Software 3.0** and give the client as much control as possible on the prompts.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eEdge-first\u003c/strong\u003e\u003c/summary\u003e\n\u003cp\u003e\n\nHAL-9100 does not require internet access by focusing on **open source LLMs**. Which means you own your data and your models. It runs on a Raspberry PI (LLM included).\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eOpenAI-compatible\u003c/strong\u003e\u003c/summary\u003e\n\u003cp\u003e\n\nOpenAI spent a large amount of the best brain power to design this API, which makes it an incredible experience for developers. Support for OpenAI LLMs are not a priority at all though.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eReliable and deterministic\u003c/strong\u003e\u003c/summary\u003e\n\u003cp\u003e\n\nHAL-9100 focus on reliability and being as deterministic as possible by default. That's why everything has to be tested and benchmarked.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eFlexible\u003c/strong\u003e\u003c/summary\u003e\n\u003cp\u003e\n\nA minimal number of hard-coded prompts and behaviors, a wide range of models, infrastructure components and deployment options and it play well with the open-source ecosystem, while only integrating projects that have stood the test of time.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n# Quickstart\n\nGet started in less than a minute through GitHub Codespaces:\n\n[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/llm-edge/hal-9100?quickstart=1)\n\nOr:\n\n```bash\ngit clone https://github.com/llm-edge/hal-9100\ncd hal-9100\n```\n\nTo get started quickly, let's use Anyscale API.\nGet an API key from Anyscale. You can get it [here](https://app.endpoints.anyscale.com/credentials). Replace in [hal-9100.toml](./hal-9100.toml) the `model_api_key` with your API key.\n\n\u003cdetails\u003e\n\u003csummary\u003eUsage w/ ollama\u003c/summary\u003e\n\u003cp\u003e\n\n1. use `model_url = \"http://localhost:11434/v1/chat/completions\"`\n2. set `gemma:2b` in [examples/quickstart.js](./examples/quickstart.js)\n3. and run `ollama run gemma:2b \u0026 \u0026\u0026 docker compose --profile api -f docker/docker-compose.yml up`\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\nInstall OpenAI SDK: `npm i openai`\n\nStart the infra:\n\n```bash\ndocker compose --profile api -f docker/docker-compose.yml up\n```\n\nRun the [quickstart](./examples/quickstart.js):\n\n```bash\nnode examples/quickstart.js\n```\n\n# 🤔 FAQ\n\n\u003cdetails\u003e\n\u003csummary\u003eIs there a hosted version?\u003c/summary\u003e\n\nNo. HAL-9100 is not a hosted service. It's a software that you can deploy on your infrastructure. We can help you deploy it on your infrastructure. [Contact us](https://cal.com/louis030195/applied-ai).\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eWhich LLM API can I use?\u003c/summary\u003e\n\n\nExamples of LLM APIs that does support OpenAI API-like, that you can use:\n- ollama\n- [MLC-LLM](https://github.com/mlc-ai/mlc-llm)\n- [FastChat (good if you have a mac)](https://github.com/llm-edge/hal-9100/tree/main/examples/hello-world-mistral-curl)\n- [vLLM (good if you have a modern gpu)](https://docs.vllm.ai/en/latest/getting_started/quickstart.html#openai-compatible-server)\n- [Perplexity API](https://github.com/llm-edge/hal-9100/tree/main/examples/hello-world-code-interpreter-mixtral-nodejs)\n- Mistral API\n- anyscale\n- together ai\n\nWe recommend these models:\n- mistralai/Mixtral-8x7B-Instruct-v0.1\n- mistralai/mistral-7b\n\nOther models have not been extensively tested and may not work as expected, but you can try them.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eWhat's the difference with LangChain?\u003c/summary\u003e\n1. LangChain spans proprietary LLM and open source, among the thousands of things it spans. HAL-9100 laser focuses on Software 3.0 for the edge. \n\n\n2. You can write AI products in 50 lines of code instead of 5000 and having to learn a whole new abstraction\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eAre you related to OpenAI?\u003c/summary\u003e\nNo.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eI don't use Assistants API. Can I use this?\u003c/summary\u003e\nWe recommend switching to the Assistants API for a more streamlined experience, allowing you to focus more on your product than on infrastructure.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eDoes the Assistants API support audio and images?\u003c/summary\u003e\nImages soon, working on it.\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllm-edge%2Fhal-9100","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fllm-edge%2Fhal-9100","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllm-edge%2Fhal-9100/lists"}