{"id":13456957,"url":"https://github.com/edgenai/edgen","last_synced_at":"2025-04-05T18:09:38.432Z","repository":{"id":219994931,"uuid":"750465456","full_name":"edgenai/edgen","owner":"edgenai","description":"⚡  Edgen: Local, private GenAI server alternative to OpenAI. No GPU required. Run AI models locally: LLMs (Llama2, Mistral, Mixtral...), Speech-to-text (whisper) and many others.","archived":false,"fork":false,"pushed_at":"2024-05-23T14:21:38.000Z","size":2481,"stargazers_count":357,"open_issues_count":25,"forks_count":16,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-29T17:08:47.349Z","etag":null,"topics":["chatgpt","edge","genai","llm","localai","ml","openai","rust","tauri","vertex-ai","watson"],"latest_commit_sha":null,"homepage":"https://docs.edgen.co/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/edgenai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-30T17:40:42.000Z","updated_at":"2025-03-28T17:58:31.000Z","dependencies_parsed_at":"2024-04-15T11:18:41.311Z","dependency_job_id":"97db73bb-0b9e-4677-be76-6999b73b830f","html_url":"https://github.com/edgenai/edgen","commit_stats":{"total_commits":242,"total_committers":10,"mean_commits":24.2,"dds":0.6363636363636364,"last_synced_commit":"dbdc84ac491eb7cf92f99e4750ab2c8714647f5b"},"previous_names":["edgenai/edgen"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgenai%2Fedgen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgenai%2Fedgen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgenai%2Fedgen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgenai%2Fedgen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/edgenai","download_url":"https://codeload.github.com/edgenai/edgen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247378149,"owners_count":20929297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","edge","genai","llm","localai","ml","openai","rust","tauri","vertex-ai","watson"],"created_at":"2024-07-31T08:01:30.849Z","updated_at":"2025-04-05T18:09:38.408Z","avatar_url":"https://github.com/edgenai.png","language":"Rust","funding_links":[],"categories":["Rust","Machine Learning","rust","chatgpt"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003cimg alt=\"GitHub\" src=\"https://img.shields.io/github/license/edgenai/edgen\"\u003e\n    \u003c!-- TODO: uncomment to show discord --\u003e\n    \u003c!-- \u003cimg alt=\"Discord\" src=\"https://img.shields.io/discord/1163068604074426408?logo=discord\u0026label=Discord\u0026link=https%3A%2F%2Fdiscord.gg%2FMMUcgBtV\"\u003e --\u003e\n\u003c/p\u003e\n\n\u003ch3 align=\"center\"\u003e\n    A Local GenAI API Server: A drop-in replacement for OpenAI's API for Local GenAI\n\u003c/h3\u003e\n\n\u003cp align=\"center\"\u003e\n    |\n    \u003c!-- TODO: add proper links --\u003e\n    \u003ca href=\"https://docs.edgen.co\"\u003e\u003cb\u003eDocumentation\u003c/b\u003e\u003c/a\u003e |\n    \u003ca href=\"https://blog.edgen.co\"\u003e\u003cb\u003eBlog\u003c/b\u003e\u003c/a\u003e |\n    \u003ca href=\"https://discord.gg/QUXbwqdMRs\"\u003e\u003cb\u003eDiscord\u003c/b\u003e\u003c/a\u003e |\n    \u003ca href=\"https://github.com/orgs/edgenai/projects/1/views/1\"\u003e\u003cb\u003eRoadmap\u003c/b\u003e\u003c/a\u003e |\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://edgen.co/images/demo.gif\" alt=\"EdgenChat, a local chat app powered by ⚡Edgen\"\u003e\n    \u003cp align=\"center\"\u003e\n        \u003ca href=\"https://chat.edgen.co\"\u003eEdgenChat\u003c/a\u003e, a local chat app powered by ⚡Edgen\n    \u003c/p\u003e\n\u003c/div\u003e\n\n- [x] **OpenAI Compliant API**: ⚡Edgen implements an [OpenAI compatible API](https://docs.edgen.co/api-reference), making it a drop-in replacement.\n- [x] **Multi-Endpoint Support**: ⚡Edgen exposes multiple AI endpoints such as chat completions (LLMs) and speech-to-text (Whisper) for audio transcriptions.\n- [x] **Model Agnostic**: LLMs (Llama2, Mistral, Mixtral...), Speech-to-text (whisper) and [many others](https://docs.edgen.co/documentation/models).\n- [x] **Optimized Inference**: You don't need to take a PhD in AI optimization. ⚡Edgen abstracts the complexity of optimizing inference for different hardware, platforms and models.\n- [x] **Modular**: ⚡Edgen is **model** and **runtime** agnostic. New models can be added easily and ⚡Edgen can select the best runtime for the user's hardware: you don't need to keep up about the latest models and ML runtimes - **⚡Edgen will do that for you**.\n- [x] **Model Caching**: ⚡Edgen caches foundational models locally, so 1 model can power hundreds of different apps - users don't need to download the same model multiple times.\n- [x] **Native**: ⚡Edgen is built in 🦀Rust and is natively compiled to all popular platforms: **Windows, MacOS and Linux**. No docker required.\n- [ ] **Graphical Interface**: A graphical user interface to help users efficiently manage their models, endpoints and permissions.\n\n⚡Edgen lets you use GenAI in your app, completely **locally** on your user's devices, for **free** and with **data-privacy**. It's a drop-in replacement for OpenAI (it uses the a compatible API), supports various functions like text generation, speech-to-text and works on Windows, Linux, and MacOS.\n\n### Features\n\n- [x] Session Caching: ⚡Edgen maintains top performance with big contexts (big chat histories), by caching sessions. Sessions are auto-detected in function of the chat history.\n- [x] [GPU support](https://github.com/edgenai/edgen#gpu-support): CUDA, Vulkan. Metal\n\n### Endpoints\n\n- [x] \\[Chat\\] [Completions](https://docs.edgen.co/api-reference/chat)\n- [x] \\[Audio\\] [Transcriptions](https://docs.edgen.co/api-reference/audio)\n- [x] \\[Embeddings\\] [Embeddings](https://platform.openai.com/docs/api-reference/embeddings)\n- [ ] \\[Image\\] Generation\n- [ ] \\[Chat\\] Multimodal chat completions\n- [ ] \\[Audio\\] Speech\n\n### Supported Models\n\nCheck in the [documentation](https://docs.edgen.co/documentation/models)\n\n### Supported platforms\n\n- [x] Windows\n- [x] Linux\n- [x] MacOS\n\n## 🔥 Hot Topics\n\n## Why local GenAI?\n\n- **Data Private**: On-device inference means **users' data** never leave their devices.\n\n- **Scalable**: More and more users? No need to increment cloud computing infrastructure. Just let your users use their own hardware.\n\n- **Reliable**: No internet, no downtime, no rate limits, no API keys.\n\n- **Free**: It runs locally on hardware the user already owns.\n\n## Quickstart\n\n1. [Download](https://edgen.co/download) and start ⚡Edgen\n2. Chat with ⚡[EdgenChat](https://chat.edgen.co)\n\nReady to start your own GenAI application? [Checkout our guides](https://docs.edgen.co/guides)!\n\n⚡Edgen usage:\n\n```\nUsage: edgen [\u003ccommand\u003e] [\u003cargs\u003e]\n\nToplevel CLI commands and options. Subcommands are optional. If no command is provided \"serve\" will be invoked with default options.\n\nOptions:\n  --help            display usage information\n\nCommands:\n  serve             Starts the edgen server. This is the default command when no\n                    command is provided.\n  config            Configuration-related subcommands.\n  version           Prints the edgen version to stdout.\n  oasgen            Generates the Edgen OpenAPI specification.\n```\n\n`edgen serve` usage:\n\n```\nUsage: edgen serve [-b \u003curi...\u003e] [-g]\n\nStarts the edgen server. This is the default command when no command is provided.\n\nOptions:\n  -b, --uri         if present, one or more URIs/hosts to bind the server to.\n                    `unix://` (on Linux), `http://`, and `ws://` are supported.\n                    For use in scripts, it is recommended to explicitly add this\n                    option to make your scripts future-proof.\n  -g, --nogui       if present, edgen will not start the GUI; the default\n                    behavior is to start the GUI.\n  --help            display usage information\n```\n\n## GPU Support\n\n⚡Edgen also supports compilation and execution on a GPU, when building from source, through Vulkan, CUDA and Metal.\nThe following cargo features enable the GPU:\n\n- `llama_vulkan` - execute LLM models using Vulkan. Requires a Vulkan SDK to be installed.\n- `llama_cuda` - execute LLM models using CUDA. Requires a CUDA Toolkit to be installed.\n- `llama_metal` - execute LLM models using Metal.\n- `whisper_cuda` - execute Whisper models using CUDA. Requires a CUDA Toolkit to be installed.\n\nNote that, at the moment, `llama_vulkan`, `llama_cuda` and `llama_metal` cannot be enabled at the same time.\n\nExample usage (building from source, [you need to first install the prerequisites](https://docs.edgen.co/documentation/getting-started)):\n\n```\ncargo run --features llama_vulkan --release -- serve\n```\n\n## Architecture Overview\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"docs/assets/edgen_architecture_overview.svg\" alt=\"⚡Edgen architecture overview\" width=\"400\"\u003e\n    \u003cp align=\"center\"\u003e⚡Edgen architecture overview\u003c/p\u003e\n\u003c/div\u003e\n\n## Contribute\n\nIf you don't know where to start, check [Edgen's roadmap](https://github.com/orgs/edgenai/projects/1/views/1)!\nBefore you start working on something, see if there's an existing issue/pull-request. Pop into Discord to check with the team or see if someone's already tackling it.\n\n## Communication Channels\n\n- [Edgen Discord server](https://discord.gg/QUXbwqdMRs): Real time discussions with the ⚡Edgen team and other users.\n- [GitHub issues](https://github.com/edgenai/edgen/issues): Feature requests, bugs.\n- [GitHub discussions](https://github.com/edgenai/edgen/discussions/): Q\u0026A.\n- [Blog](https://blog.edgen.co): Big announcements.\n\n## Special Thanks\n\n- [`llama.cpp`](https://github.com/ggerganov/llama.cpp/tree/master),\n  [`whisper.cpp`](https://github.com/ggerganov/whisper.cpp), and [`ggml`](https://github.com/ggerganov/ggml) for being\n  an excellent getting-on point for this space.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedgenai%2Fedgen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fedgenai%2Fedgen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedgenai%2Fedgen/lists"}