{"id":50718189,"url":"https://github.com/SearchSavior/OpenArc","last_synced_at":"2026-06-26T22:00:48.303Z","repository":{"id":278182900,"uuid":"930733499","full_name":"SearchSavior/OpenArc","owner":"SearchSavior","description":"Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.","archived":false,"fork":false,"pushed_at":"2026-06-26T05:12:19.000Z","size":256126,"stargazers_count":471,"open_issues_count":12,"forks_count":40,"subscribers_count":16,"default_branch":"main","last_synced_at":"2026-06-26T07:09:29.689Z","etag":null,"topics":["agentic-ai","fastapi","inference-engine","openvino-genai","openvino-toolkit","optimum-intel","transformers"],"latest_commit_sha":null,"homepage":"https://searchsavior.github.io/OpenArc/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SearchSavior.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-02-11T05:46:58.000Z","updated_at":"2026-06-26T05:12:24.000Z","dependencies_parsed_at":"2025-02-18T13:22:07.957Z","dependency_job_id":"f3807b24-faf0-4470-8286-d86de61d9306","html_url":"https://github.com/SearchSavior/OpenArc","commit_stats":null,"previous_names":["searchsavior/openarc"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/SearchSavior/OpenArc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SearchSavior%2FOpenArc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SearchSavior%2FOpenArc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SearchSavior%2FOpenArc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SearchSavior%2FOpenArc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SearchSavior","download_url":"https://codeload.github.com/SearchSavior/OpenArc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SearchSavior%2FOpenArc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34834415,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-26T02:00:06.560Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic-ai","fastapi","inference-engine","openvino-genai","openvino-toolkit","optimum-intel","transformers"],"created_at":"2026-06-09T21:00:25.963Z","updated_at":"2026-06-26T22:00:48.289Z","avatar_url":"https://github.com/SearchSavior.png","language":"Python","funding_links":[],"categories":["*Ops for AI"],"sub_categories":["Model Serving \u0026 Inference"],"readme":"![openarc_DOOM](assets/openarc_DOOM.png)\n\n[![Discord](https://img.shields.io/discord/1341627368581628004?logo=Discord\u0026logoColor=%23ffffff\u0026label=Discord\u0026link=https%3A%2F%2Fdiscord.gg%2FmaMY7QjG)](https://discord.gg/Bzz9hax9Jq)\n[![Hugging Face](https://img.shields.io/badge/🤗%20Hugging%20Face-Echo9Zulu-yellow)](https://huggingface.co/Echo9Zulu)\n[![Devices](https://img.shields.io/badge/Devices-CPU%2FGPU%2FNPU-blue)](https://github.com/openvinotoolkit/openvino)\n[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/SearchSavior/OpenArc)\n[![Documentation](https://img.shields.io/badge/📖%20Documentation-blue)](https://searchsavior.github.io/OpenArc/)\n\n\u003e [!NOTE]\n\u003e OpenArc is under active development.\n\n**OpenArc** is an inference engine for Intel devices. \n\nServe LLMs, VLMs, Whisper, Kokoro-TTS, Qwen-TTS, Qwen-ASR, Embedding and Reranker models over OpenAI compatible endpoints, powered by OpenVINO on your device. Local, private, open source AI.  \n\nOpenArc is a community-driven effort to make acceleration from OpenVINO easier to access, deploy and leverage for our usecases.\n\nIf you are interested in using Intel devices for AI and machine learning, feel free to stop by our [Discord](https://discord.gg/y9hgrtjbcD), where we are tracking almost the whole stack, including development of llama.cpp SYCL backend. \n\nThanks to everyone on Discord for their continued support!\n\n\u003e [!NOTE]\n\u003e Documentation lives [here](https://searchsavior.github.io/OpenArc/)\n\n\n## Quickstart\n\n- [Linux](https://searchsavior.github.io/OpenArc/install/#linux)\n- [Windows](https://searchsavior.github.io/OpenArc/install/#windows)\n- [Docker](https://searchsavior.github.io/OpenArc/install/#docker)\n\n## Features\n\n  - NEW! Containerization with Docker #60 by @meatposes\n  - NEW! Speculative decoding support for LLMs #57 by @meatposes\n  - NEW! Streaming cancellation support for LLMs and VLMs\n  - Multi GPU Pipeline Paralell\n  - CPU offload/Hybrid device\n  - NPU device support\n  - OpenAI compatible endpoints\n      - `/v1/models`\n      - `/v1/completions`: `llm` only\n      - `/v1/chat/completions`\n      - `/v1/audio/transcriptions`: `whisper`, `qwen3_asr`\n      - `/v1/audio/speech`: `kokoro` only       \n      - `/v1/embeddings`: `qwen3-embedding` #33 by @mwrothbe\n      - `/v1/rerank`: `qwen3-reranker` #39 by @mwrothbe\n  - `jinja` templating with `AutoTokenizers`\n  - OpenAI Compatible tool calls with streaming and paralell \n    - tool call parser currently reads \"name\", \"argument\" \n  - Fully async multi engine, multi task architecture\n  - Model concurrency: load and infer multiple models at once\n  - Automatic unload on inference failure\n  - `llama-bench` style benchmarking for `llm` w/automatic sqlite database\n  - metrics on every request\n    - ttft\n    - prefill_throughput\n    - decode_throughput\n    - decode_duration\n    - tpot\n    - load time\n    - stream mode\n  - More OpenVINO [examples](examples/)\n  - OpenVINO implementation of [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M)\n  - OpenVINO implementation of Qwen3-TTS and Qwen3-ASR\n  \n\n\u003e [!NOTE] \n\u003e Interested in contributing? Please open an issue before submitting a PR!\n\n\n## Acknowledgments\n\nOpenArc stands on the shoulders of many other projects:\n\n[Optimum-Intel](https://github.com/huggingface/optimum-intel)\n\n[OpenVINO](https://github.com/openvinotoolkit/openvino)\n\n[OpenVINO GenAI](https://github.com/openvinotoolkit/openvino.genai)\n\n[llama.cpp](https://github.com/ggml-org/llama.cpp)\n\n[vLLM](https://github.com/vllm-project/vllm)\n\n[Transformers](https://github.com/huggingface/transformers)\n\n[FastAPI](https://github.com/fastapi/fastapi)\n\n[click](https://github.com/pallets/click)\n\n[rich-click](https://github.com/ewels/rich-click)\n\n```\n@article{zhou2024survey,\n  title={A Survey on Efficient Inference for Large Language Models},\n  author={Zhou, Zixuan and Ning, Xuefei and Hong, Ke and Fu, Tianyu and Xu, Jiaming and Li, Shiyao and Lou, Yuming and Wang, Luning and Yuan, Zhihang and Li, Xiuhong and Yan, Shengen and Dai, Guohao and Zhang, Xiao-Ping and Dong, Yuhan and Wang, Yu},\n  journal={arXiv preprint arXiv:2404.14294},\n  year={2024}\n}\n```\nThanks for your work!!\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSearchSavior%2FOpenArc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSearchSavior%2FOpenArc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSearchSavior%2FOpenArc/lists"}