{"id":17769332,"url":"https://github.com/theroyallab/tabbyAPI","last_synced_at":"2025-03-15T14:30:39.002Z","repository":{"id":207656042,"uuid":"716906582","full_name":"theroyallab/tabbyAPI","owner":"theroyallab","description":"An OAI compatible exllamav2 API that's both lightweight and fast","archived":false,"fork":false,"pushed_at":"2025-02-02T17:23:11.000Z","size":877,"stargazers_count":767,"open_issues_count":19,"forks_count":93,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-02-06T21:34:00.537Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/theroyallab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"ko_fi":"kingbri"}},"created_at":"2023-11-10T05:54:02.000Z","updated_at":"2025-02-06T05:50:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"5a8298f9-228e-43e7-850e-3c1c0c9f6d0f","html_url":"https://github.com/theroyallab/tabbyAPI","commit_stats":null,"previous_names":["theroyallab/tabbyapi"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theroyallab%2FtabbyAPI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theroyallab%2FtabbyAPI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theroyallab%2FtabbyAPI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theroyallab%2FtabbyAPI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/theroyallab","download_url":"https://codeload.github.com/theroyallab/tabbyAPI/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243742591,"owners_count":20340666,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-26T21:01:19.092Z","updated_at":"2025-03-15T14:30:38.995Z","avatar_url":"https://github.com/theroyallab.png","language":"Python","funding_links":["https://ko-fi.com/kingbri","https://ko-fi.com/I2I3BDTSW"],"categories":["\u003cimg src=\"./assets/cpu.svg\" width=\"16\" height=\"16\" style=\"vertical-align: middle;\"\u003e Backends","Python","Inference Runtimes \u0026 Backends","3. Inference Engines \u0026 Serving","Browse The Shelves"],"sub_categories":["Local LLM developer tools"],"readme":"# TabbyAPI\n\n\u003cp align=\"left\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Python-3.10%20|%203.11%20|%203.12-blue\" alt=\"Python 3.10, 3.11, and 3.12\"\u003e\n    \u003ca href=\"/LICENSE\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/License-AGPLv3-blue.svg\" alt=\"License: AGPL v3\"/\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://discord.gg/sYQxnuD7Fj\"\u003e\n        \u003cimg src=\"https://img.shields.io/discord/545740643247456267.svg?logo=discord\u0026color=blue\" alt=\"Discord Server\"/\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"left\"\u003e\n    \u003ca href=\"https://theroyallab.github.io/tabbyAPI\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Documentation-API-orange\" alt=\"Developer facing API documentation\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"left\"\u003e\n    \u003ca href=\"https://ko-fi.com/I2I3BDTSW\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Support_on_Ko--fi-FF5E5B?logo=ko-fi\u0026style=for-the-badge\u0026logoColor=white\" alt=\"Support on Ko-Fi\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003e [!IMPORTANT]\n\u003e\n\u003e  In addition to the README, please read the [Wiki](https://github.com/theroyallab/tabbyAPI/wiki/1.-Getting-Started) page for information about getting started!\n\n\u003e [!NOTE]\n\u003e \n\u003e  Need help? Join the [Discord Server](https://discord.gg/sYQxnuD7Fj) and get the `Tabby` role. Please be nice when asking questions.\n\n\u003e [!NOTE]\n\u003e \n\u003e Want to run GGUF models? Take a look at [YALS](https://github.com/theroyallab/YALS), TabbyAPI's sister project.\n\nA FastAPI based application that allows for generating text using an LLM (large language model) using the [Exllamav2 backend](https://github.com/turboderp/exllamav2)\n\nTabbyAPI is also the official API backend server for ExllamaV2.\n\n## Disclaimer\n\nThis project is marked as rolling release. There may be bugs and changes down the line. Please be aware that you might need to reinstall dependencies if needed.\n\nTabbyAPI is a hobby project made for a small amount of users. It is not meant to run on production servers. For that, please look at other solutions that support those workloads.\n\n## Getting Started\n\n\u003e [!IMPORTANT]\n\u003e \n\u003e Looking for more information? Check out the Wiki.\n\nFor a step-by-step guide, choose the format that works best for you:\n\n📖 Read the [Wiki](https://github.com/theroyallab/tabbyAPI/wiki/1.-Getting-Started) – Covers installation, configuration, API usage, and more.\n\n🎥 Watch the [Video Guide](https://www.youtube.com/watch?v=03jYz0ijbUU) – A hands-on walkthrough to get you up and running quickly.\n\n## Features\n\n- OpenAI compatible API\n- Loading/unloading models\n- HuggingFace model downloading\n- Embedding model support\n- JSON schema + Regex + EBNF support\n- AI Horde support\n- Speculative decoding via draft models\n- Multi-lora with independent scaling (ex. a weight of 0.9)\n- Inbuilt proxy to override client request parameters/samplers\n- Flexible Jinja2 template engine for chat completions that conforms to HuggingFace\n- Concurrent inference with asyncio\n- Utilizes modern python paradigms\n- Continuous batching engine using paged attention\n- Fast classifier-free guidance\n- OAI style tool/function calling\n\nAnd much more. If something is missing here, PR it in!\n\n## Supported Model Types\n\nTabbyAPI uses Exllamav2 as a powerful and fast backend for model inference, loading, etc. Therefore, the following types of models are supported:\n\n- Exl2 (Highly recommended)\n\n- GPTQ\n\n- FP16 (using Exllamav2's loader)\n\nIn addition, TabbyAPI supports parallel batching using paged attention for Nvidia Ampere GPUs and higher.\n\n## Contributing\n\nUse the template when creating issues or pull requests, otherwise the developers may not look at your post.\n\nIf you have issues with the project:\n\n- Describe the issue in detail\n\n- If you have a feature request, please indicate it as such.\n\nIf you have a Pull Request\n\n- Describe the pull request in detail, what, and why you are changing something\n\n## Acknowldgements\n\nTabbyAPI would not exist without the work of other contributors and FOSS projects:\n\n- [ExllamaV2](https://github.com/turboderp/exllamav2)\n- [Aphrodite Engine](https://github.com/PygmalionAI/Aphrodite-engine)\n- [infinity-emb](https://github.com/michaelfeil/infinity)\n- [FastAPI](https://github.com/fastapi/fastapi)\n- [Text Generation WebUI](https://github.com/oobabooga/text-generation-webui)\n- [SillyTavern](https://github.com/SillyTavern/SillyTavern)\n\n## Developers and Permissions\n\nCreators/Developers:\n\n- [kingbri](https://github.com/bdashore3)\n\n- [Splice86](https://github.com/Splice86)\n\n- [Turboderp](https://github.com/turboderp)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheroyallab%2FtabbyAPI","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftheroyallab%2FtabbyAPI","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheroyallab%2FtabbyAPI/lists"}