{"id":13422958,"url":"https://github.com/hyperonym/basaran","last_synced_at":"2025-09-30T20:31:45.268Z","repository":{"id":118309418,"uuid":"603131458","full_name":"hyperonym/basaran","owner":"hyperonym","description":"Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.","archived":true,"fork":false,"pushed_at":"2024-01-24T05:50:55.000Z","size":5634,"stargazers_count":1290,"open_issues_count":37,"forks_count":80,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-10-29T10:06:23.873Z","etag":null,"topics":["generative","gpt","huggingface","language-model","llama","llm","model","natural-language-processing","nlp","openai-api","python","text-generation","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hyperonym.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["hyperonym"]}},"created_at":"2023-02-17T17:22:29.000Z","updated_at":"2024-10-27T16:27:44.000Z","dependencies_parsed_at":"2024-01-13T03:45:31.229Z","dependency_job_id":"546b25f8-3da2-4569-bec2-9b600502a95a","html_url":"https://github.com/hyperonym/basaran","commit_stats":null,"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperonym%2Fbasaran","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperonym%2Fbasaran/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperonym%2Fbasaran/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperonym%2Fbasaran/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hyperonym","download_url":"https://codeload.github.com/hyperonym/basaran/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234774942,"owners_count":18884527,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generative","gpt","huggingface","language-model","llama","llm","model","natural-language-processing","nlp","openai-api","python","text-generation","transformers"],"created_at":"2024-07-30T23:01:00.656Z","updated_at":"2025-09-30T20:31:39.697Z","avatar_url":"https://github.com/hyperonym.png","language":"Python","readme":"# Basaran\n\n[![Python](https://github.com/hyperonym/basaran/actions/workflows/python.yml/badge.svg)](https://github.com/hyperonym/basaran/actions/workflows/python.yml)\n[![codecov](https://codecov.io/gh/hyperonym/basaran/branch/master/graph/badge.svg?token=8HUSH6HSAN)](https://codecov.io/gh/hyperonym/basaran)\n[![PyPI](https://img.shields.io/pypi/v/basaran)](https://pypi.org/project/basaran/)\n[![Status](https://img.shields.io/badge/status-beta-blue)](https://github.com/hyperonym/basaran)\n\nBasaran is an open-source alternative to the [OpenAI text completion API](https://platform.openai.com/docs/api-reference/completions/create). It provides a compatible streaming API for your [Hugging Face Transformers](https://huggingface.co/docs/transformers/index)-based [text generation models](https://huggingface.co/models?pipeline_tag=text-generation).\n\nThe open source community will eventually witness the [Stable Diffusion](https://stability.ai/blog/stable-diffusion-public-release) moment for large language models (LLMs), and Basaran allows you to replace OpenAI's service with the latest open-source model to power your application [without modifying a single line of code](https://github.com/hyperonym/basaran/blob/master/README.md#openai-client-library).\n\nThe key features of Basaran are:\n\n* Streaming generation using various decoding strategies.\n* Support for both decoder-only and encoder-decoder models.\n* Detokenizer that handles surrogates and whitespace.\n* Multi-GPU support with optional quantization.\n* Real-time partial progress using [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#Event_stream_format).\n* Compatibility with OpenAI API and client libraries.\n* Comes with a fancy web-based playground!\n\n\u003cimg src=\"https://github.com/hyperonym/basaran/blob/master/docs/assets/playground.gif?raw=true\" width=\"640\"\u003e\n\n## Quick Start\n\n### TL;DR\n\nReplace `user/repo` with your [selected model](https://huggingface.co/models?pipeline_tag=text-generation) and `X.Y.Z` with the [latest version](https://hub.docker.com/r/hyperonym/basaran/tags), then run:\n\n```bash\ndocker run -p 80:80 -e MODEL=user/repo hyperonym/basaran:X.Y.Z\n```\n\nAnd you're good to go! 🚀\n\n```\nPlayground: http://127.0.0.1/\nAPI:        http://127.0.0.1/v1/completions\n```\n\n### Installation\n\n#### Using Docker (Recommended)\n\nDocker images are available on [Docker Hub](https://hub.docker.com/r/hyperonym/basaran/tags) and [GitHub Packages](https://github.com/orgs/hyperonym/packages?repo_name=basaran).\n\nFor GPU acceleration, you also need to install the [NVIDIA Driver](https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html) and [NVIDIA Container Runtime](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). Basaran's image already comes with related libraries such as CUDA and cuDNN, so there is no need to install them manually.\n\nBasaran's image can be used in three ways:\n\n* **Run directly**: By specifying the `MODEL=\"user/repo\"` environment variable, the corresponding model can be downloaded from Hugging Face Hub during the first startup.\n* **Bundling**: Create a new Dockerfile to [preload a public model](https://github.com/hyperonym/basaran/blob/master/deployments/bundle/bloomz-560m.Dockerfile) or [bundle a private model](https://github.com/hyperonym/basaran/blob/master/deployments/bundle/private.Dockerfile).\n* **Bind mount**: Mount a model from the local file system into the container and point the `MODEL` environment variable to the corresponding path.\n\nFor the above use cases, you can find sample [Dockerfiles](https://github.com/hyperonym/basaran/tree/master/deployments/bundle) and [docker-compose files](https://github.com/hyperonym/basaran/tree/master/deployments/compose) in the [deployments directory](https://github.com/hyperonym/basaran/tree/master/deployments).\n\n#### Using pip\n\nBasaran is tested on Python 3.8+ and PyTorch 1.13+. You should create a [virtual environment](https://docs.python.org/3/library/venv.html) with the version of Python you want to use, and activate it before proceeding.\n\n1. Install with `pip`:\n\n```bash\npip install basaran\n```\n\n2. Install dependencies required for GPU acceleration (optional):\n\n```bash\npip install accelerate bitsandbytes\n```\n\n3. Replace `user/repo` with the selected model and run Basaran:\n\n```bash\nMODEL=user/repo PORT=80 python -m basaran\n```\n\nFor a complete list of environment variables, see [`__init__.py`](https://github.com/hyperonym/basaran/blob/master/basaran/__init__.py).\n\n#### Running From Source\n\nIf you want to access the latest features or hack it yourself, you can choose to run from source using `git`.\n\n1. Clone the repository:\n\n```bash\ngit clone https://github.com/hyperonym/basaran.git \u0026\u0026 cd basaran\n```\n\n2. Install dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\n3. Replace `user/repo` with the selected model and run Basaran:\n\n```bash\nMODEL=user/repo PORT=80 python -m basaran\n```\n\n### Basic Usage\n\n#### cURL\n\nBasaran's HTTP request and response formats are consistent with the [OpenAI API](https://platform.openai.com/docs/api-reference).\n\nTaking [text completion](https://platform.openai.com/docs/api-reference/completions/create) as an example:\n\n```bash\ncurl http://127.0.0.1/v1/completions \\\n    -H 'Content-Type: application/json' \\\n    -d '{ \"prompt\": \"once upon a time,\", \"echo\": true }'\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eExample response\u003c/summary\u003e\n\n```json\n{\n    \"id\": \"cmpl-e08c701b4ba032c09ef080e1\",\n    \"object\": \"text_completion\",\n    \"created\": 1678003509,\n    \"model\": \"bigscience/bloomz-560m\",\n    \"choices\": [\n        {\n            \"text\": \"once upon a time, the human being faces a complicated situation and he needs to find a new life.\",\n            \"index\": 0,\n            \"logprobs\": null,\n            \"finish_reason\": \"length\"\n        }\n    ],\n    \"usage\": {\n        \"prompt_tokens\": 5,\n        \"completion_tokens\": 21,\n        \"total_tokens\": 26\n    }\n}\n```\n\u003c/details\u003e\n\n#### OpenAI Client Library\n\nIf your application uses [client libraries](https://github.com/openai/openai-python) provided by OpenAI, you only need to modify the `OPENAI_API_BASE` environment variable to match Basaran's endpoint:\n\n```bash\nOPENAI_API_BASE=\"http://127.0.0.1/v1\" python your_app.py\n```\n\nThe [examples](https://github.com/hyperonym/basaran/tree/master/examples) directory contains examples of [using the OpenAI Python library](https://github.com/hyperonym/basaran/blob/master/examples/openai-python-library/main.py).\n\n#### Using as a Python Library\n\nBasaran is also available as a library on [PyPI](https://pypi.org/project/basaran/). It can be used directly in Python without the need to start a separate API server.\n\n1. Install with `pip`:\n\n```bash\npip install basaran\n```\n\n2. Use the `load_model` function to load a model:\n\n```python\nfrom basaran.model import load_model\n\nmodel = load_model(\"user/repo\")\n```\n\n3. Generate streaming output by calling the model:\n\n```python\nfor choice in model(\"once upon a time\"):\n    print(choice)\n```\n\nThe [examples](https://github.com/hyperonym/basaran/tree/master/examples) directory contains examples of [using Basaran as a library](https://github.com/hyperonym/basaran/blob/master/examples/basaran-python-library/main.py).\n\n## Compatibility\n\nBasaran's API format is consistent with OpenAI's, with differences in compatibility mainly in terms of parameter support and response fields. The following sections provide detailed information on the compatibility of each endpoint.\n\n### Models\n\nEach Basaran process serves only one model, so the result will only contain that model.\n\n### Completions\n\nAlthough Basaran does not support the `model` parameter, the OpenAI client library requires it to be present. Therefore, you can enter any random model name.\n\n| Parameter | Basaran | OpenAI | Default Value | Maximum Value |\n| --- | --- | --- | --- | --- |\n| `model` | ○ | ● | - | - |\n| `prompt` | ● | ● | `\"\"` | `COMPLETION_MAX_PROMPT` |\n| `suffix` | ○ | ● | - | - |\n| `min_tokens` | ● | ○ | `0` | `COMPLETION_MAX_TOKENS` |\n| `max_tokens` | ● | ● | `16` | `COMPLETION_MAX_TOKENS` |\n| `temperature` | ● | ● | `1.0` | - |\n| `top_p` | ● | ● | `1.0` | - |\n| `n` | ● | ● | `1` | `COMPLETION_MAX_N` |\n| `stream` | ● | ● | `false` | - |\n| `logprobs` | ● | ● | `0` | `COMPLETION_MAX_LOGPROBS` |\n| `echo` | ● | ● | `false` | - |\n| `stop` | ○ | ● | - | - |\n| `presence_penalty` | ○ | ● | - | - |\n| `frequency_penalty` | ○ | ● | - | - |\n| `best_of` | ○ | ● | - | - |\n| `logit_bias` | ○ | ● | - | - |\n| `user` | ○ | ● | - | - |\n\n### Chat\n\nProviding a unified chat API is currently difficult because each model has a different format for chat history.\n\nTherefore, it is recommended to pre-format the chat history based on the requirements of the specific model and use it as the prompt for the completion API.\n\n#### [GPT-NeoXT-Chat-Base-20B](https://huggingface.co/togethercomputer/GPT-NeoXT-Chat-Base-20B)\n\n```\n**Summarize a long document into a single sentence and ...**\n\n\u003chuman\u003e: Last year, the travel industry saw a big ...\n\n\u003cbot\u003e: If you're traveling this spring break, ...\n\n\u003chuman\u003e: But ...\n\n\u003cbot\u003e:\n```\n\n#### [chatglm-6b](https://huggingface.co/THUDM/chatglm-6b)\n\n```\n[Round 0]\n问：你好\n答：你好!有什么我可以帮助你的吗?\n[Round 1]\n问：你是谁？\n答：\n```\n\n## Roadmap\n\n- [x] API\n    - [x] Models\n        - [x] List models\n        - [x] Retrieve model\n    - [x] Completions\n        - [x] Create completion\n    - [ ] Chat\n        - [ ] Create chat completion\n- [x] Model\n    - [x] Architectures\n        - [x] Encoder-decoder\n        - [x] Decoder-only\n    - [x] Decoding strategies\n        - [x] Random sampling with temperature\n        - [x] Nucleus-sampling (top-p)\n        - [ ] Stop sequences\n        - [ ] Presence and frequency penalties\n\nSee the [open issues](https://github.com/hyperonym/basaran/issues) for a full list of proposed features.\n\n## Contributing\n\nThis project is open-source. If you have any ideas or questions, please feel free to reach out by creating an issue!\n\nContributions are greatly appreciated, please refer to [CONTRIBUTING.md](https://github.com/hyperonym/basaran/blob/master/CONTRIBUTING.md) for more information.\n\n## License\n\nBasaran is available under the [MIT License](https://github.com/hyperonym/basaran/blob/master/LICENSE).\n\n---\n\n© 2023 [Hyperonym](https://hyperonym.org)\n","funding_links":["https://github.com/sponsors/hyperonym"],"categories":["[THUDM/ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B)","Python","Langchain","A01_文本生成_文本对话","Building","Generative KI"],"sub_categories":["Other LLaMA-derived projects:","大语言对话模型及数据","LLM Models"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperonym%2Fbasaran","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhyperonym%2Fbasaran","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperonym%2Fbasaran/lists"}