{"id":22354766,"url":"https://github.com/pprp/lightllm-gradio","last_synced_at":"2025-08-12T06:34:17.897Z","repository":{"id":211087061,"uuid":"728168999","full_name":"pprp/lightllm-gradio","owner":"pprp","description":"Based on LightLLM, we provide interface using Gradio. Feel free to use!","archived":false,"fork":false,"pushed_at":"2023-12-08T09:08:07.000Z","size":1187,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-07T14:06:18.304Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pprp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-06T11:26:18.000Z","updated_at":"2024-11-09T13:25:04.000Z","dependencies_parsed_at":"2025-01-31T13:43:16.719Z","dependency_job_id":"48e26b8d-854c-4d42-a63d-4a8399ebe765","html_url":"https://github.com/pprp/lightllm-gradio","commit_stats":null,"previous_names":["pprp/lightllm-gradio"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pprp/lightllm-gradio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2Flightllm-gradio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2Flightllm-gradio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2Flightllm-gradio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2Flightllm-gradio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pprp","download_url":"https://codeload.github.com/pprp/lightllm-gradio/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2Flightllm-gradio/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270014429,"owners_count":24512652,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-04T13:15:08.529Z","updated_at":"2025-08-12T06:34:17.822Z","avatar_url":"https://github.com/pprp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003cimg alt=\"LightLLM\" src=\"assets/lightllm.drawio.png\" width=90%\u003e\n  \u003c/picture\u003e\n\u003c/div\u003e\n\n---\n\u003cdiv align=\"center\"\u003e\n\n[![docs](https://img.shields.io/badge/docs-latest-blue)](https://github.com/ModelTC/lightllm/blob/main/docs/TokenAttention.md)\n[![Docker](https://github.com/ModelTC/lightllm/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/ModelTC/lightllm/actions/workflows/docker-publish.yml)\n[![stars](https://img.shields.io/github/stars/ModelTC/lightllm?style=social)](https://github.com/ModelTC/lightllm)\n[![Discord Banner](https://img.shields.io/discord/1139835312592392214?logo=discord\u0026logoColor=white)](https://discord.gg/WzzfwVSguU)\n[![license](https://img.shields.io/github/license/ModelTC/lightllm)](https://github.com/ModelTC/lightllm/blob/main/LICENSE)\n\u003c/div\u003e\n\nLightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. LightLLM harnesses the strengths of numerous well-regarded open-source implementations, including but not limited to FasterTransformer, TGI, vLLM, and FlashAttention.\n\n## Features\n\n- Tri-process asynchronous collaboration: tokenization, model inference, and detokenization are performed asynchronously, leading to a considerable improvement in GPU utilization.\n- Nopad (Unpad): offers support for nopad attention operations across multiple models to efficiently handle requests with large length disparities.\n- Dynamic Batch: enables dynamic batch scheduling of requests\n- [FlashAttention](https://github.com/Dao-AILab/flash-attention): incorporates FlashAttention to improve speed and reduce GPU memory footprint during inference.\n- Tensor Parallelism: utilizes tensor parallelism over multiple GPUs for faster inference.\n- [Token Attention](./docs/TokenAttention.md): implements token-wise's KV cache memory management mechanism, allowing for zero memory waste during inference.\n- High-performance Router: collaborates with Token Attention to meticulously manage the GPU memory of each token, thereby optimizing system throughput.\n- Int8KV Cache: This feature will increase the capacity of tokens to almost twice as much. only llama support.\n\n## Supported Model List\n\n- [BLOOM](https://huggingface.co/bigscience/bloom)\n- [LLaMA](https://github.com/facebookresearch/llama)\n- [LLaMA V2](https://huggingface.co/meta-llama)\n- [StarCoder](https://github.com/bigcode-project/starcoder)\n- [Qwen-7b](https://github.com/QwenLM/Qwen-7B)\n- [ChatGLM2-6b](https://github.com/THUDM/ChatGLM2-6B)\n- [Baichuan-7b](https://github.com/baichuan-inc/Baichuan-7B)\n- [Baichuan2-7b](https://github.com/baichuan-inc/Baichuan2)\n- [Baichuan2-13b](https://github.com/baichuan-inc/Baichuan2)    \n- [Baichuan-13b](https://github.com/baichuan-inc/Baichuan-13B)\n- [InternLM-7b](https://github.com/InternLM/InternLM)\n- [Yi-34b](https://huggingface.co/01-ai/Yi-34B)  \n\n\u003e When you start Qwen-7b, you need to set the parameter '--eos_id 151643 --trust_remote_code'.\n\n\u003e ChatGLM2 needs to set the parameter '--trust_remote_code'.\n\n\u003e Baichuan and Baichuan2 needs to set the parameter '--trust_remote_code'.\n\n\u003e InternLM needs to set the parameter '--trust_remote_code'.\n\n## Get started\n\n### Requirements\n\nThe code has been tested with Pytorch\u003e=1.3, CUDA 11.8, and Python 3.9. To install the necessary dependencies, please refer to the provided **requirements.txt** and follow the instructions as\n\n~~~shell\npip install -r requirements.txt\n~~~\n\n### Container\n\nYou can use the official Docker container to run the model more easily. To do this, follow these steps:\n\n- Pull the container from the GitHub Container Registry:\n\n    ```shell\n    docker pull ghcr.io/modeltc/lightllm:main\n    ```\n\n- Run the container with GPU support and port mapping:\n\n    ```shell\n    docker run -it --gpus all -p 8080:8080                  \\\n            --shm-size 1g -v your_local_path:/data/         \\\n            ghcr.io/modeltc/lightllm:main /bin/bash\n    ```\n\n- Alternatively, you can build the container yourself:\n\n    ```shell\n    docker build -t \u003cimage_name\u003e .\n    docker run -it --gpus all -p 8080:8080                  \\\n            --shm-size 1g -v your_local_path:/data/         \\\n            \u003cimage_name\u003e /bin/bash\n    ```\n\n- You can also use a helper script to launch both the container and the server:\n\n    ```shell\n    python tools/quick_launch_docker.py --help\n    ```\n\n- Note: If you use multiple GPUs, you may need to increase the shared memory size by adding `--shm-size` to the `docker run` command.\n\n### Installation\n\n- Install from the source code by\n\n~~~shell\npython setup.py install\n~~~\n\n The code has been tested on a range of GPUs including A100, A800, 4090, and H800. If you are running the code on A100, A800, etc., we recommend using triton==2.1.0 or triton==2.0.0.dev20221202. If you are running the code on H800, etc., it is necessary to compile and install the source code of [triton==2.1.0](https://github.com/openai/triton/tree/main) from the GitHub repository. If the code doesn't work on other GPUs, try modifying the triton kernel used in model inference.\n- Install Triton Package\n\nuse triton==2.1.0  (Better performance, but the code is under continuous development and may be unstable.)\n~~~shell\npip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly\n~~~\n\nuse triton==2.0.0.dev20221202 (This version has a memory leak bug. Refer to the [issue #209](https://github.com/ModelTC/lightllm/issues/209) for the fix method. )\n~~~shell\npip install triton==2.0.0.dev20221202\n~~~\n\n### RUN LLaMA\nWith efficient Routers and TokenAttention, LightLLM can be deployed as a service and achieve the state-of-the-art throughput performance.\n\nLaunch the server:\n\n~~~shell\npython -m lightllm.server.api_server --model_dir /path/llama-7B     \\\n                                     --host 0.0.0.0                 \\\n                                     --port 8080                    \\\n                                     --tp 1                         \\\n                                     --max_total_token_num 120000\n~~~\n\nThe parameter `max_total_token_num` is influenced by the GPU memory of the deployment environment. A larger value for this parameter allows for the processing of more concurrent requests, thereby increasing system concurrency. For more startup parameters, please refer to [api_server.py](lightllm/server/api_server.py) or [ApiServerArgs.md](docs/ApiServerArgs.md).\n\nTo initiate a query in the shell:\n\n~~~shell\ncurl http://127.0.0.1:8080/generate     \\\n    -X POST                             \\\n    -d '{\"inputs\":\"What is AI?\",\"parameters\":{\"max_new_tokens\":17, \"frequency_penalty\":1}}' \\\n    -H 'Content-Type: application/json'\n~~~\n\nTo query from Python:\n\n~~~python\nimport time\nimport requests\nimport json\n\nurl = 'http://localhost:8080/generate'\nheaders = {'Content-Type': 'application/json'}\ndata = {\n    'inputs': 'What is AI?',\n    \"parameters\": {\n        'do_sample': False,\n        'ignore_eos': False,\n        'max_new_tokens': 1024,\n    }\n}\nresponse = requests.post(url, headers=headers, data=json.dumps(data))\nif response.status_code == 200:\n    print(response.json())\nelse:\n    print('Error:', response.status_code, response.text)\n~~~\n\n## Performance\n\n### Service Performance\n\nWe compared the service performance of LightLLM and vLLM==0.1.2 on LLaMA-7B using an A800 with 80G GPU memory.\n\nTo begin, prepare the data as follows:\n\n~~~shell\nwget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json\n~~~\n\nLaunch the service:\n\n~~~shell\npython -m lightllm.server.api_server --model_dir /path/llama-7b --tp 1 --max_total_token_num 121060 --tokenizer_mode auto\n~~~\n\nEvaluation:\n\n~~~shell\ncd test\npython benchmark_serving.py --tokenizer /path/llama-7b --dataset /path/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 2000 --request-rate 200\n~~~\n\nThe performance comparison results are presented below:\n\n| vLLM                                                 | LightLLM                                              |\n| ---------------------------------------------------- | ----------------------------------------------------- |\n| Total time: 361.79 s\u003cbr/\u003eThroughput: 5.53 requests/s | Total time: 188.85 s\u003cbr/\u003eThroughput: 10.59 requests/s |\n\n### Static inference performance\n\nFor debugging, we offer static performance testing scripts for various models. For instance, you can evaluate the inference performance of the LLaMA model by\n\n~~~shell\ncd test/model\npython test_llama.py\n~~~\n\n### FAQ\n\n- The LLaMA tokenizer fails to load.\n    - consider resolving this by running the command `pip install protobuf==3.20.0`.\n- `error   : PTX .version 7.4 does not support .target sm_89`\n    - launch with `bash tools/resolve_ptx_version python -m lightllm.server.api_server ... `\n\n## Community\n\nFor further information and discussion, [join our discord server](https://discord.gg/WzzfwVSguU).\n\n## License\n\nThis repository is released under the [Apache-2.0](LICENSE) license.\n\n## Acknowledgement\n\nWe learned a lot from the following projects when developing LightLLM.\n- [Faster Transformer](https://github.com/NVIDIA/FasterTransformer)\n- [Text Generation Inference](https://github.com/huggingface/text-generation-inference)\n- [vLLM](https://github.com/vllm-project/vllm)\n- [Flash Attention 1\u00262](https://github.com/Dao-AILab/flash-attention)\n- [OpenAI Triton](https://github.com/openai/triton)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpprp%2Flightllm-gradio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpprp%2Flightllm-gradio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpprp%2Flightllm-gradio/lists"}