{"id":20373497,"url":"https://github.com/megengine/inferllm","last_synced_at":"2025-04-12T19:49:20.213Z","repository":{"id":161914089,"uuid":"636092099","full_name":"MegEngine/InferLLM","owner":"MegEngine","description":"a lightweight LLM model inference framework","archived":false,"fork":false,"pushed_at":"2024-04-07T01:57:37.000Z","size":5146,"stargazers_count":723,"open_issues_count":42,"forks_count":91,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-03T23:11:28.060Z","etag":null,"topics":["deeplearning","inference","llm","mobile"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MegEngine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-04T05:40:17.000Z","updated_at":"2025-04-02T01:12:18.000Z","dependencies_parsed_at":"2024-10-27T19:14:39.538Z","dependency_job_id":"0c87c751-c6cd-4eeb-b0bc-3a702211e496","html_url":"https://github.com/MegEngine/InferLLM","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MegEngine%2FInferLLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MegEngine%2FInferLLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MegEngine%2FInferLLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MegEngine%2FInferLLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MegEngine","download_url":"https://codeload.github.com/MegEngine/InferLLM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248625501,"owners_count":21135513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deeplearning","inference","llm","mobile"],"created_at":"2024-11-15T01:18:41.298Z","updated_at":"2025-04-12T19:49:20.183Z","avatar_url":"https://github.com/MegEngine.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# InferLLM \n[中文 README](./README_Chinese.md)\n\nInferLLM is a lightweight LLM model inference framework that mainly references and borrows from the llama.cpp project. llama.cpp puts almost all core code and kernels in a single file and use a large number of macros, making it difficult for developers to read and modify. InferLLM has the following features:\n\n- Simple structure, easy to get started and learning, and decoupled the framework part from the kernel part.\n- High efficiency, ported most of the kernels in llama.cpp.\n- Defined a dedicated KVstorage type for easy caching and management.\n- Compatible with multiple model formats (currently only supporting alpaca Chinese and English int4 models).\n- Currently supports CPU and GPU, optimized for Arm, x86, CUDA and riscv-vector. And it can be deployed on mobile phones, with acceptable speed.\n\nIn short, InferLLM is a simple and efficient LLM CPU inference framework that can deploy quantized models in LLM locally and has good inference speed.\n\n## Latest News\n- 2023.08.16: Add support for LLama-2-7B model.\n- 2023.08.8: Optimized the performance on Arm, which optimized the int4 matmul kernel with arm asm and kernel packing.\n- berfor: support chatglm/chatglm2, baichuan, alpaca, ggml-llama model.\n\n## How to use\n### Download model\nCurrently, InferLLM uses the same models as llama.cpp and can download models from the llama.cpp project. In addition, models can also be downloaded directly from Hugging Face [kewin4933/InferLLM-Model](https://huggingface.co/kewin4933/InferLLM-Model/tree/main). Currently, two alpaca, llama2, chatglm/chatglm2 and baichuan models are uploaded in this project, one is the Chinese int4 model and the other is the English int4 model.\n\n### Compile InferLLM\n#### Local compilation\n```shell\nmkdir build\ncd build\ncmake ..\nmake\n```\nGPU is disabled default, if you want to enable GPU, please use `cmake -DENABLE_GPU=ON ..` to enable GPU. Now only CUDA is supported, before use CUDA, please install CUDA toolkit first. \n\n#### Android cross compilation\nAccording to the cross compilation, you can use the pre-prepared tools/android_build.sh script. You need to install NDK in advance and configure the path of NDK to the NDK_ROOT environment variable.\n```shell\nexport NDK_ROOT=/path/to/ndk\n./tools/android_build.sh\n```\n### Run InferLLM\nRunning ChatGLM model please refer to [ChatGLM model documentation](./application/chatglm/Readme.md).\n\nIf it is executed locally, execute `./chatglm -m chatglm-q4.bin -t 4` directly. If you want to execute it on your mobile phone, you can use the adb command to copy alpaca and the model file to your mobile phone, and then execute `adb shell ./chatglm -m chatglm-q4.bin -t 4`. \n\nThe default device is CPU, if you want to inference with GPU, please use `./chatglm -m chatglm-q4.bin -g GPU` to specify the GPU device.\n\n- x86 is：Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz\n![x86 running](./assets/ChatGLM-x86.gif )\n- android is xiaomi9，Qualcomm SM8150 Snapdragon 855\n![android running](./assets/arm-mi9.gif)\n- CPU is SG2042, with riscv-vector 0.7, 64 threads\n![sg2042 running](./assets/sg2042.gif)\n\nAccording to [x86 profiling result](./docs/profile.md), we strongly advise using 4 threads.\n\n### Supported model\nNow InferLLM supports the following models:\n* [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B): usage please refer to [ChatGLM](./application/chatglm/Readme.md)\n* [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B): usage please refer to [ChatGLM](./application/chatglm/Readme.md)\n* [llama](https://github.com/facebookresearch/llama)\n* [llama2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)\n* [alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html)\n* [baichuan](https://github.com/baichuan-inc/baichuan-7B) : usage please refer to [baichuan](./application/baichuan/Readme.md)\n### License\nInferLLM is licensed under the Apache License, Version 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmegengine%2Finferllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmegengine%2Finferllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmegengine%2Finferllm/lists"}