{"id":25664457,"url":"https://github.com/acai66/qwen2.5_numpy","last_synced_at":"2025-04-22T14:05:24.557Z","repository":{"id":277876603,"uuid":"933700065","full_name":"acai66/qwen2.5_numpy","owner":"acai66","description":"使用numpy实现DeepSeek-R1-Distill-Qwen-1.5B的推理过程，易于学习LLM推理与移植到其它编程语言加速。 Implementing the inference process of DeepSeek-R1-Distill-Qwen-1.5B using numpy, making it easy to learn LLM (Large Language Model) inference and to port to other programming languages for acceleration.","archived":false,"fork":false,"pushed_at":"2025-02-20T04:51:21.000Z","size":32,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-22T14:05:07.759Z","etag":null,"topics":["deepseek","deepseek-r1","llama-cpp","llm-inference","numpy","qwen","qwen2"],"latest_commit_sha":null,"homepage":"https://hyacm.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/acai66.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-16T13:52:58.000Z","updated_at":"2025-03-21T01:41:32.000Z","dependencies_parsed_at":"2025-02-16T18:44:58.292Z","dependency_job_id":null,"html_url":"https://github.com/acai66/qwen2.5_numpy","commit_stats":null,"previous_names":["acai66/qwen2.5_numpy"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acai66%2Fqwen2.5_numpy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acai66%2Fqwen2.5_numpy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acai66%2Fqwen2.5_numpy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acai66%2Fqwen2.5_numpy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/acai66","download_url":"https://codeload.github.com/acai66/qwen2.5_numpy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250255709,"owners_count":21400410,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deepseek","deepseek-r1","llama-cpp","llm-inference","numpy","qwen","qwen2"],"created_at":"2025-02-24T06:18:45.189Z","updated_at":"2025-04-22T14:05:24.489Z","avatar_url":"https://github.com/acai66.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 阿里通义千问 Qwen2.5 numpy推理(支持Deekseek-R1蒸馏的Qwen模型)\n\n- 只使用 `numpy` 实现 `Qwen2.5` 的推理，不使用 `torch`、`transformers` 等框架，易于学习LLM的推理过程，以及移植到其它语言\n- 支持阿里云原始的通义千问 `Qwen2.5` 模型、`Deekseek-R1` 蒸馏的 `Qwen2.5` 模型，其它微调模型暂未测试(理论上支持)\n- 支持 `batch` 推理\n- 支持 `temperature`、`topk`、`topp`、`penalty`等参数\n- 支持 `KV缓存`\n- 支持 `q8_0量化`\n- 以学习为目的，约400行代码实现了完整的llm推理过程，不含 `tokenization` 部分\n\n## 测试\n\n### 1. 安装依赖\n\n```bash\npip install numpy tokenizers\n```\n\n### 2. 下载 `safetensors` 模型\n\n到模型分享平台下载完整模型，参考 [`modelscope` 平台下载说明](https://www.modelscope.cn/docs/models/download)\n\n  1. [Qwen2.5-0.5B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen2.5-0.5B-Instruct/summary)\n  2. [Qwen2.5-1.5B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen2.5-1.5B-Instruct/summary)\n  3. [DeepSeek-R1-Distill-Qwen-1.5B](https://www.modelscope.cn/models/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B/summary)\n\n### 3. 转换模型\n\n使用 `parse_safetensors.py` 脚本转换模型，提供下载的模型目录，转换后的npy模型保存目录，例如:\n\n```bash\npython parse_safetensors.py --model_dir 下载的模型目录 --npy_save_dir 转换后的npy模型保存目录\n```\n\n### 4. 运行推理\n\n修改 `model.py` 中的模型路径、`prompt`，运行 `model.py`，或者手动从 `model.py` 中导入 `Model` 类，参考 `model.py` 中 `main` 函数的使用方法\n\n```python\nif __name__ == '__main__':\n    # chat_template = '\u003c|im_start|\u003esystem\\n{}\u003c|im_end|\u003e\\n\u003c|im_start|\u003euser\\n{}\u003c|im_end|\u003e\\n\u003c|im_start|\u003eassistant\\n'\n    # model_weights_path = '/Users/acai/Downloads/models/Qwen2.5_1.5B_Instruct_npy'\n    chat_template = '\u003c｜begin▁of▁sentence｜\u003e\u003c｜begin▁of▁sentence｜\u003e{}\u003c｜User｜\u003e{}\u003c｜Assistant｜\u003e\u003cthink\u003e\\n'\n    model_weights_path = '/Users/acai/Downloads/models/DeepSeek_R1_Distill_Qwen_1.5B_npy_FP32'\n\n    model = Model(model_weights_path)\n\n    role_system_content = \"You are a helpful assistant.\"\n    prompt = [\n        # \"怎么用python numpy实softmax？\",\n        \"你是谁？\",\n        # \"计算456+826\",\n    ] # 批次\n    text = list(map(lambda x: chat_template.format(role_system_content, x), prompt))\n\n    model_inputs = np.array([model.tokenizer.encode_batch_fast(text)[i].ids for i in range(len(text))], dtype=np.int32)\n\n    generated_ids = model.generate(\n        model_inputs,\n        max_new_tokens=2048\n    )\n\n    response = model.tokenizer.decode_batch(generated_ids, skip_special_tokens=True)\n    print(f'{\"\\n\".join(response)}')\n```\n\n## Benchmark\n\n与 [`llama.cpp`](https://github.com/ggml-org/llama.cpp/releases/tag/b4722)对比每秒 `Tokens` 速度，测试平台为 `Mac mini M4`，内存16G\n\n|模型|精度|numpy|llama.cpp|\n|:---:|:---:|:---:|:---:|\n|Qwen2.5_0.5B_Instruct|float32|29.77|45.6|\n|Qwen2.5_0.5B_Instruct|float16|-|86.44|\n|Qwen2.5_0.5B_Instruct|q8_0|1.94|140.53|\n|DeepSeek_R1_Distill_Qwen_1.5B|float32|10.31|15.55|\n|DeepSeek_R1_Distill_Qwen_1.5B|float16|-|31.55|\n|DeepSeek_R1_Distill_Qwen_1.5B|q8_0|0.68|54.47|\n\n7B模型用float32精度时需要30G左右内存，机器内存不足，未测试\n\n比较震惊的是 `numpy` 的矩阵加速只支持float32、float64，不支持整数、半精度等，导致float32速度是最快的，float32模型内存占用很大，容易导致内存不足，同时对内存带宽的要求很高，估计只能通过移植到其它语言，从底层优化矩阵运算才能加速到 `llama.cpp` 的速度\n\n## 参考\n\n- [Qwen2.5](https://qwenlm.github.io/blog/qwen2.5/)\n- [DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1/)\n- [llama.cpp](https://github.com/ggml-org/llama.cpp)\n- [transformers](https://github.com/huggingface/transformers)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facai66%2Fqwen2.5_numpy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Facai66%2Fqwen2.5_numpy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facai66%2Fqwen2.5_numpy/lists"}