{"id":19001294,"url":"https://github.com/oneflow-inc/faster-chatglm-6b","last_synced_at":"2025-09-13T01:14:23.997Z","repository":{"id":198838708,"uuid":"616826446","full_name":"Oneflow-Inc/faster-chatglm-6b","owner":"Oneflow-Inc","description":null,"archived":false,"fork":false,"pushed_at":"2023-08-23T08:06:46.000Z","size":896,"stargazers_count":6,"open_issues_count":2,"forks_count":0,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-04-22T17:47:14.474Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Oneflow-Inc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-03-21T06:52:28.000Z","updated_at":"2023-10-08T11:23:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"6dcc5ad8-effe-4807-bb08-fa5ad05f3a6e","html_url":"https://github.com/Oneflow-Inc/faster-chatglm-6b","commit_stats":null,"previous_names":["oneflow-inc/faster-chatglm-6b"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Oneflow-Inc/faster-chatglm-6b","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Ffaster-chatglm-6b","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Ffaster-chatglm-6b/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Ffaster-chatglm-6b/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Ffaster-chatglm-6b/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Oneflow-Inc","download_url":"https://codeload.github.com/Oneflow-Inc/faster-chatglm-6b/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Ffaster-chatglm-6b/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274904087,"owners_count":25371257,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-12T02:00:09.324Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T18:10:39.469Z","updated_at":"2025-09-13T01:14:23.931Z","avatar_url":"https://github.com/Oneflow-Inc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# faster-chatglm-6b\n\n`faster-chatglm-6b` 是一个使用 [OneFlow](https://github.com/Oneflow-Inc/oneflow) 深度学习框架为后端加速[THUDM/chatglm-6b](https://huggingface.co/THUDM/chatglm-6b)运行的项目。\n\n![demo](images/demo.gif)\n\n注：上图为加速效果展示，左侧为 OneFlow ，硬件为 NVIDIA A100 40G\n\n## 软件依赖\n\n请直接使用下面命令安装 OneFlow nightly CUDA 11.7 版本的安装包：\n\n```shell\npython3 -m pip install --pre oneflow -f https://staging.oneflow.info/branch/master/cu117\n```\n\n## 安装方式\n\n下载或 clone 本项目后，在项目目录中运行：\n```shell\npython3 -m pip install -e .\n```\n或\n```shell\npython3 -m pip install git+https://github.com/Oneflow-Inc/faster-chatglm-6b.git\n```\n\n## 代码调用\n\n```ipython\n\u003e\u003e\u003e import faster_chatglm_6b\n\u003e\u003e\u003e from transformers import AutoTokenizer, AutoModel\n\u003e\u003e\u003e tokenizer = AutoTokenizer.from_pretrained(\"THUDM/chatglm-6b\", trust_remote_code=True)\n\u003e\u003e\u003e model = AutoModel.from_pretrained(\"THUDM/chatglm-6b\", trust_remote_code=True).half().cuda()\n\u003e\u003e\u003e response, history = model.chat(tokenizer, \"你好\", history=[])\n\u003e\u003e\u003e print(response)\n你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。\n```\n\n注意：这里和[THUDM/chatglm-6b 官方示例](https://huggingface.co/THUDM/chatglm-6b#%E4%BB%A3%E7%A0%81%E8%B0%83%E7%94%A8)不同的是，在官方示例的最前面增加了一行 `import faster_chatglm_6b`，这样就把后台切换成了 OneFlow。\n\n下面解释一下，这一行代码主要发生了什么:\n\n1. 设置 OneFlow 的一些环境变量，用于控制 OneFlow 框架的行为。\n2. 使用 OneFlow 的 `mock_torch` 方法把所有的 PyTorch 模块替换成对应的 OneFlow 模块。\n3. 利用 transformers 模块的动态模块工具，把找到的原 ChatGLM-6B 中的 `ChatGLMForConditionalGeneration` 模块替换成经过 OneFlow 优化的 `ChatGLMForConditionalGeneration` 模块。\n\n这一行的详细行为，请参考 `faster_chatglm_6b/__init__.py`。\n\n## 更多演示\n\n我们模仿 https://github.com/THUDM/ChatGLM-6B 项目也提供了命令行和网页版的演示，请参考 `examples` 目录下的文件。\n\n```shell\nexamples/\n├── demo.py     # 单轮对话演示\n├── cli_demo.py # 命令行多轮对话演示\n└── web_demo.py # 网页版对话演示\n```\n\n## TODOs\n\n1. OneFlow 支持 skip_init 功能之后，`faster_chatglm_6b/__init__.py`中移除`new_skip_init`。\n2. 移除 `cli_demo.py` 和 `web_demo.py` 中对 `torch.no_grad` 的依赖。\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foneflow-inc%2Ffaster-chatglm-6b","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foneflow-inc%2Ffaster-chatglm-6b","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foneflow-inc%2Ffaster-chatglm-6b/lists"}