{"id":18428994,"url":"https://github.com/codefuse-ai/fastertransformer4codefuse","last_synced_at":"2025-04-07T17:32:26.404Z","repository":{"id":193454076,"uuid":"688398339","full_name":"codefuse-ai/FasterTransformer4CodeFuse","owner":"codefuse-ai","description":"High-performance LLM inference based on our optimized version of FastTransfomer","archived":false,"fork":false,"pushed_at":"2023-12-14T08:38:26.000Z","size":29570,"stargazers_count":108,"open_issues_count":0,"forks_count":8,"subscribers_count":3,"default_branch":"main","last_synced_at":"2023-12-14T09:40:40.471Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codefuse-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-09-07T09:10:09.000Z","updated_at":"2023-12-09T13:27:43.000Z","dependencies_parsed_at":"2023-09-08T08:59:31.336Z","dependency_job_id":"2216fe51-7556-46db-b134-2b1ddd8f13f1","html_url":"https://github.com/codefuse-ai/FasterTransformer4CodeFuse","commit_stats":null,"previous_names":["codefuse-ai/fastertransformer4codefuse"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codefuse-ai%2FFasterTransformer4CodeFuse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codefuse-ai%2FFasterTransformer4CodeFuse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codefuse-ai%2FFasterTransformer4CodeFuse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codefuse-ai%2FFasterTransformer4CodeFuse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codefuse-ai","download_url":"https://codeload.github.com/codefuse-ai/FasterTransformer4CodeFuse/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223286332,"owners_count":17120001,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T05:15:19.920Z","updated_at":"2024-11-06T05:15:20.642Z","avatar_url":"https://github.com/codefuse-ai.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FasterTransformer4CodeFuse\n\n\u003cdiv align=\"center\"\u003e\n\u003cp\u003e\u003cimg src=\"./assets/LOGO.png\" width=\"50%\" /\u003e\u003c/p\u003e\n\u003cp\u003e\n    \u003ca href=\"https://github.com/codefuse-ai/FasterTransformer4CodeFuse\"\u003e\n        \u003cimg alt=\"stars\" src=\"https://img.shields.io/github/stars/codefuse-ai/FasterTransformer4CodeFuse?style=social\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/codefuse-ai/FasterTransformer4CodeFuse\"\u003e\n        \u003cimg alt=\"forks\" src=\"https://img.shields.io/github/forks/codefuse-ai/FasterTransformer4CodeFuse?style=social\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://opensource.org/licenses/Apache-2.0\"\u003e\n      \u003cimg alt=\"License: apache 2.0\" src=\"https://img.shields.io/badge/License-Apache_2.0-blue.svg\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/codefuse-ai/FasterTransformer4CodeFuse/issues\"\u003e\n      \u003cimg alt=\"Open Issues\" src=\"https://img.shields.io/github/issues-raw/codefuse-ai/FasterTransformer4CodeFuse\" /\u003e\n    \u003c/a\u003e\n    \u003cbr/\u003e\n\u003c/p\u003e\n\n  \n| [**简体中文**](README_CN.md) |\n\u003c/div\u003e\n\n## Introduce\n\nProvide high-performance model inference, mainly supporting the CodeFuse model from Ant Group.\n\nCompared to the [original FT](https://github.com/NVIDIA/FasterTransformer), this repo has these features:\n\n- :white_check_mark: Int8 quantization of CodeFuse model\n- :white_check_mark: Prompt does not require a complete word at the end\n- :white_check_mark: Python API\n- :white_check_mark: Streaming Output with Python API\n- :white_check_mark: Higher model load speed\n- :white_check_mark: Some bugfix\n\n## Performance\n\u003e Batch size: 1\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=\"3\"\u003eModel\u003c/td\u003e\n        \u003ctd colspan=\"4\"\u003eCodeFuse 13B\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=\"3\"\u003eMeasurements\u003c/td\u003e\n        \u003ctd colspan=\"4\"\u003eLatency (ms)\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=\"3\"\u003eGPU\u003c/td\u003e\n        \u003ctd colspan=\"2\"\u003eSingle A100\u003c/td\u003e\n        \u003ctd colspan=\"2\"\u003e2 * A100 Tensor Parallelism\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=\"3\"\u003eData Type\u003c/td\u003e\n        \u003ctd\u003efp16\u003c/td\u003e\n        \u003ctd\u003eint8\u003c/td\u003e\n        \u003ctd\u003efp16\u003c/td\u003e\n        \u003ctd\u003eint8\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd rowspan=\"4\"\u003eInput/Output Length\u003c/td\u003e\n        \u003ctd\u003e16\u003c/td\u003e\n        \u003ctd\u003e8\u003c/td\u003e\n        \u003ctd\u003e160\u003c/td\u003e\n        \u003ctd\u003e195\u003c/td\u003e\n        \u003ctd\u003e238\u003c/td\u003e\n        \u003ctd\u003e84\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e64\u003c/td\u003e\n        \u003ctd\u003e32\u003c/td\u003e\n        \u003ctd\u003e608\u003c/td\u003e\n        \u003ctd\u003e369\u003c/td\u003e\n        \u003ctd\u003e373\u003c/td\u003e\n        \u003ctd\u003e295\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e256\u003c/td\u003e\n        \u003ctd\u003e128\u003c/td\u003e\n        \u003ctd\u003e2650\u003c/td\u003e\n        \u003ctd\u003e1530\u003c/td\u003e\n        \u003ctd\u003e1492\u003c/td\u003e\n        \u003ctd\u003e1130\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e1024\u003c/td\u003e\n        \u003ctd\u003e512\u003c/td\u003e\n        \u003ctd\u003e10776\u003c/td\u003e\n        \u003ctd\u003e7054\u003c/td\u003e\n        \u003ctd\u003e6786\u003c/td\u003e\n        \u003ctd\u003e5415\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=\"3\"\u003eTokens Per Sec\u003c/td\u003e\n        \u003ctd\u003e48\u003c/td\u003e\n        \u003ctd\u003e75\u003c/td\u003e\n        \u003ctd\u003e77\u003c/td\u003e\n        \u003ctd\u003e98\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n## Get Start\n\nWe run in the container environment: `nvcr.io/nvidia/pytorch:22.09-py3`。\n\n#### 1. Install requirements\n\n```\npip install --no-cache-dir pybind11==2.6.2 transformers accelerate sentencepiece\n\necho \"export pybind11_DIR=/opt/conda/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/\" \u003e\u003e ~/.bashrc\nexport pybind11_DIR=/opt/conda/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/\n```\n\n#### 2. Build\n\n```\nmkdir build ; cd build\nexport TORCH_PYTHON_LIBRARIES=/opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so\ncmake -DCMAKE_BUILD_TYPE=Release -DSM=\"80;75\" -DBUILD_PYT=ON -DSPARSITY_SUPPORT=OFF -DMEASURE_BUILD_TIME=ON \\\n      -DBUILD_CUTLASS_MIXED_GEMM=ON -DBUILD_MULTI_GPU=ON -DBUILD_TRT=OFF \\\n      -DENABLE_FP8=OFF -DBUILD_PYBIND=ON -DTORCH_PYTHON_LIBRARIES=${TORCH_PYTHON_LIBRARIES} ..\nmake -j\"$(grep -c ^processor /proc/cpuinfo)\"\n```\n\n#### 3. Run\n\nYou can use `examples/pytorch/codefuse/huggingface_convert.py` script to convert checkpoint files from HuggingFace to FasterTransformer.\n```\nexport MODEL_NAME=codefuse\nexport TENSOR_PARA_SIZE=2\n\npython ../examples/pytorch/codefuse/huggingface_convert.py \\\n       -o ../models/${MODEL_NAME}/fastertransformer \\\n       -i ../models/${MODEL_NAME}/transformers \\\n       -infer_gpu_num ${TENSOR_PARA_SIZE} \\\n       -processes 20 \\\n       -weight_data_type fp16 \\\n       -model_name gptneox\n```\n\nYou can use `examples/pytorch/codefuse/quant_and_save.py` script to convert fp16 or fp32 FasterTransformer checkpoint files to int8 files and scales, getting higher model load speed and smaller checkpoint files.\n```\nexport MODEL_NAME=codefuse\nexport TENSOR_PARA_SIZE=2\n\npython ../examples/pytorch/codefuse/quant_and_save.py \\\n       --in_dir ../models/${MODEL_NAME}/fastertransformer/${TENSOR_PARA_SIZE}-gpu \\\n       --out_dir ../models/${MODEL_NAME}/fastertransformer/${TENSOR_PARA_SIZE}-gpu_int8 \\\n       --lib_path ../build/lib/libth_common.so \\\n       --tensor_para_size ${TENSOR_PARA_SIZE} \\\n       --use_gptj_residual \\\n       --data_type fp16\n```\n\nYou can use `examples/pytorch/codefuse/codefuse_example.py` to run model inference.\n```\nexport MODEL_NAME=codefuse\n\n# fp16 1gpu\npython ../examples/pytorch/codefuse/codefuse_example.py \\\n       --ckpt_path ../models/${MODEL_NAME}/fastertransformer/1-gpu \\\n       --tokenizer_path ../models/${MODEL_NAME}/transformers\n\n# int8 1gpu\npython ../examples/pytorch/codefuse/codefuse_example.py \\\n       --ckpt_path ../models/${MODEL_NAME}/fastertransformer/1-gpu_int8 \\\n       --tokenizer_path ../models/${MODEL_NAME}/transformers \\\n       --int8_mode 1 \\\n       --enable_int8_weights 1\n\n# fp16 2gpus\ntorchrun --nproc_per_node 2 ../examples/pytorch/codefuse/codefuse_example.py \\\n         --world_size 2 \\\n         --ckpt_path ../models/${MODEL_NAME}/fastertransformer/2-gpu \\\n         --tokenizer_path ../models/${MODEL_NAME}/transformers\n\n# int8 2gpus\ntorchrun --nproc_per_node 2 ../examples/pytorch/codefuse/codefuse_example.py \\\n         --world_size 2 \\\n         --ckpt_path ../models/${MODEL_NAME}/fastertransformer/2-gpu_int8 \\\n         --tokenizer_path ../models/${MODEL_NAME}/transformers \\\n         --int8_mode 1 \\\n         --enable_int8_weights 1\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodefuse-ai%2Ffastertransformer4codefuse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodefuse-ai%2Ffastertransformer4codefuse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodefuse-ai%2Ffastertransformer4codefuse/lists"}