{"id":19965554,"url":"https://github.com/tencent/forward","last_synced_at":"2025-04-05T02:10:54.603Z","repository":{"id":44709118,"uuid":"346556304","full_name":"Tencent/Forward","owner":"Tencent","description":"A library for high performance deep learning inference on NVIDIA GPUs. ","archived":false,"fork":false,"pushed_at":"2022-01-29T16:34:05.000Z","size":85003,"stargazers_count":552,"open_issues_count":0,"forks_count":67,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-03-29T01:12:14.980Z","etag":null,"topics":["cuda","deep-learning","forward","gpu","inference","inference-engine","keras","neural-network","onnx","pytorch","tensorflow","tensorrt"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tencent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-11T02:40:04.000Z","updated_at":"2025-02-06T14:11:09.000Z","dependencies_parsed_at":"2022-09-06T05:31:20.739Z","dependency_job_id":null,"html_url":"https://github.com/Tencent/Forward","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tencent%2FForward","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tencent%2FForward/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tencent%2FForward/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tencent%2FForward/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tencent","download_url":"https://codeload.github.com/Tencent/Forward/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247276189,"owners_count":20912288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cuda","deep-learning","forward","gpu","inference","inference-engine","keras","neural-network","onnx","pytorch","tensorflow","tensorrt"],"created_at":"2024-11-13T02:29:28.947Z","updated_at":"2025-04-05T02:10:49.593Z","avatar_url":"https://github.com/Tencent.png","language":"C++","readme":"![](doc/img/forward_logo_1.png)\n\n# Forward 深度学习推理加速框架\n\n[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE) [![Build Status](https://travis-ci.com/Tencent/Forward.svg?branch=master)](https://travis-ci.com/Tencent/Forward)\n\n----\n\n- [Forward 深度学习推理加速框架](#forward-深度学习推理加速框架)\n  - [什么是 Forward](#什么是-forward)\n  - [为什么选择 Forward](#为什么选择-forward)\n  - [快速上手 Forward](#快速上手-forward)\n    - [环境依赖](#环境依赖)\n    - [项目构建](#项目构建)\n    - [Forward-Cpp 使用](#forward-cpp-使用)\n    - [Forward-Python 使用](#forward-python-使用)\n    - [Forward-Bert 使用](#forward-bert-使用)\n    - [更多使用方法](#更多使用方法)\n    - [Logging 日志](#logging-日志)\n  - [模型和算子支持](#模型和算子支持)\n    - [模型](#模型)\n    - [算子](#算子)\n  - [参考资料](#参考资料)\n  - [贡献](#贡献)\n  - [许可证](#许可证)\n\n----\n\n[[English Version](README_EN.md)]\n\n## 什么是 Forward\n\nForward 是一款腾讯研发的 GPU 高性能推理加速框架。它提出了一种解析方案，可直接加载主流框架模型（Tensorflow / PyTorch / Keras / ONNX）转换成 TensorRT 推理加速引擎，帮助用户节省中间繁杂的模型转换或网络构建步骤。相对于直接使用 TensorRT，Forward 更易用以及更容易扩展支持更多模型和算子。目前，Forward 除了覆盖支持主流的 CV，NLP 及推荐领域的深度学习模型外，还支持一些诸如 BERT，FaceSwap，StyleTransfer 这类高级模型。\n\n## 为什么选择 Forward\n\n- **模型性能优化高**：基于 TensorRT API 开发网络层级的支持，保证对于通用网络层级的推理性能优化处于最优级别；\n- **模型支持范围广**：除了通用的 CV，NLP，及推荐类模型，还支持一些诸如 BERT，FaceSwap，StyleTransfer 这类高级模型；\n- **多种推理模式**：支持 FLOAT / HALF / INT8 推理模式；\n- **接口简单易用**：直接导入已训练好的 Tensorflow(.pb) / PyTorch(.pth) / Keras(.h5) / ONNX(.onnx) 模型文件，隐式转换为高性能的推理 Engine 进行推理加速；\n- **支持自研扩展**：可根据业务模型[扩展支持自定义网络层级](doc/cn/usages/add_support_op_CN.md)；\n- **支持 C++ 和 Python 接口调用**。\n\n## 快速上手 Forward\n\n### 环境依赖\n\n- NVIDIA CUDA \u003e= 10.0, CuDNN \u003e= 7 (推荐 CUDA 10.2 以上)\n- TensorRT \u003e= 7.0.0.11 (推荐 TensorRT-7.2.1.6)\n- CMake \u003e= 3.12.2\n- GCC \u003e= 5.4.0, ld \u003e= 2.26.1\n- PyTorch \u003e= 1.7.0\n- TensorFlow \u003e= 1.15.0 (若使用 Linux 操作系统，需额外下载 [Tensorflow 1.15.0](https://github.com/neargye-forks/tensorflow/releases)，并将解压出来的 `.so` 文件拷贝至 `Forward/source/third_party/tensorflow/lib` 目录下)\n- Keras HDF5 (从 `Forward/source/third_party/hdf5` 源码构建)\n\n### 项目构建\n\n使用 CMake 进行构建生成 Makefiles 或者 Visual Studio 项目。根据使用目的，Forward 可构建成适用于不同框架的库，如 Fwd-Torch、Fwd-Python-Torch、Fwd-Tf、Fwd-Python-Tf、Fwd-Keras、Fwd-Python-Keras、Fwd-Onnx 和 Fwd-Python-Onnx。\n\n以 Linux 平台构建 Fwd-Tf 为例，\n\n步骤一：克隆项目\n```bash\n1 git clone https://github.com/Tencent/Forward.git\n```\n步骤二：下载 `Tensorflow 1.15.0`（仅在 Linux 平台使用 Tensorflow 框架推理时需要）\n```bash\n1 cd Forward/source/third_party/tensorflow/\n2 wget https://github.com/neargye-forks/tensorflow/releases/download/v1.15.0/libtensorflow-cpu-linux-x86_64-1.15.0.tar.gz\n3 tar -xvf libtensorflow-gpu-linux-x86_64-1.15.0.tar.gz\n```\n步骤三：创建 `build` 文件夹\n```bash\n1 cd ~/Forward/\n2 rm -rf build\n3 mkdir -p build\n4 cd build/\n```\n步骤四：使用 `cmake` 生成构建关系，需指定 `TensorRT_ROOT` 安装路径\n```bash\n1 cmake ..  -DTensorRT_ROOT=\u003cpath_to_TensorRT\u003e -DENABLE_TENSORFLOW=ON -DENABLE_UNIT_TESTS=ON\n```\n步骤五：使用 `make` 构建项目\n```bash\n1 make -j\n```\n步骤六：运行 `unit_test` 验证项目是否构建成功\n```bash\ncd bin/\n./unit_test --gtest_filter=TestTfNodes.*\n\n# 出现已下提示表示项目构建成\n# [       OK ] TestTfNodes.ZeroPadding (347 ms)\n# [----------] 22 tests from TestTfNodes (17555 ms total)\n\n# [----------] Global test environment tear-down\n# [==========] 22 tests from 1 test case ran. (17555 ms total)\n# [  PASSED  ] 22 tests.\n```\n\n更多构建流程可参考 [CMake 构建流程](doc/cn/usages/cmake_build_CN.md) 。\n\n### Forward-Cpp 使用\n\n参考 [Demo for using Forward-Cpp in Linux](demo/fwd_cpp/ReadMe_CN.md)\n\n### Forward-Python 使用\n\n参考 [Demo for using Forward-Python](demo/fwd_py/ReadMe_CN.md)\n\n### Forward-Bert 使用\n\nRefer to [Demo for using Forward-Bert](demo/bert/README_CN.md)\n\n### 更多使用方法\n\n**注意**: 模型输入名可通过模型查看器来查看, 例如用 [Netron](https://github.com/lutzroeder/Netron) 查看。\n\n- [PyTorch 使用说明](doc/cn/usages/torch_usage_CN.md)\n- [TensorFlow 使用说明](doc/cn/usages/tf_usage_CN.md)\n- [Keras 使用说明](doc/cn/usages/keras_usage_CN.md)\n- [ONNX 使用说明](doc/cn/usages/onnx_usage_CN.md)\n\n### Logging 日志\n\nForward 使用 [easylogging++](https://github.com/amrayn/easyloggingpp) 作为日志功能，并使用 `forward_log.conf` 作为日志配置文件。 \n\n- 若工作目录中存在 `forward_log.conf` 文件，Forward 将使用该配置文件，更多内容可参考 [Using-configuration-file](https://github.com/amrayn/easyloggingpp#using-configuration-file)；\n- 若工作目录中不存在 `forward_log.conf` 文件，Forward 将使用默认配置，并将日志记录到 `logs/myeasylog.log` 。\n\n`forward_log.conf` 文件配置样例\n```bash\n* GLOBAL:\n  FORMAT               =  \"[%level] %datetime %fbase(%line): %msg\"\n  FILENAME             =  \"Forward.log\"\n  ENABLED              =  true\n  TO_FILE              =  true\n  TO_STANDARD_OUTPUT   =  true\n  PERFORMANCE_TRACKING =  true\n  MAX_LOG_FILE_SIZE    =  2097152 ## 2MB - Comment starts with two hashes (##)\n  LOG_FLUSH_THRESHOLD  =  100 ## Flush after every 100 logs\n```\n\n## 模型和算子支持\n\n当前 Forward 的模型与算子支持如下所示，如有需要添加更多支持的，欢迎联系添加 Issue 反馈。如需要自行扩展添加支持的，可参考 [开源共建：扩展添加支持操作的流程](doc/cn/usages/add_support_op_CN.md)\n\n### 模型\n\n- [CV 模型](doc/operator_support.md#cv)\n- [语言模型](doc/operator_support.md#nlp)\n- [推荐模型](doc/operator_support.md#recommender)\n\n### 算子\n\n- [PyTorch](doc/operator_support.md#pytorch)\n- [TensorFlow](doc/operator_support.md#tensorflow)\n\n## 参考资料\n1. [推理流程构建过程](doc/cn/usages/forward_workflow_CN.md)\n2. [推理引擎使用方法](doc/cn/usages/inference_engine_usage_CN.md)\n3. [工具与测试](doc/cn/usages/tool_and_test_CN.md)\n4. [常见问题](doc/cn/usages/FAQ_CN.md)\n\n## 贡献   \n\n1. 联系进入开源共建交流讨论群，QQ 群：[776314438](doc/img/qq_group_776314438.png)。\n2. 请参考 [CONTRIBUTING.md](CONTRIBUTING.md) 进行开源共建。\n   \n\u003ctable\u003e\n\u003ctr\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/aster2013\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/5548857?s=460\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAster JIAN\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/yuanzexi\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/14813536?s=460\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eZexi YUAN\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/liao1995\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/12250510?s=460\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAo LI\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/lujq96\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/34331938?s=400\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ePaul LU\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/zhaoyiluo\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/14813536?s=460\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eZhaoyi LUO\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/JettHu\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/35261585?s=400\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJett Hu\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Ryosuke1eep\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/80682051?s=400\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eRyosuke1eep\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n感谢所有贡献者，欢迎更多人加入一起贡献。\n\n## 许可证\n\n详情见 [LISENCE](LICENSE)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencent%2Fforward","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftencent%2Fforward","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencent%2Fforward/lists"}