{"id":13436712,"url":"https://github.com/nndeploy/nndeploy","last_synced_at":"2025-12-14T21:55:35.980Z","repository":{"id":191205262,"uuid":"676114409","full_name":"nndeploy/nndeploy","owner":"nndeploy","description":"nndeploy是一款模型端到端部署框架。以多端推理以及基于有向无环图模型部署为基础，致力为用户提供跨平台、简单易用、高性能的模型部署体验。","archived":false,"fork":false,"pushed_at":"2024-09-12T03:31:52.000Z","size":14006,"stargazers_count":584,"open_issues_count":5,"forks_count":88,"subscribers_count":23,"default_branch":"main","last_synced_at":"2024-09-13T05:06:17.908Z","etag":null,"topics":["ascend","easy-to-use","hpc","mnn","model-deployment","multi-inference","openvino","out-of-box-model","parallel","rknn","tensorrt","yolo"],"latest_commit_sha":null,"homepage":"https://nndeploy-zh.readthedocs.io/zh/latest/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nndeploy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-08T13:13:25.000Z","updated_at":"2024-09-12T02:07:15.000Z","dependencies_parsed_at":"2024-03-09T14:46:24.391Z","dependency_job_id":"82ec5f4f-f632-49f1-9041-69d6583a8aa6","html_url":"https://github.com/nndeploy/nndeploy","commit_stats":null,"previous_names":["alwaysssssss/nndeploy","deployai/nndeploy","nndeploy/nndeploy"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nndeploy%2Fnndeploy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nndeploy%2Fnndeploy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nndeploy%2Fnndeploy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nndeploy%2Fnndeploy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nndeploy","download_url":"https://codeload.github.com/nndeploy/nndeploy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244310445,"owners_count":20432542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ascend","easy-to-use","hpc","mnn","model-deployment","multi-inference","openvino","out-of-box-model","parallel","rknn","tensorrt","yolo"],"created_at":"2024-07-31T03:00:51.487Z","updated_at":"2025-12-14T21:55:35.973Z","avatar_url":"https://github.com/nndeploy.png","language":"C++","readme":"[English](README_EN.md) | 简体中文\n\n\u003ch3 align=\"center\"\u003e\nnndeploy：一款简单易用和高性能的AI部署框架\n\u003c/h3\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://github.com/nndeploy/nndeploy/actions/workflows/linux.yml\"\u003e\n  \u003cimg src=\"https://github.com/nndeploy/nndeploy/actions/workflows/linux.yml/badge.svg\" alt=\"Linux\" style=\"height: 16px;\"\u003e\n\u003c/a\u003e\n \u003ca href=\"https://github.com/nndeploy/nndeploy/actions/workflows/windows.yml\"\u003e\n  \u003cimg src=\"https://github.com/nndeploy/nndeploy/actions/workflows/windows.yml/badge.svg\" alt=\"Windows\" style=\"height: 16px;\"\u003e\n\u003c/a\u003e\n \u003ca href=\"https://github.com/nndeploy/nndeploy/actions/workflows/android.yml\"\u003e\n  \u003cimg src=\"https://github.com/nndeploy/nndeploy/actions/workflows/android.yml/badge.svg\" alt=\"Android\" style=\"height: 16px;\"\u003e\n\u003c/a\u003e\n \u003ca href=\"https://github.com/nndeploy/nndeploy/actions/workflows/macos.yml\"\u003e\n  \u003cimg src=\"https://github.com/nndeploy/nndeploy/actions/workflows/macos.yml/badge.svg\" alt=\"macOS\" style=\"height: 16px;\"\u003e\n\u003c/a\u003e\n \u003ca href=\"https://github.com/nndeploy/nndeploy/actions/workflows/ios.yml\"\u003e\n  \u003cimg src=\"https://github.com/nndeploy/nndeploy/actions/workflows/ios.yml/badge.svg\" alt=\"iOS\" style=\"height: 16px;\"\u003e\n\u003c/a\u003e\n \u003c!-- \u003ca href=\"https://pepy.tech/projects/nndeploy\"\u003e\n  \u003cimg src=\"https://static.pepy.tech/personalized-badge/nndeploy?period=total\u0026units=INTERNATIONAL_SYSTEM\u0026left_color=BLACK\u0026right_color=GREEN\u0026left_text=downloads\" alt=\"PyPI Downloads\" style=\"height: 16px;\"\u003e\n\u003c/a\u003e --\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://nndeploy-zh.readthedocs.io/zh-cn/latest/\"\u003e\u003cb\u003e文档\u003c/b\u003e\u003c/a\u003e \n| \u003ca href=\"https://deepwiki.com/nndeploy/nndeploy\"\u003e\u003cb\u003eAsk DeepWiki\u003c/b\u003e\u003c/a\u003e\n| \u003ca href=\"docs/zh_cn/knowledge_shared/wechat.md\"\u003e\u003cb\u003e微信\u003c/b\u003e\u003c/a\u003e \n| \u003ca href=\"https://discord.gg/9rUwfAaMbr\"\u003e\u003cb\u003eDiscord\u003c/b\u003e\u003c/a\u003e \n\u003c!-- | \u003ca href=\"https://www.zhihu.com/column/c_1690464325314240512\"\u003e\u003cb\u003e知乎\u003c/b\u003e\u003c/a\u003e  --\u003e\n\u003c!-- | \u003ca href=\"https://www.bilibili.com/video/BV1HU7CznE39/?spm_id_from=333.1387.collection.video_card.click\u0026vd_source=c5d7760172919cd367c00bf4e88d6f57\"\u003e\u003cb\u003e哔哩哔哩\u003c/b\u003e\u003c/a\u003e  --\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"docs/image/workflow.png\"\u003e\n    \u003cimg alt=\"nndeploy\" src=\"docs/image/workflow/worflow_llm.gif\" width=100%\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\n---\n\n## 介绍\n\nnndeploy 是一款简单易用和高性能的 AI 部署框架。解决的是 AI 算法在端侧部署的问题，包含桌面端（Windows、macOS）、移动端（Android、iOS）、边缘计算设备（NVIDIA Jetson、Ascend310B、RK 等）以及单机服务器（RTX 系列、T4、Ascend310P 等），**基于可视化工作流和多端推理，可让 AI 算法在上述平台和硬件更高效、更高性能的落地。**\n\n**针对10B以上的大模型（如大语言模型和 AIGC 生成模型），nndeploy 适合作为一款可视化工作流工具。**\n\n### **简单易用**\n\n- **可视化工作流**：拖拽节点即可部署 AI 算法，参数实时可调，效果一目了然。\n- **自定义节点**：支持 Python/C++自定义节点，无论是用 Python 实现预处理，还是用 C++/CUDA 编写高性能节点，均可无缝集成到与可视化工作流。\n- **一键部署**：工作流支持导出为 JSON，可通过 C++/Python API 调用，适用于 Linux、Windows、macOS、Android 等平台\n\n  \u003ctable cellpadding=\"5\" cellspacing=\"0\" border=\"1\"\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e桌面端搭建AI工作流\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://github.com/nndeploy/nndeploy/blob/main/app/android/README.md\"\u003e移动端部署\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/image/workflow/worflow_segment_rmbg.gif\" width=\"500px\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/image/android_app/app-seg-result.jpg\" width=\"100px\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003c/table\u003e\n\n### **高性能**\n\n- **并行优化**：支持串行、流水线并行、任务并行等执行模式\n- **内存优化**：零拷贝、内存池、内存复用等优化策略\n- **高性能优化**：内置 C++/CUDA/Ascend C/SIMD 等优化实现的节点\n- **多端推理**：一套工作流适配多端推理，深度集成 13 种主流推理框架，全面覆盖云端服务器、桌面应用、移动设备、边缘计算等全平台部署场景。框架支持灵活选择推理引擎，可按需编译减少依赖，同时支持接入自定义推理框架的独立运行模式。\n\n  | 推理框架                                                                         | 状态 |\n  | :------------------------------------------------------------------------------- | :--- |\n  | [ONNXRuntime](https://github.com/microsoft/onnxruntime)                          | ✅    |\n  | [TensorRT](https://github.com/NVIDIA/TensorRT)                                   | ✅    |\n  | [OpenVINO](https://github.com/openvinotoolkit/openvino)                          | ✅    |\n  | [MNN](https://github.com/alibaba/MNN)                                            | ✅    |\n  | [TNN](https://github.com/Tencent/TNN)                                            | ✅    |\n  | [ncnn](https://github.com/Tencent/ncnn)                                          | ✅    |\n  | [CoreML](https://github.com/apple/coremltools)                                   | ✅    |\n  | [AscendCL](https://www.hiascend.com/zh/)                                         | ✅    |\n  | [RKNN](https://www.rock-chips.com/a/cn/downloadcenter/BriefDatasheet/index.html) | ✅    |\n  | [SNPE](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk)   | ✅    |\n  | [TVM](https://github.com/apache/tvm)                                             | ✅    |\n  | [PyTorch](https://pytorch.org/)                                                  | ✅    |\n  | [nndeploy内部推理子模块](docs/zh_cn/inference/README_INFERENCE.md)               | ✅    |\n\n### **开箱即用的算法**\n\n已部署多类 AI 模型，并开发 100+可视化节点，实现开箱即用体验。随着部署节点数量的增加，节点库的复用性不断提升，这将显著降低后续算法部署的开发成本。我们还将持续部署更多具有实用价值的算法。\n\n| Application Scenario       | Available Models                                                                | Remarks                                                                         |\n| -------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |\n| **Large Language Models**  | **QWen-2.5**, **QWen-3**                                                        | Support small B models                                                          |\n| **Image/Video Generation** | Stable Diffusion 1.5, Stable Diffusion XL, Stable Diffusion 3, HunyuanDiT, etc. | Support text-to-image, image-to-image, image inpainting, based on **diffusers** |\n| **Face Swapping**          | **deep-live-cam**                                                               |                                                                                 |\n| **OCR**                    | **Paddle OCR**                                                                  |                                                                                 |\n| **Object Detection**       | **YOLOv5, YOLOv6, YOLOv7, YOLOv8, YOLOv11, YOLOx**                              |                                                                                 |\n| **Object Tracking**        | FairMot                                                                         |                                                                                 |\n| **Image Segmentation**     | RBMGv1.4, PPMatting, **Segment Anything**                                       |                                                                                 |\n| **Classification**         | ResNet, MobileNet, EfficientNet, PPLcNet, GhostNet, ShuffleNet, SqueezeNet      |                                                                                 |\n| **API Services**           | OPENAI, DeepSeek, Moonshot                                                      | Support LLM and AIGC services                                                   |\n\n\u003e 更多查看[已部署模型列表详解](docs/zh_cn/quick_start/model_list.md)\n\n## 快速开始\n\n- **步骤一：安装**\n\n  ```bash\n  pip install --upgrade nndeploy\n  ```\n\n- **步骤二：启动可视化界面**\n\n  ```bash\n  # 方式一：命令行\n  nndeploy-app --port 8000\n  # 方式二：代码启动\n  cd path/to/nndeploy\n  python app.py --port 8000\n  ```\n\n  启动成功后，打开 http://localhost:8000 即可访问工作流编辑器。在这里，你可以拖拽节点、调整参数、实时预览效果，所见即所得。\n\n  \u003cp align=\"left\"\u003e\n    \u003cpicture\u003e\n      \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"quick_start.gif\"\u003e\n      \u003cimg alt=\"nndeploy\" src=\"docs/image/quick_start.gif\" width=100%\u003e\n    \u003c/picture\u003e\n  \u003c/p\u003e\n\n- **步骤三：保存并加载运行**\n\n  在可视化界面中搭建、调试完成后，点击保存，工作流会导出 JSON 文件，文件中封装了所有的处理流程。你可以用以下两种方式在**生产环境**中运行：\n\n  - 方式一：命令行运行\n\n    用于调试\n\n    ```bash\n    # Python CLI\n    nndeploy-run-json --json_file path/to/workflow.json\n    # C++ CLI\n    nndeploy_demo_run_json --json_file path/to/workflow.json\n    ```\n\n  - 方式 2：在 Python/C++ 代码中加载运行\n\n    可以将 JSON 文件集成到你现有的 Python 或 C++ 项目中，以下是一个加载和运行 LLM 工作流的示例代码：\n\n    - Python API 加载运行 LLM 工作流\n      ```Python\n      graph = nndeploy.dag.Graph(\"\")\n      graph.remove_in_out_node()\n      graph.load_file(\"path/to/llm_workflow.json\")\n      graph.init()\n      input = graph.get_input(0)\n      text = nndeploy.tokenizer.TokenizerText()\n      text.texts_ = [ \"\u003c|im_start|\u003euser\\nPlease introduce NBA superstar Michael Jordan\u003c|im_end|\u003e\\n\u003c|im_start|\u003eassistant\\n\" ]\n      input.set(text)\n      status = graph.run()\n      output = graph.get_output(0)\n      result = output.get_graph_output()\n      graph.deinit()\n      ```\n    - C++ API 加载运行 LLM 工作流\n      ```C++\n      std::shared_ptr\u003cdag::Graph\u003e graph = std::make_shared\u003cdag::Graph\u003e(\"\");\n      base::Status status = graph-\u003eloadFile(\"path/to/llm_workflow.json\");\n      graph-\u003eremoveInOutNode();\n      status = graph-\u003einit();\n      dag::Edge* input = graph-\u003egetInput(0);\n      tokenizer::TokenizerText* text = new tokenizer::TokenizerText();\n      text-\u003etexts_ = {\n          \"\u003c|im_start|\u003euser\\nPlease introduce NBA superstar Michael Jordan\u003c|im_end|\u003e\\n\u003c|im_start|\u003eassistant\\n\"};\n      input-\u003eset(text, false);\n      status = graph-\u003erun();\n      dag::Edge* output = graph-\u003egetOutput(0);\n      tokenizer::TokenizerText* result =\n          output-\u003egetGraphOutput\u003ctokenizer::TokenizerText\u003e();\n      status = graph-\u003edeinit();\n      ```\n\n\u003e 要求 Python 3.10+，默认包含 ONNXRuntime、MNN，更多推理后端请采用开发者模式。\n\n## 文档\n\n- [编译](docs/zh_cn/quick_start/build.md)\n- [可视化工作流](docs/zh_cn/quick_start/workflow.md)\n- [最佳实践](docs/zh_cn/quick_start/deploy.md)\n- [Python 自定义节点开发手册](docs/zh_cn/quick_start/plugin_python.md)\n- [C++自定义节点开发手册](docs/zh_cn/quick_start/plugin.md)\n- [接入新推理框架](docs/zh_cn/developer_guide/how_to_support_new_inference.md)\n  \n## 性能测试\n\n测试环境：Ubuntu 22.04，i7-12700，RTX3060\n\n- **流水线并行加速**。以 YOLOv11s 端到端工作流总耗时，串行 vs 流水线并行\n\n  \u003cimg src=\"docs/image/workflow/yolo_performance.png\" width=\"60%\"\u003e\n\n  | 运行方式\\推理引擎 | ONNXRuntime | OpenVINO  | TensorRT  |\n  | ----------------- | ----------- | --------- | --------- |\n  | 串行              | 54.803 ms   | 34.139 ms | 13.213 ms |\n  | 流水线并行        | 47.283 ms   | 29.666 ms | 5.681 ms  |\n  | 性能提升          | 13.7%       | 13.1%     | 57%       |\n\n- **任务并行加速**。组合任务(分割 RMBGv1.4+检测 YOLOv11s+分类 ResNet50)的端到端总耗时，串行 vs 任务并行\n\n  \u003cimg src=\"docs/image/workflow/rmbg_yolo_resnet.png\" width=\"60%\"\u003e\n\n  | 运行方式\\推理引擎 | ONNXRuntime | OpenVINO   | TensorRT  |\n  | ----------------- | ----------- | ---------- | --------- |\n  | 串行              | 654.315 ms  | 489.934 ms | 59.140 ms |\n  | 任务并行          | 602.104 ms  | 435.181 ms | 51.883 ms |\n  | 性能提升          | 7.98%       | 11.2%      | 12.2%     |\n\n## 下一步计划\n\n- [工作流生态](https://github.com/nndeploy/nndeploy/issues/191)\n- [端侧大模型推理](https://github.com/nndeploy/nndeploy/issues/161)\n- [架构优化](https://github.com/nndeploy/nndeploy/issues/189)\n- [AI Box](https://github.com/nndeploy/nndeploy/issues/190)\n\n## 联系我们\n\n- 如果你热爱开源、喜欢折腾，不论是出于学习目的，亦或是有更好的想法，欢迎加入我们\n\n- 微信：Always031856（欢迎加好友，进群交流，备注：nndeploy\\_姓名）\n\n## 致谢\n\n- 感谢以下项目：[TNN](https://github.com/Tencent/TNN)、[FastDeploy](https://github.com/PaddlePaddle/FastDeploy)、[opencv](https://github.com/opencv/opencv)、[CGraph](https://github.com/ChunelFeng/CGraph)、[tvm](https://github.com/apache/tvm)、[mmdeploy](https://github.com/open-mmlab/mmdeploy)、[FlyCV](https://github.com/PaddlePaddle/FlyCV)、[oneflow](https://github.com/Oneflow-Inc/oneflow)、[flowgram.ai](https://github.com/bytedance/flowgram.ai)、[deep-live-cam](https://github.com/hacksider/Deep-Live-Cam)。\n\n- 感谢[HelloGithub](https://hellogithub.com/repository/nndeploy/nndeploy)推荐\n\n  \u003ca href=\"https://hellogithub.com/repository/314bf8e426314dde86a8c62ea5869cb7\" target=\"_blank\"\u003e\u003cimg src=\"https://abroad.hellogithub.com/v1/widgets/recommend.svg?rid=314bf8e426314dde86a8c62ea5869cb7\u0026claim_uid=mu47rJbh15yQlAs\" alt=\"Featured｜HelloGitHub\" style=\"width: 250px; height: 54px;\" width=\"250\" height=\"54\" /\u003e\u003c/a\u003e\n\n## 贡献者\n\n\u003ca href=\"https://github.com/nndeploy/nndeploy/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=nndeploy/nndeploy\" /\u003e\n\u003c/a\u003e\n\n[![Star History Chart](https://api.star-history.com/svg?repos=nndeploy/nndeploy\u0026type=Date)](https://star-history.com/#nndeploy/nndeploy)\n","funding_links":[],"categories":["Deployment and Serving","C++","Software"],"sub_categories":["[Tools](#tools-1)"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnndeploy%2Fnndeploy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnndeploy%2Fnndeploy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnndeploy%2Fnndeploy/lists"}