{"id":19054663,"url":"https://github.com/ssbuild/nn-sdk","last_synced_at":"2025-04-24T03:22:44.378Z","repository":{"id":117215201,"uuid":"401313837","full_name":"ssbuild/nn-sdk","owner":"ssbuild","description":"tensorflow onnx torch infer","archived":false,"fork":false,"pushed_at":"2024-09-23T13:12:08.000Z","size":56,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-18T12:17:52.550Z","etag":null,"topics":["tensorflow-onnx-torch-infer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ssbuild.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-08-30T11:09:45.000Z","updated_at":"2022-06-21T08:26:15.000Z","dependencies_parsed_at":"2023-12-11T13:28:18.579Z","dependency_job_id":"7d314624-cc10-415b-af22-65b32618db09","html_url":"https://github.com/ssbuild/nn-sdk","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbuild%2Fnn-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbuild%2Fnn-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbuild%2Fnn-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbuild%2Fnn-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ssbuild","download_url":"https://codeload.github.com/ssbuild/nn-sdk/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250552796,"owners_count":21449276,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["tensorflow-onnx-torch-infer"],"created_at":"2024-11-08T23:39:19.846Z","updated_at":"2025-04-24T03:22:44.357Z","avatar_url":"https://github.com/ssbuild.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nn_sdk\n\n### nn_sdk 简介\n    基于tensorflow(v1 ,v2),onnx,tensorrt,fasttext神经网络高性能推理接口\n    深度学习模型推理 支持python  c/c++  java\n    pip包地址:\n    https://pypi.org/project/nn-sdk/#description\n\n### 配置\n```commandline\n当前支持开发语言c/c++,python,java\n当前支持推理引擎tensorflow(v1,v2) onnxruntime tensorrt,fasttext 注:tensorrt 7,8测试通过(建议8),目前tensorrt只支持linux系统\n当前支持多子图,支持图多输入多输出, 支持pb [tensorflow 1,2] , ckpt [tensorflow] , trt [tensorrt] , fasttext\n当前支持fastertransformer pb [32精度 相对于传统tf,加速1.9x] ,安装 pip install tf2pb  , 进行模型转换\ntf2pb pb模型转换参考: https://pypi.org/project/tf2pb\n模型加密参考test_aes.py,目前支持tensorflow 1 pb模型 , onnx模型 , tensorrt fasttext模型加密\n推荐环境ubuntu16 ubuntu18 ubuntu20 centos7 centos8 windows系列\npython (test_py.py) , c语言 (test.c) , java语言包 (nn_sdk.java)\n使用过程中遇到问题或者有建议可加qq group: 759163831\n更多使用参见: https://github.com/ssbuild\n\npython 推理demo\nconfig 字段介绍:\n    aes: 加密参考test_aes.py\n    engine: 推理引擎 0: tensorflow , 1: onnx , 2: tensorrt 3: fasttext\n    log_level: 日志类型 0 fatal , 2 error , 4 info , 8 debug\n    model_type: tensorflow 模型类型, 0 pb format , 1 ckpt format\n    fastertransformer:  fastertransformer算子选项, 参考 https://pypi.org/project/tf2pb\n    ConfigProto: tensorflow 显卡配置\n    device_id: GPU id\n    engine_version: 推理引擎主版本 tf 0,1  tensorrt 7 或者 8 , fasttext 0需正确配置\n    graph: 多子图配置 \n        node: 例子: tensorflow 1 input_ids:0 ,  tensorflow 2: input_ids , onnx: input_ids\n        data_type: 节点的类型根据模型配置，对于c++/java支持 int int64 long longlong float double str\n        shape:  尺寸维度\n更新详情:\n2021-10-16 优化 c++/java接口,可预测动态batch\n2021-10-7 增加 fasttext 向量和标签推理\n\n```\n\n### 推理测试\n    \n    \n```python\n#例子: 推理 train 目录model.ckpt\nimport numpy as np\nfrom nn_sdk import csdk_object\n\nconfig = {\n    \"model_dir\": r'./train/model.ckpt',\n    \"aes\": {\n        \"use\": False,\n        \"key\": bytes([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]),\n        \"iv\": bytes([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]),\n    },\n    \"log_level\": 8,\n    'engine': 0,\n    \"device_id\": 0,\n     'tf':{\n        \"ConfigProto\": {\n            \"log_device_placement\": False,\n            \"allow_soft_placement\": True,\n            \"gpu_options\": {\n                \"allow_growth\": True\n            },\n            \"graph_options\":{\n                \"optimizer_options\":{\n                    \"global_jit_level\": 1\n                }\n            },\n        },\n        \"engine_version\": 1, # tensorflow majar version , must be correct.\n        \"model_type\": 1,# 0 pb , 1 ckpt\n        #配置pb模型\n        \"saved_model\":{\n            # model_type为 1 pb , 模型有效,\n            # 模型是否是是否启用saved_model冻结 , 如果是,则 use=True并且配置tags\n            # 普通 freeze pb , use = False\n            'use': False, # 是否启用saved_model\n            'tags': ['serve'],\n            'signature_key': 'serving_default',\n        },\n        \"fastertransformer\":{\n            \"use\": False,\n            \"cuda_version\":\"11.3\", #pip install tf2pb ,支持10.2 11.3 ,\n        }\n    },\n    'onnx': {\n        \"engine_version\": 1,\n    },\n    'trt': {\n        'enable_graph': 0,\n        \"engine_version\": 8,\n    },\n    \"graph\": [\n        {\n            \"input\": [\n                {\"node\": \"x1:0\", \"dtype\": \"float32\"},\n                {\"node\": \"x2:0\", \"dtype\": \"float32\"},\n            ],\n            \"output\": [\n                {\"node\": \"pred_ids:0\", \"dtype\": \"float32\"},\n            ],\n        }\n    ]\n}\n\nbatch_size = 1\nseq_length = 10\nx1 = np.random.randint(1,10,size=(batch_size,seq_length))\nx2 = np.random.randint(1,10,size=(batch_size,seq_length))\n\ninputs = (x1,x2)\n\nsdk_inf = csdk_object(config)\nif sdk_inf.valid():\n    net_stage = 0\n    ret, out = sdk_inf.process(net_stage, *inputs)\n\n    print(x1)\n    print(x2)\n    print(ret)\n    print(out)\n    sdk_inf.close()\n```\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssbuild%2Fnn-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fssbuild%2Fnn-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssbuild%2Fnn-sdk/lists"}