{"id":19334060,"url":"https://github.com/stefanolusardi/tiny_inference_engine","last_synced_at":"2025-04-23T00:31:31.442Z","repository":{"id":44701019,"uuid":"432812850","full_name":"StefanoLusardi/tiny_inference_engine","owner":"StefanoLusardi","description":"Client/Server system to perform distributed inference on high load systems. ","archived":false,"fork":false,"pushed_at":"2023-01-23T23:21:59.000Z","size":11583,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2023-03-04T15:29:41.907Z","etag":null,"topics":["ai","cmake","conan","cpp","deep-neural-networks","docker","grpc","inference-client","inference-engine","inference-server","kserve","onnxruntime"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StefanoLusardi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-28T20:04:58.000Z","updated_at":"2023-02-14T17:48:20.000Z","dependencies_parsed_at":"2023-02-13T04:45:29.903Z","dependency_job_id":null,"html_url":"https://github.com/StefanoLusardi/tiny_inference_engine","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StefanoLusardi%2Ftiny_inference_engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StefanoLusardi%2Ftiny_inference_engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StefanoLusardi%2Ftiny_inference_engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StefanoLusardi%2Ftiny_inference_engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StefanoLusardi","download_url":"https://codeload.github.com/StefanoLusardi/tiny_inference_engine/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223906212,"owners_count":17223046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","cmake","conan","cpp","deep-neural-networks","docker","grpc","inference-client","inference-engine","inference-server","kserve","onnxruntime"],"created_at":"2024-11-10T02:56:06.112Z","updated_at":"2024-11-10T02:56:06.558Z","avatar_url":"https://github.com/StefanoLusardi.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tiny inference engine\nWelcome to **tiny inference engine**!  \nThis repository contains a small Client/Server system to perform Machine Learning inference on high load systems.\nThis system allows to scale CPU \u0026 GPU resources serving multiple clients using a single server instance: it is mainly suitable for distributed scenarios and demanding applications.\n\n## Server\nThe server application is written in modern, cross-platform C++ and can run on the 3 major platforms: Windows and Linux with both GPU and CPU support, while on MacOS only CPU is supported.\nIt is also possible to run the server in a Docker container.\n\n## Client\nThe client is available as a library in order to be consumed by any application.\nIt is written in C++ and can run on Windows, Linux and MacOS.\nSee examples for more details.\n\n### Communication Protocol: gRPC \u0026 HTTP\nThe inter process communication layer uses gRPC since it provides better performances over plain HTTP.\nBecause of this it is possible to make inference calls using only the client library.\nThe server however exposes also some HTTP endpoints to provide metrics and allow dynamic tuning at runtime so that other applications (e.g. curl) can be used to interact with it.\n\n### Machine Learning Backend: ONNX Runtime\nCurrently only ONNX Runtime is supported as server backend.\nThe client is completely decoupled from the server backend: client applications must apply preprocessing in order to send properly formatted requests for the model required on server side.\n\n---\n\n## Requirements:\n- Python3 (\u003e 3.8)\n- Conan (\u003e 1.44)\n- CMake (\u003e 3.16)\n- Ninja (\u003e 1.9)\n- C++17 compiler (see specific OS instruction)\n\n### Ubuntu 20.04\n- GCC (\u003e 9.3.0)\n- Clang (\u003e 11.0.0)\n\n```console\napt install python3.8 pip cmake ninja-build conan\npip install conan\n```\n\n### Windows 10\n- Visual Studio 2019\n```console\npip install conan\n```\n\n### MacOS\n- Apple Clang (\u003e 11.0.0)\n- GCC (\u003e 9.3.0)\n```console\npipx install conan\n```\n\n## Build\n```console\ngit clone https://github.com/StefanoLusardi/tiny_inference_engine\nmkdir -p build \u0026\u0026 cd build\ncmake -G Ninja -D CMAKE_BUILD_TYPE=Release ..\ncmake --build . --config Release\ncmake --install . --prefix ../install/\n```\n\n## Unit Tests\n```console\ncmake -G Ninja -D CMAKE_BUILD_TYPE=Release -D TIE_BUILD_CLIENT_UNIT_TESTS=ON -D TIE_BUILD_SERVER_UNIT_TESTS=ON ..\ncmake --build . --config Release\nctest .\n```\n\n## Examples\n```console\ncmake -G Ninja -D CMAKE_BUILD_TYPE=Release -D TIE_BUILD_CLIENT_EXAMPLES=ON ..\ncmake --build . --config Release\ncmake --install . --prefix ../install/\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefanolusardi%2Ftiny_inference_engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefanolusardi%2Ftiny_inference_engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefanolusardi%2Ftiny_inference_engine/lists"}