{"id":19621615,"url":"https://github.com/deeprec-ai/serving","last_synced_at":"2025-04-28T03:32:22.272Z","repository":{"id":38475705,"uuid":"479284370","full_name":"DeepRec-AI/serving","owner":"DeepRec-AI","description":"A high-performance serving system for DeepRec based on TensorFlow Serving.","archived":false,"fork":false,"pushed_at":"2023-11-15T09:44:48.000Z","size":5075,"stargazers_count":18,"open_issues_count":0,"forks_count":19,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-05T06:11:14.875Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"tensorflow/serving","license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DeepRec-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-04-08T07:13:06.000Z","updated_at":"2024-10-22T09:18:14.000Z","dependencies_parsed_at":"2023-11-15T10:30:23.792Z","dependency_job_id":"fd33bf89-3a9b-4791-a1ae-a0d7148301cd","html_url":"https://github.com/DeepRec-AI/serving","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepRec-AI%2Fserving","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepRec-AI%2Fserving/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepRec-AI%2Fserving/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepRec-AI%2Fserving/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DeepRec-AI","download_url":"https://codeload.github.com/DeepRec-AI/serving/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251246283,"owners_count":21558762,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T11:23:44.465Z","updated_at":"2025-04-28T03:32:19.614Z","avatar_url":"https://github.com/DeepRec-AI.png","language":"C++","readme":"# DeepRec Serving\n\nDeepRec Serving is a high-performance serving system for DeepRec based on TensorFlow Serving.\nDeepRec Serving could highly improve performance and cpu/gpu utilization in inference, such as SessionGroup, CUDA multi-stream, etc.\n\nFew features in DeepRec Serving:\n-   Support SessionGroup which is shared-variable (only variables shared) architecture for mutliple session in serving process.\n-   Support CUDA Multiple Stream, could highly improve QPS and GPU Utilization in GPU Inference.\n\n## Installation\n\n### Prepare for build\n\n**CPU Dev Docker**\n\n| GCC Version | Python Version |                           IMAGE                           |\n| ----------- | -------------- | --------------------------------------------------------- |\n|   9.4.0     |    3.8.10      | alideeprec/deeprec-build:deeprec-dev-cpu-py38-ubuntu20.04 |\n\n**GPU(cuda11.6) Dev Docker**\n\n| GCC Version | Python Version | CUDA VERSION |                           IMAGE                                 |\n| ----------- | -------------- | ------------ | --------------------------------------------------------------- |\n|    9.4.0    |    3.8.10      | CUDA 11.6.2  | alideeprec/deeprec-build:deeprec-dev-gpu-py38-cu116-ubuntu20.04 |\n\n### Build from source\n\nDevelop Branch: master, Latest Release Branch: deeprec2302\n\n**Build Package Builder-CPU**\n\n```bash\nbazel build -c opt tensorflow_serving/...\n```\n\n**Build CPU Package Builder with OneDNN + Eigen Threadpool**\n\n```bash\nbazel build  -c opt --config=mkl_threadpool --define build_with_mkl_dnn_v1_only=true tensorflow_serving/...\n```\n\n**Build Package Builder-GPU**\n\n```bash\nbazel build -c opt --config=cuda tensorflow_serving/...\n```\n\n**Build Package**\n\n```bash\nbazel-bin/tensorflow_serving/tools/pip_package/build_pip_package /tmp/tf_serving_client_whl\n```\n\n**Server Bin**\n\nServer Bin would generated in following directory:\n```bash\nbazel-bin/tensorflow_serving/model_servers/tensorflow_model_server\n```\n## More details \n\n* [SessionGroup](https://github.com/DeepRec-AI/DeepRec/blob/main/docs/docs_en/SessionGroup.md)\n* [CUDA MultiStream](https://github.com/DeepRec-AI/DeepRec/blob/main/docs/docs_en/GPU-MultiStream.md)\n* [Device Placement Optimization](https://github.com/DeepRec-AI/DeepRec/blob/main/docs/docs_en/Device-Placement.md)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeprec-ai%2Fserving","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeeprec-ai%2Fserving","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeprec-ai%2Fserving/lists"}