https://github.com/deeprec-ai/serving
A high-performance serving system for DeepRec based on TensorFlow Serving.
https://github.com/deeprec-ai/serving
Last synced: 10 months ago
JSON representation
A high-performance serving system for DeepRec based on TensorFlow Serving.
- Host: GitHub
- URL: https://github.com/deeprec-ai/serving
- Owner: DeepRec-AI
- License: apache-2.0
- Created: 2022-04-08T07:13:06.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2023-11-15T09:44:48.000Z (about 2 years ago)
- Last Synced: 2025-04-05T06:11:14.875Z (10 months ago)
- Language: C++
- Homepage:
- Size: 4.84 MB
- Stars: 18
- Watchers: 1
- Forks: 19
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# DeepRec Serving
DeepRec Serving is a high-performance serving system for DeepRec based on TensorFlow Serving.
DeepRec Serving could highly improve performance and cpu/gpu utilization in inference, such as SessionGroup, CUDA multi-stream, etc.
Few features in DeepRec Serving:
- Support SessionGroup which is shared-variable (only variables shared) architecture for mutliple session in serving process.
- Support CUDA Multiple Stream, could highly improve QPS and GPU Utilization in GPU Inference.
## Installation
### Prepare for build
**CPU Dev Docker**
| GCC Version | Python Version | IMAGE |
| ----------- | -------------- | --------------------------------------------------------- |
| 9.4.0 | 3.8.10 | alideeprec/deeprec-build:deeprec-dev-cpu-py38-ubuntu20.04 |
**GPU(cuda11.6) Dev Docker**
| GCC Version | Python Version | CUDA VERSION | IMAGE |
| ----------- | -------------- | ------------ | --------------------------------------------------------------- |
| 9.4.0 | 3.8.10 | CUDA 11.6.2 | alideeprec/deeprec-build:deeprec-dev-gpu-py38-cu116-ubuntu20.04 |
### Build from source
Develop Branch: master, Latest Release Branch: deeprec2302
**Build Package Builder-CPU**
```bash
bazel build -c opt tensorflow_serving/...
```
**Build CPU Package Builder with OneDNN + Eigen Threadpool**
```bash
bazel build -c opt --config=mkl_threadpool --define build_with_mkl_dnn_v1_only=true tensorflow_serving/...
```
**Build Package Builder-GPU**
```bash
bazel build -c opt --config=cuda tensorflow_serving/...
```
**Build Package**
```bash
bazel-bin/tensorflow_serving/tools/pip_package/build_pip_package /tmp/tf_serving_client_whl
```
**Server Bin**
Server Bin would generated in following directory:
```bash
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
```
## More details
* [SessionGroup](https://github.com/DeepRec-AI/DeepRec/blob/main/docs/docs_en/SessionGroup.md)
* [CUDA MultiStream](https://github.com/DeepRec-AI/DeepRec/blob/main/docs/docs_en/GPU-MultiStream.md)
* [Device Placement Optimization](https://github.com/DeepRec-AI/DeepRec/blob/main/docs/docs_en/Device-Placement.md)