https://github.com/scaleoutsystems/tee-serve
Lightweight model serving in Trusted Execution Environments.
https://github.com/scaleoutsystems/tee-serve
ai mlops modelserving tee
Last synced: 8 months ago
JSON representation
Lightweight model serving in Trusted Execution Environments.
- Host: GitHub
- URL: https://github.com/scaleoutsystems/tee-serve
- Owner: scaleoutsystems
- Created: 2021-11-23T10:43:24.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2022-01-12T13:37:19.000Z (over 3 years ago)
- Last Synced: 2025-01-10T18:23:31.542Z (9 months ago)
- Topics: ai, mlops, modelserving, tee
- Language: Dockerfile
- Homepage: https://medium.com/p/f17f26a65646
- Size: 73.2 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Lightweight model serving in TEE
Lightweight model serving in [Intel SGX TEE](https://www.intel.com/content/www/us/en/developer/tools/software-guard-extensions/overview.html) using [Gramine](https://grapheneproject.io) and [TensorFlow lite](https://www.tensorflow.org/lite) C++ API. For more details please refer to the companion blog post: https://medium.com/p/f17f26a65646.
## Table of contents
- [Lightweight model serving in TEE](#lightweight-model-serving-in-tee)
- [Table of contents](#table-of-contents)
- [Quickstart](#quickstart)
- [Building the binaries](#building-the-binaries)
- [Running in direct mode (without SGX)](#running-in-direct-mode-without-sgx)
- [Running in SGX](#running-in-sgx)
- [Getting access to SGX](#getting-access-to-sgx)
## Quickstart
The only prerequisite to run this proof of concept is https://docker.io. Once you have installed docker you can clone this repository, locate into it and launch the environment as if follows.
```console
$ bin/launch.sh
```
This is going to build and start a container with all of the required dependencies to build and run the codebase.
## Building the binaries
To build the `server` and the `client` binaries you can run:
```console
$ bin/build.sh
```
This command is going to build the binaries, sign the `server` and generate other necessary Gramine files.
## Running in direct mode (without SGX)
The `gramine-direct` command can be used for testing purposed (or if a SGX machine is not available). To start the server in this manner you can run:
```console
$ gramine-direct server resources/plain/model.tflite
```
> **Note** `server` is the server executable and `resources/plain/model.tflite` is a TensorFlow lite model.
To test the server you can open a new console and run e.g. `./client 0.5`.
## Running in SGX
The `gramine-sgx` command can be used run the server in the SGX enclave as it follows.
```console
$ gramine-sgx server resources/model.tflite
```
> **Note** `server` is the server executable and `resources/model.tflite` is an encrypted TensorFlow lite model.
To test the server you can open a new console and run e.g. `./client 0.5`.
## Getting access to SGX
Azure offers Intel SGX instances that you can get on demand: https://azure.microsoft.com/en-us/solutions/confidential-compute.