https://github.com/comet-ml/blog-serving-hugging-face-models
https://github.com/comet-ml/blog-serving-hugging-face-models
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/comet-ml/blog-serving-hugging-face-models
- Owner: comet-ml
- Created: 2021-04-28T18:22:53.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2021-04-28T18:26:43.000Z (about 5 years ago)
- Last Synced: 2025-04-03T02:54:46.446Z (about 1 year ago)
- Language: Shell
- Size: 10.1 MB
- Stars: 20
- Watchers: 7
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# How to easily and efficiently deploy Hugging Face models
Code repository to reproduce the benchmarking results presented [here](https://www.comet.ml/site/how-to-10x-throughput-when-serving-hugging-face-models-without-a-gpu/)
The repo is broken down into 2 sections:
* python-api: The Python inference services that we tested:
* baseline: Baseline inference service using default parameters - This is **not** optimized
* model-hardware-optimized: Optimized inference service for DistilBert - This is the most optimized inference service
* model-optimized: Optimized inference service that support both Bert and DistilBert models as well as optional quantization
* load-testing: Utilities to run performance benchmarks
## How to run the benchmarks
Given all our benchmarks are run on GCP, you will need to have a Google Cloud project.
Running the benchmarks is done in three steps:
1. Create the docker images for each Python API:
```bash
cd python-api
sh create_docker_images.sh
```
2. Deploy a virtual machine for the python api we wish to test:
```bash
cd load_testing/machine_provisioning
sh baseline_benchmark.sh GOOGLE_PROJECT_ID>
```
3. We will now create a virtual machine from which to run our load testing software:
```bash
cd load_testing/machine_provisioning
sh locust_machine.sh
```
4. The last part of the script run in step 3 is to connect via ssh to the virtual machine. In this terminal we can run our load testing script:
```bash
cd /locust
sh run_load_test.sh
```