https://github.com/premai-io/serverless-examples
🚀 End-to-end examples and analysis of deploying LLMs serverless using Modal, Runpod, and Beam
https://github.com/premai-io/serverless-examples
Last synced: 10 months ago
JSON representation
🚀 End-to-end examples and analysis of deploying LLMs serverless using Modal, Runpod, and Beam
- Host: GitHub
- URL: https://github.com/premai-io/serverless-examples
- Owner: premAI-io
- Created: 2024-03-13T16:08:06.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-03-25T16:09:15.000Z (about 2 years ago)
- Last Synced: 2025-05-01T07:37:49.613Z (about 1 year ago)
- Language: Python
- Homepage:
- Size: 69.3 KB
- Stars: 27
- Watchers: 6
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Serverless LLM Deployment Examples
Welcome to the repository containing a set of hackable examples for serverless deployement of Large Language Models (LLMs). Here, we explore and analyze three services: [Modal Labs](https://modal.com/), [Beam Cloud](https://www.beam.cloud/), and [Runpod](https://www.runpod.io/), each abstracting out the deployment process at different levels.
| Service | Blogpost | Implementation |
|---------------|-------------------------------------------------------------|------------------|
| Modal Labs | [Tutorial Blogpost](https://blog.premai.io/serverless-deployment-using-huggingface-and-modal/) | [Modal Labs Deployment](/deploy_modal/) |
| Beam Cloud | [Tutorial Blogpost](https://blog.premai.io/deploy-google-gemma-serverless-using-beam-cloud/) | [Beam Cloud Deployment](/deploy_beam/) |
| RunPod | [Tutorial Blogpost](https://blog.premai.io/serverless-deploy-mistral-2-7b-runpod/) | [RunPod Deployment](/deploy_runpod/) |
We provide blog posts for each service, as well as dedicated repositories containing full code examples and instructions on how to run and test them.
## Test Deployed Model
If you've followed our tutorials and deployed your models using any of the mentioned services, you can test the deployments from here. Please note that testing is currently available only for streaming. However, if you want to make changes, feel free to do so. Before getting started, please install the requirements from here.
```bash
pip install -r requirements.txt
```
Now, assuming you deployed your model using either of the services, you can run `test.py` as shown below:
For Modal and Beam Cloud:
```bash
python3 test.py modal --url --prompt "hello"
```
For RunPod, you also need to provide the service ID:
```bash
python3 test.py modal \
--url \
--prompt "hello" \
--runpod_id
```
Replace `` with a value that looks like this: `80r0eh3jel99f8` (this is an example ID).