https://github.com/datvodinh/serve-llm

Serve high throughput and scalable LLM using Ray and vLLM
https://github.com/datvodinh/serve-llm

kubernetes llm ray torch transformers vllm

Last synced: about 17 hours ago
JSON representation

Serve high throughput and scalable LLM using Ray and vLLM

• Host: GitHub
• URL: https://github.com/datvodinh/serve-llm
• Owner: datvodinh
• License: mit
• Created: 2024-08-17T10:17:51.000Z (3 months ago)
• Default Branch: main
• Last Pushed: 2024-08-21T08:31:52.000Z (3 months ago)
• Last Synced: 2024-08-22T04:51:03.110Z (3 months ago)
• Topics: kubernetes, llm, ray, torch, transformers, vllm
• Language: Python
• Homepage:
• Size: 312 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Serve LLM with high throughput and scalable using Ray and vLLM

## Install

```bash

cd serve

make

```

## Serve

- Locally

  

```bash

make serve

```

- Docker

```bash

docker run --gpus all -v .:/code ghcr.io/datvodinh/serve:latest

```

- Kubernetes

```bash

kubectl apply -f k8s/ray-service.yaml

```