https://github.com/g1ibby/llm-deploy

Tool to manage ollama model on vast.ai
https://github.com/g1ibby/llm-deploy

litellm llm ollama openai-api vastai

Last synced: 2 months ago
JSON representation

Tool to manage ollama model on vast.ai

• Host: GitHub
• URL: https://github.com/g1ibby/llm-deploy
• Owner: g1ibby
• License: mit
• Created: 2023-12-22T08:24:35.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2024-04-19T05:28:00.000Z (over 1 year ago)
• Last Synced: 2025-04-10T05:43:14.410Z (6 months ago)
• Topics: litellm, llm, ollama, openai-api, vastai
• Language: Python
• Homepage:
• Size: 90.8 KB
• Stars: 17
• Watchers: 1
• Forks: 1
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
### LLM deploy - tool to manage your LLMs on vast.ai servers

#### Introduction

"llm-deploy" is a Python tool for deploying and managing large language models (LLMs) on vast.ai using ollama. It uses Typer for command-line interactions.

#### Requirements

- Python 3.11 or later

- Poetry for dependency management

#### Installation

1. Clone the repository or download the source code.

2. Navigate to the project directory.

3. Run `poetry install` to install dependencies.

#### Configuration

Create a `llms.yaml` file with your model configurations, like this:

```yaml

models:

  llama:

    model: "phi:2.7b-chat-v2-q5_K_M"

    priority: low

```

Copy file `env.sh.dist` to `env.sh` and set your keys there. 

Run `source env.sh` 

### Usage

#### Config-Mode Commands:

- Apply LLMs Configuration:

`poetry run llm-deploy apply`

    Applies configurations from llms.yaml.

- Destroy LLMs Configuration:

`poetry run llm-deploy destroy`

    Reverts configurations and destroys created instances based on the current state.

#### Manual-Mode Commands:

- List Current Instances:

`poetry run llm-deploy infra ls`

    Lists all current instances.

- Create New Instance (Manual):

`poetry run llm-deploy infra create --gpu-memory  --disk `

    Manually creates a new instance with specified GPU memory, disk space, and public IP option.

- Remove an Instance:

`poetry run llm-deploy infra destroy `

    Removes an instance by ID.

- Show Instance Details:

`poetry run llm-deploy infra inspect `

    Shows details of an instance.

- Retrieve Logs for an Instance:

`poetry run llm-deploy logs  --max-logs `

    Retrieves and displays logs for a specified instance.

- Deploy a Model to an Instance:

`poetry run llm-deploy model deploy  `

    Deploys a specified model to an instance.

- Remove a Model from an Instance:

`poetry run llm-deploy model remove  `

    Removes a deployed model from an instance.

- List Models on Instances:

`poetry run llm-deploy model ls`

    Lists models deployed across instances.