https://github.com/fly-apps/self-host-ollama
A simple repo for deploying Ollama on Fly.io.
https://github.com/fly-apps/self-host-ollama
Last synced: 4 months ago
JSON representation
A simple repo for deploying Ollama on Fly.io.
- Host: GitHub
- URL: https://github.com/fly-apps/self-host-ollama
- Owner: fly-apps
- Created: 2024-07-26T23:05:18.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-09-09T22:14:25.000Z (almost 2 years ago)
- Last Synced: 2025-01-17T18:53:34.286Z (over 1 year ago)
- Size: 4.88 KB
- Stars: 26
- Watchers: 6
- Forks: 5
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Self-hosted Ollama app on Fly.io
A simple repo for deploying Ollama on Fly.io.
## Getting started
If you don't already have a Fly.io account, do that first (https://fly.io/app/sign-up), and make sure you have the Fly CLI installed (https://fly.io/docs/flyctl/install/).
First, clone this repository and `cd` into the repo directory:
```bash
$ git clone https://github.com/fly-apps/self-host-ollama
$ cd self-host-ollama
```
Feel free to change the `app` name in the `fly.toml`, and then launch it as a new Fly App with:
```bash
$ fly launch --flycast
```
When it asks **if you'd like to copy the configuration, say YES.** The `fly.toml` file in this repo is all the configuration you need to deploy this Ollama app.
The `--flycast` flag will make your application private. To access your application, you'll use the address `http://.flycast` and ensure you're connected over your Wireguard VPN. This will work in both production and local development. More on this later.
Now that your app is launched, let's download some Ollama models. First, SSH into one of your Fly Machines like so:
```bash
$ fly ssh console
```
Next, set the value of your `OLLAMA_HOST`, so our `ollama` commands know what to use:
```bash
$ export OLLAMA_HOST=.flycast
```
Finally, pull down the model of your choice:
```bash
$ ollama pull llava
```
And you're done! Your Ollama app is now available for use.
## Connecting to your app
This Ollama app will exist separate from whatever app you're building. Since our app is private (we don't randos eating up our Fly GPU usage), we'll need to connect to it over a secure Wireguard connection. When developing locally, the easiest method is to run:
```bash
# you don't need -a if you're in the Ollama app directory
$ fly proxy 11434:80 -a
```
This command proxies requests from a local port (`11434`) to port `80` on your Ollama Fly Machine, over a secure Wireguard tunnel.
When using Ollama in your app **locally**, you'll set the host to `http://localhost:11434`. Note that while `11434` is the standard port used by Ollama, since this is just a proxy, that number can really be anything.
In **production**, you'll use the host `http://.flycast` instead.
## Examples
The following code would live in an app separate from your Ollama app; This allows you to auto-start and stop your Ollama app, so you're not paying for GPUs when not in use.
**JavaScript (npm `ollama` package)**
```typescript
import { Ollama } from "ollama";
const ollama = new Ollama({
host: process.env.OLLAMA_APP_URL // either http://localhost:11434 or http://.flycast on production
});
const response = await ollama.generate({
model: 'llama3.1',
prompt: "Give me a week's worth of healthy vegetarian meal ideas and their recipes.",
stream: false,
})
```
**JavaScript (basic fetch request)**
```typescript
const params = {
model: 'llama3.1',
prompt: "Give me a week's worth of healthy vegetarian meal ideas and their recipes.",
stream: false,
};
let resp = await fetch("http://sparkling-violet-709.flycast/api/generate", {
method: "POST",
body: JSON.stringify(params),
});
```
## Watch the video
Check out the accompanying video here: https://youtu.be/xkWcGmbhZRQ