Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/presbrey/ollamafarm
Manage and use multiple Ollama instances with automatic offline detection/failover and model availability tracking
https://github.com/presbrey/ollamafarm
Last synced: 4 months ago
JSON representation
Manage and use multiple Ollama instances with automatic offline detection/failover and model availability tracking
- Host: GitHub
- URL: https://github.com/presbrey/ollamafarm
- Owner: presbrey
- License: mit
- Created: 2024-08-25T05:16:29.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2024-08-25T21:20:44.000Z (4 months ago)
- Last Synced: 2024-08-26T06:27:35.555Z (4 months ago)
- Language: Go
- Homepage:
- Size: 14.6 KB
- Stars: 5
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
- awesome-go-cn - OllamaFarm - balance, and failover packs of Ollamas [![godoc][D]](https://godoc.org/github.com/presbrey/ollamafarm) (Artificial Intelligence)
- awesome-go - OllamaFarm - Manage, load-balance, and failover packs of Ollamas (Artificial Intelligence)
- awesome-go-with-stars - OllamaFarm - Manage, load-balance, and failover packs of Ollamas (Artificial Intelligence)
- awesome-go-plus - OllamaFarm - Manage, load-balance, and failover packs of Ollamas  (Artificial Intelligence)
- awesome-go - OllamaFarm - Manage, load-balance, and failover packs of Ollamas (Artificial Intelligence)
- fucking-awesome-go - OllamaFarm - Manage, load-balance, and failover packs of Ollamas (Artificial Intelligence)
- fucking-awesome-go - OllamaFarm - Manage, load-balance, and failover packs of Ollamas (Artificial Intelligence)
- awesome-go-plus - OllamaFarm - Manage, load-balance, and failover packs of Ollamas (Artificial Intelligence)
README
[](https://goreportcard.com/report/github.com/presbrey/ollamafarm)
[](https://codecov.io/gh/presbrey/ollamafarm)
[](https://godoc.org/github.com/presbrey/ollamafarm)
[](https://opensource.org/licenses/MIT)
# OllamaFarm
OllamaFarm is a Go package that manages multiple Ollama instances, providing a convenient way to interact with a farm of Ollama servers. It offers features like automatic offline detection and failover, model availability tracking, and server selection based on criteria such as model.
## Installation
To install OllamaFarm, use the following command:
```bash
go get github.com/presbrey/ollamafarm
```
## Usage
Here's an example of how to use OllamaFarm with multiple Ollamas in the same group and different priorities:
```go
package main
import (
"context"
"fmt"
"log"
"github.com/presbrey/ollamafarm"
"github.com/ollama/ollama/api"
)
func main() {
farm := ollamafarm.New()
// Register Ollama servers in the same group with different priorities
farm.RegisterURL("http://ollama1:11434", &ollamafarm.Properties{Group: "4090", Priority: 1})
farm.RegisterURL("http://ollama2:11434", &ollamafarm.Properties{Group: "4090", Priority: 2})
farm.RegisterURL("http://ollama3:11434", &ollamafarm.Properties{Group: "3090", Priority: 1})
// Select an Ollama instance
ollama := farm.First(&ollamafarm.Where{Model: "llama3.1:8b-instruct-fp16"})
if ollama != nil {
// Perform a Chat call
req := &api.ChatRequest{
Model: "llama3.1:8b-instruct-fp16",
Messages: []api.Message{
{Role: "user", Content: "How many letter R are in the word Strawberry?"},
},
}
err := ollama.Client().Chat(context.Background(), req, func(resp api.ChatResponse) error {
fmt.Print(resp.Message.Content)
return nil
})
if err != nil {
log.Fatalf("Chat error: %v", err)
}
}
// Get model counts
modelCounts := farm.ModelCounts(nil)
fmt.Printf("Available models: %v\n", modelCounts)
}
```
Note: When an Ollama instance goes offline, OllamaFarm automatically selects the next online Ollama with the highest priority (lowest priority number) within the same group. This ensures continuous operation and optimal resource utilization without manual intervention.
## API Reference
### Types
* `Farm`: The main struct that manages multiple Ollama instances.
* `Ollama`: Represents an individual Ollama server.
* `Options`: Defines the options for a Farm. All fields are optional.
```go
type Options struct {
Client *http.Client
Heartbeat time.Duration
ModelsTTL time.Duration
}
```
* `Properties`: Defines the properties of an Ollama client. All fields are optional.
```go
type Properties struct {
Group string
Offline bool
Priority int
}
```
* `Where`: Defines the selection criteria for Ollama clients.
```go
type Where struct {
Group string
Model string
Offline bool
}
```
### Functions
* `New() *Farm`: Creates a new Farm instance with default options.
* `NewWithOptions(options *Options) *Farm`: Creates a new Farm instance with the given options.
### Farm Methods
* `RegisterClient(id string, client *api.Client, properties *Properties)`: Adds a new Ollama to the Farm if it doesn't already exist.
* `RegisterURL(baseURL string, properties *Properties) error`: Adds a new Ollama to the Farm using the baseURL as the ID.
* `First(where *Where) *Ollama`: Returns the first Ollama that matches the given criteria.
* `Select(where *Where) []*Ollama`: Returns a list of Ollamas that match the given criteria, sorted by ascending Priority.
* `ModelCounts(where *Where) map[string]uint`: Returns a count of all models available across all registered Ollamas.
### Ollama Methods
* `Client() *api.Client`: Returns the Ollama client.
* `Farm() *Farm`: Returns the Farm that this Ollama belongs to.
* `Group() string`: Returns the Ollama's group.
* `Online() bool`: Returns whether the Ollama is online.
* `Priority() int`: Returns the Ollama's priority.
## Contributing
Contributions to OllamaFarm are welcome! Please note the following guidelines:
1. All pull requests must maintain or improve the existing test coverage.
2. New features or changes must not break any existing APIs.
3. Write clear, concise commit messages.
4. Follow Go best practices and style guidelines.
## License
This project is licensed under the [MIT LICENSE](LICENSE.txt) file in the root directory of this repository.