An open API service indexing awesome lists of open source software.

https://github.com/synw/goinfer

Lightweight inference server for local language models
https://github.com/synw/goinfer

Last synced: about 2 months ago
JSON representation

Lightweight inference server for local language models

Awesome Lists containing this project

README

        

# Goinfer

Inference api server for local gguf language models. Based on [Llama.cpp](https://github.com/ggerganov/llama.cpp)

- **Multi models**: switch between models at runtime
- **Inference queries**: http api and streaming response support
- **Tasks**: predefined language model tasks

Works with the [Infergui](https://github.com/synw/infergui) frontend

:books: Read the documentation

- [Get started](https://synw.github.io/goinfer/get_started)
- [Install](https://synw.github.io/goinfer/get_started/install)
- [Configure](https://synw.github.io/goinfer/get_started/configure)
- [Run](https://synw.github.io/goinfer/get_started/run)
- [Llama api](https://synw.github.io/goinfer/llama_api)
- [Models state](https://synw.github.io/goinfer/llama_api/models_state)
- [Load model](https://synw.github.io/goinfer/llama_api/load_model)
- [Inference](https://synw.github.io/goinfer/llama_api/inference)
- [Tasks](https://synw.github.io/goinfer/llama_api/tasks)
- [Templates](https://synw.github.io/goinfer/llama_api/templates)
- [Openai api](https://synw.github.io/goinfer/openai_api)
- [Configure](https://synw.github.io/goinfer/openai_api/configure)
- [Endpoints](https://synw.github.io/goinfer/openai_api/endpoints)