https://github.com/fxnai/fxn-llm

Use local LLMs in your Python apps. Register at https://fxn.ai
https://github.com/fxnai/fxn-llm

Last synced: about 2 months ago
JSON representation

Use local LLMs in your Python apps. Register at https://fxn.ai

• Host: GitHub
• URL: https://github.com/fxnai/fxn-llm
• Owner: fxnai
• License: apache-2.0
• Created: 2024-08-24T15:00:06.000Z (5 months ago)
• Default Branch: main
• Last Pushed: 2024-10-12T22:20:13.000Z (3 months ago)
• Last Synced: 2024-10-29T00:39:58.142Z (2 months ago)
• Language: Python
• Homepage: https://docs.fxn.ai/samples/py/openai
• Size: 17.6 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: Changelog.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
# Function LLM for Python

![function logo](https://raw.githubusercontent.com/fxnai/.github/main/logo_wide.png)

[![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fdiscord.com%2Fapi%2Finvites%2Fy5vwgXkz2f%3Fwith_counts%3Dtrue&query=%24.approximate_member_count&logo=discord&logoColor=white&label=Function%20community)](https://discord.gg/fxn)

[![X (formerly Twitter) Follow](https://img.shields.io/twitter/follow/fxnai)](https://twitter.com/fxnai)

Use local LLMs in your Python apps, with GPU acceleration and zero dependencies. This package is designed to patch `OpenAI` and `Anthropic` clients for running inference locally, using predictors hosted on [Function](https://fxn.ai/explore).

> [!TIP]

> We offer a similar package for use in the browser and Node.js. Check out [fxn-llm-js](https://github.com/fxnai/fxn-llm-js).

> [!IMPORTANT]

> This package is still a work-in-progress, so the API could change drastically between **all** releases.

## Installing Function LLM

Function is distributed on PyPi. To install, open a terminal and run the following command:

```bash

# Install Function LLM

$ pip install --upgrade fxn-llm

```

> [!NOTE]

> Function LLM requires Python 3.10+

> [!IMPORTANT]

> Make sure to create an access key by signing onto [Function](https://fxn.ai/settings/developer). You'll need it to fetch the predictor at runtime.

## Using the OpenAI Client Locally

To run text generation and embedding models locally using the OpenAI client, patch your `OpenAI` instance with the `locally` function:

```py

from openai import OpenAI

from fxn_llm import locally

# 💥 Create your OpenAI client

openai = OpenAI()

# 🔥 Make it local

openai = locally(openai)

# 🚀 Generate embeddings

embeddings = openai.embeddings.create(

    model="@nomic/nomic-embed-text-v1.5-quant",

    input="search_query: Hello world!"

)

```

> [!WARNING]

> Currently, only `openai.embeddings.create` is supported. Text generation is coming soon!

___

## Useful Links

- [Discover predictors to use in your apps](https://fxn.ai/explore).

- [Join our Discord community](https://discord.gg/fxn).

- [Check out our docs](https://docs.fxn.ai).

- Learn more about us [on our blog](https://blog.fxn.ai).

- Reach out to us at [[email protected]](mailto:[email protected]).

Function is a product of [NatML Inc](https://github.com/natmlx).