{"id":14964940,"url":"https://github.com/friendliai/friendli-client","last_synced_at":"2025-04-05T03:04:46.870Z","repository":{"id":182710640,"uuid":"668722770","full_name":"friendliai/friendli-client","owner":"friendliai","description":"Friendli: the fastest serving engine for generative AI","archived":false,"fork":false,"pushed_at":"2025-01-24T05:18:52.000Z","size":5112,"stargazers_count":43,"open_issues_count":3,"forks_count":8,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-28T15:46:19.655Z","etag":null,"topics":["ai","generative-ai","gpt","gpt3","inference","inference-engine","inference-server","llama2","llm","llm-inference","llm-ops","llm-serving","llmops","llms","mistral","ml","mlops","serving","stable-diffusion"],"latest_commit_sha":null,"homepage":"https://friendli.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/friendliai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-20T12:57:24.000Z","updated_at":"2025-03-28T11:59:41.000Z","dependencies_parsed_at":"2023-10-20T09:08:34.431Z","dependency_job_id":"421cdbdf-1974-4fcf-b30a-00d13faf7627","html_url":"https://github.com/friendliai/friendli-client","commit_stats":{"total_commits":62,"total_committers":9,"mean_commits":6.888888888888889,"dds":0.3548387096774194,"last_synced_commit":"e30f3517332e7c15e80fc3d4d12df81f2530a29b"},"previous_names":["friendliai/periflow-client","friendliai/friendli-client"],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/friendliai%2Ffriendli-client","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/friendliai%2Ffriendli-client/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/friendliai%2Ffriendli-client/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/friendliai%2Ffriendli-client/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/friendliai","download_url":"https://codeload.github.com/friendliai/friendli-client/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247280262,"owners_count":20912967,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","generative-ai","gpt","gpt3","inference","inference-engine","inference-server","llama2","llm","llm-inference","llm-ops","llm-serving","llmops","llms","mistral","ml","mlops","serving","stable-diffusion"],"created_at":"2024-09-24T13:34:00.031Z","updated_at":"2025-04-05T03:04:46.843Z","avatar_url":"https://github.com/friendliai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!---\nCopyright (c) 2022-present, FriendliAI Inc. All rights reserved.\n--\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"10%\" alt=\"Friendli Logo\" src=\"https://friendli.ai/icon.svg\"\u003e\n\u003c/p\u003e\n\n\u003ch2\u003e\u003cp align=\"center\"\u003eSupercharge Generative AI Serving with Friendli 🚀\u003c/p\u003e\u003c/h2\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/friendliai/friendli-client/actions/workflows/ci.yaml\"\u003e\n    \u003cimg alt=\"CI Status\" src=\"https://github.com/friendliai/friendli-client/actions/workflows/ci.yaml/badge.svg\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/friendli-client/\"\u003e\n    \u003cimg alt=\"Python Version\" src=\"https://img.shields.io/pypi/pyversions/friendli-client?logo=Python\u0026logoColor=white\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/friendli-client/\"\u003e\n      \u003cimg alt=\"PyPi Package Version\" src=\"https://img.shields.io/pypi/v/friendli-client?logo=PyPI\u0026logoColor=white\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://friendli.ai/docs/\"\u003e\n    \u003cimg alt=\"Documentation\" src=\"https://img.shields.io/badge/read-doc-blue?logo=ReadMe\u0026logoColor=white\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/friendliai/friendli-client/blob/main/LICENSE\"\u003e\n      \u003cimg alt=\"License\" src=\"https://img.shields.io/badge/License-Apache%202.0-green.svg?logo=Apache\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\nThe Friendli Client offers convenient interface to interact with endpoint services provided by [Friendli Suite](https://suite.friendli.ai/), the ultimate solution for serving generative AI models. Designed for flexibility and performance, it supports both synchronous and asynchronous operations, making it easy to integrate powerful AI capabilities into your applications.\n\n# Installation\n\nTo get started with Friendli, install the client package using `pip`:\n\n```sh\npip install friendli-client\n```\n\n\u003e [!IMPORTANT]\n\u003e You must set `FRIENDLI_TOKEN` environment variable before initializing the client instance with `client = Friendli()`.\n\u003e Alternatively, you can provide the value of your personal access token as the `token` argument when creating the client, like so:\n\u003e\n\u003e ```python\n\u003e from friendli import Friendli\n\u003e\n\u003e client = Friendli(token=\"YOUR PERSONAL ACCESS TOKEN\")\n\u003e ```\n\n# Friendli Serverless Endpoints\n\n[Friendli Serverless Endpoint](https://friendli.ai/products/serverless-endpoints) offer a simple, click-and-play interface for accessing popular open-source models like Llama 3.1.\nWith pay-per-token billing, this is ideal for exploration and experimentation.\n\nTo interact with models hosted by serverless endpoints, provide the model code you want to use in the `model` argument. Refer to the [pricing table](https://friendli.ai/docs/guides/serverless_endpoints/pricing/) for a list of available model codes and their pricing.\n\n```python\nfrom friendli import Friendli\n\nclient = Friendli()\n\nchat_completion = client.chat.completions.create(\n    model=\"meta-llama-3.1-8b-instruct\",\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Tell me how to make a delicious pancake\",\n        }\n    ],\n)\nprint(chat_completion.choices[0].message.content)\n```\n\n# Friendli Dedicated Endpoints\n\n[Friendli Dedicated Endpoints](https://friendli.ai/products/dedicated-endpoints) enable you to run your custom generative AI models on dedicated GPU resources.\n\nTo interact with dedicated endpoints, provide the endpoint ID in the `model` argument.\n\n```python\nimport os\nfrom friendli import Friendli\n\nclient = Friendli(\n    team_id=os.environ[\"TEAM_ID\"],  # If not provided, default team is used.\n    use_dedicated_endpoint=True,\n)\n\nchat_completion = client.chat.completions.create(\n    model=os.environ[\"ENDPOINT_ID\"],\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Tell me how to make a delicious pancake\",\n        }\n    ],\n)\nprint(chat_completion.choices[0].message.content)\n```\n\n# Friendli Container\n\n[Friendli Container](https://friendli.ai/products/container) is perfect for users who prefer to serve LLMs within their own infrastructure. By deploying the Friendli Engine in containers on your on-premise or cloud GPUs, you can maintain complete control over your data and operations, ensuring security and compliance with internal policies.\n\n```python\nfrom friendli import Friendli\n\nclient = Friendli(base_url=\"http://0.0.0.0:8000\")\n\nchat_completion = client.chat.completions.create(\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Tell me how to make a delicious pancake\",\n        }\n    ],\n)\nprint(chat_completion.choices[0].message.content)\n```\n\n# Async Usage\n\n```python\nimport asyncio\nfrom friendli import AsyncFriendli\n\nclient = AsyncFriendli()\n\nasync def main() -\u003e None:\n    chat_completion = await client.chat.completions.create(\n        model=\"meta-llama-3.1-8b-instruct\",\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": \"Tell me how to make a delicious pancake\",\n            }\n        ],\n    )\n    print(chat_completion.choices[0].message.content)\n\n\nasyncio.run(main())\n```\n\n# Streaming Usage\n\n```python\nfrom friendli import Friendli\n\nclient = Friendli()\n\nstream = client.chat.completions.create(\n    model=\"meta-llama-3.1-8b-instruct\",\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Tell me how to make a delicious pancake\",\n        }\n    ],\n    stream=True,\n)\nfor chunk in stream:\n    print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n```\n\nThe async client (`AsyncFriendli`) uses the same interface to stream the response.\n\n```python\nimport asyncio\nfrom friendli import AsyncFriendli\n\nclient = AsyncFriendli()\n\nasync def main() -\u003e None:\n    stream = await client.chat.completions.create(\n        model=\"meta-llama-3.1-8b-instruct\",\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": \"Tell me how to make a delicious pancake\",\n            }\n        ],\n        stream=True,\n    )\n    async for chunk in stream:\n        print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n\n\nasyncio.run(main())\n```\n\n# Advanced Usage\n\n## Sending Requests to LoRA Adapters\n\nIf your endpoint is serving a Multi-LoRA model, you can send request to one of the adapters by providing the adapter route in the `model` argument.\n\nFor Friendli Dedicated Endpoints, provide the endpoint ID and the adapter route separated by a colon (`:`).\n\n```python\nimport os\nfrom friendli import Friendli\n\nclient = Friendli(\n    team_id=os.environ[\"TEAM_ID\"],  # If not provided, default team is used.\n    use_dedicated_endpoint=True,\n)\n\nchat_completion = client.lora.completions.create(\n    model=f\"{os.environ['ENDPOINT_ID']}:{os.environ['ADAPTER_ROUTE']}\",\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Tell me how to make a delicious pancake\",\n        }\n    ],\n)\n```\n\nFor Friendli Container, just provide the adapter name.\n\n```python\nimport os\nfrom friendli import Friendli\n\nclient = Friendli(base_url=\"http://0.0.0.0:8000\")\n\nchat_completion = client.lora.completions.create(\n    model=os.environ[\"ADAPTER_NAME\"],\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Tell me how to make a delicious pancake\",\n        }\n    ],\n)\n```\n\n## Using the gRPC Interface\n\n\u003e [!IMPORTANT]\n\u003e gRPC is only supported by Friendli Container, and only the streaming API of `v1/completions` is available.\n\nWhen Frienldi Container is running in gPRC mode, the client can interact with the gRPC server\nby initializing it with `use_grpc=True` argument.\n\n```python\nfrom friendli import Friendli\n\nclient = Friendli(base_url=\"0.0.0.0:8000\", use_grpc=True)\n\nstream = client.chat.completions.create(\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Tell me how to make a delicious pancake\",\n        }\n    ],\n    stream=True,  # Only streaming mode is available\n)\n\nfor chunk in stream:\n    print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n```\n\n## Configuring the HTTP Client\n\nThe client uses `httpx` to send HTTP requests. You can provide the customized `httpx.Client` when initializing `Friendli`.\n\n```python\nimport httpx\nfrom friendli import Friendli\n\nwith httpx.Client() as client:\n    client = Friendli(http_client=http_client)\n```\n\nFor the async client, you can provide `httpx.AsyncClient`.\n\n```python\nimport httx\nfrom friendli import AsyncFriendli\n\nwith httpx.AsyncClient() as client:\n    client = AsyncFriendli(http_client=http_client)\n```\n\n## Configuring the gRPC Channel\n\n```python\nimport grpc\nfrom friendli import Friendli\n\nwith grpc.insecure_channel(\"0.0.0.0:8000\") as channel:\n    client = Friendli(use_grpc=True, grpc_channel=channel)\n```\n\nYou can use the same interface for the async client.\n\n```python\nimport grpc.aio\nfrom friendli import AsyncFriendli\n\nasync with grpc.aio.insecure_channel(\"0.0.0.0:8000\") as channel:\n    client = AsyncFriendli(use_grpc=True, grpc_channel=channel)\n```\n\n## Managing Resource\n\nThe Friendli client provides several methods to manage and release resources.\n\n### Closing the Client\n\nBoth the `Friendli` and `AsyncFriendli` clients can hold network connections or other resources during their lifetime.\nTo ensure these resources are properly released, you should either call the `close()` method or use the client within a context manager.\n\n```python\nfrom friendli import Friendli\n\nclient = Friendli()\n\n# Use the client for various operations...\n\n# When done, close the client to release resources\nclient.close()\n```\n\nFor the asynchronous client, the pattern is similar:\n\n```python\nimport asyncio\nfrom friendli import AsyncFriendli\n\nclient = AsyncFriendli()\n\n# Use the client for various async operations...\n\n# When done, close the client to release resources\nawait client.close()\n```\n\nYou can also use context manager to automatically close the client and releases resources when the block is exited, making it a safer and more convenient way to manage resources.\n\n```python\nfrom friendli import Friendli\n\nwith Friendli() as client:\n    ...\n```\n\nFor asynchronous usage:\n\n```python\nimport asyncio\nfrom friendli import AsyncFriendli\n\nasync def main():\n    async with AsyncFriendli() as client:\n        ...\n\n\nasyncio.run(main())\n```\n\n### Managing Streaming Responses\n\nWhen using streaming responses, it’s crucial to properly close the HTTP connection after the interaction is complete.\nBy default, the connection is automatically closed once all data from the stream has been consumed (i.e., when the for-loop reaches the end).\nHowever, if streaming is interrupted by exceptions or other issues, the connection may remain open and won’t be released until it is garbage-collected.\nTo ensure that all underlying connections and resources are properly released, it’s important to explicitly close the connection, particularly when streaming is prematurely terminated.\n\n```python\nfrom friendli import Friendli\n\nclient = Friendli()\n\nstream = client.chat.completions.create(\n    model=\"meta-llama-3.1-8b-instruct\",\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Tell me how to make a delicious pancake\",\n        }\n    ],\n    stream=True,\n)\n\ntry:\n    for chunk in stream:\n        print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\nfinally:\n    stream.close()  # Ensure the stream is closed after use\n```\n\nFor asynchronous streaming:\n\n```python\nimport asyncio\nfrom friendli import AsyncFriendli\n\nclient = AsyncFriendli()\n\nasync def main():\n    stream = await client.chat.completions.create(\n        model=\"meta-llama-3.1-8b-instruct\",\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": \"Tell me how to make a delicious pancake\",\n            }\n        ],\n        stream=True,\n    )\n\n    try:\n        async for chunk in stream:\n            print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n    finally:\n        await stream.close()  # Ensure the stream is closed after use\n\nasyncio.run(main())\n```\n\nYou can also use context manager to automatically close the client and releases resources when the block is exited, making it a safer and more convenient way to manage resources.\n\n```python\nfrom friendli import Friendli\n\nclient = Friendli()\n\nwith client.chat.completions.create(\n    model=\"meta-llama-3.1-8b-instruct\",\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Tell me how to make a delicious pancake\",\n        }\n    ],\n    stream=True,\n) as stream:\n    for chunk in stream:\n        print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n```\n\nFor asynchronous streaming:\n\n```python\nimport asyncio\nfrom friendli import AsyncFriendli\n\nclient = AsyncFriendli()\n\nasync def main():\n    async with await client.chat.completions.create(\n        model=\"meta-llama-3.1-8b-instruct\",\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": \"Tell me how to make a delicious pancake\",\n            }\n        ],\n        stream=True,\n    ) as stream:\n        async for chunk in stream:\n            print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n\nasyncio.run(main())\n```\n\n### Canceling a gRPC Stream\n\nWhen using the gRPC interface with streaming, you might want to cancel an ongoing stream operation before it completes. This is particularly useful if you need to stop the stream due to a timeout or some other condition.\n\nFor synchronous gRPC streaming:\n\n```python\nfrom friendli import Friendli\n\nclient = Friendli(base_url=\"0.0.0.0:8000\", use_grpc=True)\n\nstream = client.chat.completions.create(\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Tell me how to make a delicious pancake\",\n        }\n    ],\n    stream=True,\n)\n\ntry:\n    for chunk in stream:\n        print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\nexcept SomeException:\n    stream.cancel()  # Cancel the stream in case of an error or interruption\n```\n\nFor asynchronous gRPC streaming:\n\n```python\nimport asyncio\nfrom friendli import AsyncFriendli\n\nclient = AsyncFriendli(base_url=\"0.0.0.0:8000\", use_grpc=True)\n\nasync def main():\n    stream = await client.chat.completions.create(\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": \"Tell me how to make a delicious pancake\",\n            }\n        ],\n        stream=True,\n    )\n\n    try:\n        async for chunk in stream:\n            print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n    except SomeException:\n        stream.cancel()  # Cancel the stream in case of an error or interruption\n\nasyncio.run(main())\n```\n\n# CLI Examples\n\nYou can also call the generation APIs directly with CLI.\n\n```sh\nfriendli api chat-completions create \\\n  -g \"user Tell me how to make a delicious pancake\" \\\n  -m meta-llama-3.1-8b-instruct\n```\n\nFor further information about the `friendli` command, run `friendli --help` in your terminal shell.\nThis will provide you with a detailed list of available options and usage instructions.\n\n\u003e [!TIP] \u003e **Check out our [official documentation](https://friendli.ai/docs/) to learn more!**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffriendliai%2Ffriendli-client","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffriendliai%2Ffriendli-client","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffriendliai%2Ffriendli-client/lists"}