{"id":13627610,"url":"https://github.com/uripeled2/llm-client-sdk","last_synced_at":"2026-02-23T08:07:08.226Z","repository":{"id":155098394,"uuid":"632060664","full_name":"uripeled2/llm-client-sdk","owner":"uripeled2","description":"SDK for using LLM","archived":false,"fork":false,"pushed_at":"2023-07-22T20:29:19.000Z","size":81,"stargazers_count":76,"open_issues_count":6,"forks_count":11,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-10-11T15:39:11.949Z","etag":null,"topics":["ai","ai21labs","api","async","bard","bard-api","chatgpt-api","generative-ai","gpt","huggingface","huggingface-transformers","large-language-models","llm","llms","nlp","openai","openai-api","palm-api","python","sdk"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uripeled2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-04-24T16:17:08.000Z","updated_at":"2024-08-03T23:41:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"307eeeee-7240-4f43-848a-aa758783f543","html_url":"https://github.com/uripeled2/llm-client-sdk","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uripeled2%2Fllm-client-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uripeled2%2Fllm-client-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uripeled2%2Fllm-client-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uripeled2%2Fllm-client-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uripeled2","download_url":"https://codeload.github.com/uripeled2/llm-client-sdk/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223735037,"owners_count":17194030,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai21labs","api","async","bard","bard-api","chatgpt-api","generative-ai","gpt","huggingface","huggingface-transformers","large-language-models","llm","llms","nlp","openai","openai-api","palm-api","python","sdk"],"created_at":"2024-08-01T22:00:36.205Z","updated_at":"2026-02-23T08:07:08.184Z","avatar_url":"https://github.com/uripeled2.png","language":"Python","funding_links":[],"categories":["Projects by main language"],"sub_categories":["python"],"readme":"# LLM-Client-SDK\n[![Test](https://github.com/uripeled2/llm-client-sdk/actions/workflows/test.yml/badge.svg)](https://github.com/uripeled2/llm-client-sdk/actions/workflows/test.yml)\n[![License: MIT](https://img.shields.io/github/license/uripeled2/llm-client-sdk.svg)](https://opensource.org/licenses/MIT)\n\nLLM-Client-SDK is an SDK for seamless integration with generative AI large language models\n(We currently support - OpenAI, Google, AI21, HuggingfaceHub, Aleph Alpha, Anthropic,\nLocal models with transformers - and many more soon).\n\nOur vision is to provide async native and production ready SDK while creating \na powerful and fast integration with different LLM without letting the user lose \nany flexibility (API params, endpoints etc.). *We also provide sync version, see\nmore details below in Usage section.\n\n## Base Interface\nThe package exposes two simple interfaces for seamless integration with LLMs (In the future, we \nwill expand the interface to support more tasks like list models, edits, etc.):\n```python\nfrom abc import ABC, abstractmethod\nfrom dataclasses import dataclass, field\nfrom typing import Any, Optional\nfrom enum import Enum\nfrom dataclasses_json import dataclass_json, config\nfrom aiohttp import ClientSession\n\n\nclass BaseLLMClient(ABC):\n    @abstractmethod\n    async def text_completion(self, prompt: str, **kwargs) -\u003e list[str]:\n        raise NotImplementedError()\n\n    async def get_tokens_count(self, text: str, **kwargs) -\u003e int:\n        raise NotImplementedError()\n\n\nclass Role(Enum):\n    SYSTEM = \"system\"\n    USER = \"user\"\n    ASSISTANT = \"assistant\"\n\n\n@dataclass_json\n@dataclass\nclass ChatMessage:\n    role: Role = field(metadata=config(encoder=lambda role: role.value, decoder=Role))\n    content: str\n    name: Optional[str] = field(default=None, metadata=config(exclude=lambda name: name is None))\n    example: bool = field(default=False, metadata=config(exclude=lambda _: True))\n    \n\n@dataclass\nclass LLMAPIClientConfig:\n    api_key: str\n    session: ClientSession\n    base_url: Optional[str] = None\n    default_model: Optional[str] = None\n    headers: dict[str, Any] = field(default_factory=dict)\n\n\nclass BaseLLMAPIClient(BaseLLMClient, ABC):\n    def __init__(self, config: LLMAPIClientConfig):\n        ...\n\n    @abstractmethod\n    async def text_completion(self, prompt: str, model: Optional[str] = None, max_tokens: int | None = None,\n                              temperature: Optional[float] = None, top_p: Optional[float] = None, **kwargs) -\u003e list[str]:\n        raise NotImplementedError()\n\n    async def chat_completion(self, messages: list[ChatMessage], temperature: float = 0,\n                              max_tokens: int = 16, model: Optional[str] = None, **kwargs) -\u003e list[str]:\n        raise NotImplementedError()\n\n    async def embedding(self, text: str, model: Optional[str] = None, **kwargs) -\u003e list[float]:\n        raise NotImplementedError()\n\n    async def get_chat_tokens_count(self, messages: list[ChatMessage], **kwargs) -\u003e int:\n        raise NotImplementedError()\n```\n\n## Requirements\n\nPython 3.9+\n\n## Installation\nIf you are worried about the size of the package you can install only the clients you need,\nby default we install none of the clients.\n\nFor all current clients support\n```console\n$ pip install llm-client[all]\n```\nFor only the base interface and some light LLMs clients (AI21 and Aleph Alpha)\n```console\n$ pip install llm-client\n```\n### Optional Dependencies\nFor all current api clients support\n```console\n$ pip install llm-client[api]\n```\nFor only local client support\n```console\n$ pip install llm-client[local]\n```\nFor sync support\n```console\n$ pip install llm-client[sync]\n```\nFor only OpenAI support\n```console\n$ pip install llm-client[openai]\n```\nFor only HuggingFace support\n```console\n$ pip install llm-client[huggingface]\n```\n\n\n## Usage\n\nUsing OpenAI directly through OpenAIClient - Maximum control and best practice in production\n```python\nimport os\nfrom aiohttp import ClientSession\nfrom llm_client import ChatMessage, Role, OpenAIClient, LLMAPIClientConfig\n\nOPENAI_API_KEY = os.environ[\"API_KEY\"]\nOPENAI_ORG_ID = os.getenv(\"ORG_ID\")\n\n\nasync def main():\n    async with ClientSession() as session:\n        llm_client = OpenAIClient(LLMAPIClientConfig(OPENAI_API_KEY, session, default_model=\"text-davinci-003\",\n                                                     headers={\"OpenAI-Organization\": OPENAI_ORG_ID}))  # The headers are optional\n        text = \"This is indeed a test\"\n        messages = [ChatMessage(role=Role.USER, content=\"Hello!\"),\n                    ChatMessage(role=Role.SYSTEM, content=\"Hi there! How can I assist you today?\")]\n\n        print(\"number of tokens:\", await llm_client.get_tokens_count(text))  # 5\n        print(\"number of tokens for chat completion:\", await llm_client.get_chat_tokens_count(messages, model=\"gpt-3.5-turbo\"))  # 23\n        print(\"generated chat:\", await llm_client.chat_completion(messages, model=\"gpt-3.5-turbo\"))  # ['Hi there! How can I assist you today?']\n        print(\"generated text:\", await llm_client.text_completion(text))  # [' string\\n\\nYes, this is a test string. Test strings are used to']\n        print(\"generated embedding:\", await llm_client.embedding(text))  # [0.0023064255, -0.009327292, ...]\n```\nUsing LLMAPIClientFactory - Perfect if you want to move fast and to not handle the client session yourself\n```python\nimport os\nfrom llm_client import LLMAPIClientFactory, LLMAPIClientType\n\nOPENAI_API_KEY = os.environ[\"API_KEY\"]\n\n\nasync def main():\n    async with LLMAPIClientFactory() as llm_api_client_factory:\n        llm_client = llm_api_client_factory.get_llm_api_client(LLMAPIClientType.OPEN_AI,\n                                                               api_key=OPENAI_API_KEY,\n                                                               default_model=\"text-davinci-003\")\n\n        await llm_client.text_completion(prompt=\"This is indeed a test\")\n        await llm_client.text_completion(prompt=\"This is indeed a test\", max_tokens=50)\n\n        \n# Or if you don't want to use async\nfrom llm_client import init_sync_llm_api_client\n\nllm_client = init_sync_llm_api_client(LLMAPIClientType.OPEN_AI, api_key=OPENAI_API_KEY,\n                                      default_model=\"text-davinci-003\")\n\nllm_client.text_completion(prompt=\"This is indeed a test\")\nllm_client.text_completion(prompt=\"This is indeed a test\", max_tokens=50)\n```\nLocal model\n```python\nimport os\nfrom transformers import AutoModelForCausalLM, AutoModelForSeq2SeqLM, AutoTokenizer\nfrom llm_client import LocalClientConfig, LocalClient\n\nasync def main():\n    try:\n        model = AutoModelForCausalLM.from_pretrained(os.environ[\"MODEL_NAME_OR_PATH\"])\n    except ValueError:\n        model = AutoModelForSeq2SeqLM.from_pretrained(os.environ[\"MODEL_NAME_OR_PATH\"])\n    tokenizer = AutoTokenizer.from_pretrained(os.environ[\"MODEL_NAME_OR_PATH\"])\n    llm_client = LocalClient(LocalClientConfig(model, tokenizer, os.environ[\"TENSORS_TYPE\"], os.environ[\"DEVICE\"]))\n\n    await llm_client.text_completion(prompt=\"This is indeed a test\")\n    await llm_client.text_completion(prompt=\"This is indeed a test\", max_tokens=50)\n\n\n# Or if you don't want to use async\nimport async_to_sync\n\ntry:\n    model = AutoModelForCausalLM.from_pretrained(os.environ[\"MODEL_NAME_OR_PATH\"])\nexcept ValueError:\n    model = AutoModelForSeq2SeqLM.from_pretrained(os.environ[\"MODEL_NAME_OR_PATH\"])\ntokenizer = AutoTokenizer.from_pretrained(os.environ[\"MODEL_NAME_OR_PATH\"])\nllm_client = LocalClient(LocalClientConfig(model, tokenizer, os.environ[\"TENSORS_TYPE\"], os.environ[\"DEVICE\"]))\n\nllm_client = async_to_sync.methods(llm_client)\n\nllm_client.text_completion(prompt=\"This is indeed a test\")\nllm_client.text_completion(prompt=\"This is indeed a test\", max_tokens=50)\n```\n\n## Contributing\n\nContributions are welcome! Please check out the todos below, and feel free to open issue or a pull request.\n\n### Todo\n*The list is unordered*\n\n- [x] Add support for more LLMs\n  - [x] Anthropic\n  - [x] Google\n  - [ ] Cohere\n- [x] Add support for more functions via LLMs \n  - [x] embeddings\n  - [x] chat\n  - [ ] list models\n  - [ ] edits\n  - [ ] more\n- [ ] Add contributing guidelines and linter\n- [ ] Create an easy way to run multiple LLMs in parallel with the same prompts\n- [x] Convert common models parameter\n  - [x] temperature \n  - [x] max_tokens\n  - [x] top_p\n  - [ ] more\n\n### Development\nTo install the package in development mode, run the following command:\n```console\n$ pip install -e \".[all,test]\"\n```\nTo run the tests, run the following command:\n```console\n$ pytest tests\n```\nIf you want to add a new LLMClient you need to implement BaseLLMClient or BaseLLMAPIClient.\n\nIf you are adding a BaseLLMAPIClient you also need to add him in LLMAPIClientFactory.\n\nYou can add dependencies to your LLMClient in [pyproject.toml](pyproject.toml) also make sure you are adding a\nmatrix.flavor in [test.yml](.github%2Fworkflows%2Ftest.yml). \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furipeled2%2Fllm-client-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Furipeled2%2Fllm-client-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furipeled2%2Fllm-client-sdk/lists"}