{"id":13596661,"url":"https://github.com/kagisearch/pyllms","last_synced_at":"2025-05-14T14:08:24.555Z","repository":{"id":152457388,"uuid":"625730598","full_name":"kagisearch/pyllms","owner":"kagisearch","description":"Minimal Python library to connect to LLMs (OpenAI, Anthropic, Google, Groq, Reka, Together, AI21, Cohere, Aleph Alpha, HuggingfaceHub), with a built-in model performance benchmark.","archived":false,"fork":false,"pushed_at":"2025-03-24T17:44:28.000Z","size":715,"stargazers_count":770,"open_issues_count":9,"forks_count":52,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-05-11T19:47:26.015Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://kagi.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kagisearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-04-10T01:26:04.000Z","updated_at":"2025-05-05T23:30:35.000Z","dependencies_parsed_at":"2023-10-04T11:03:36.007Z","dependency_job_id":"f8caac7c-918e-4621-838b-56d6bfe1a7fb","html_url":"https://github.com/kagisearch/pyllms","commit_stats":{"total_commits":303,"total_committers":12,"mean_commits":25.25,"dds":0.6435643564356436,"last_synced_commit":"70ec2a4fa4a233ee8011685a90d2446e42539f9d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kagisearch%2Fpyllms","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kagisearch%2Fpyllms/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kagisearch%2Fpyllms/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kagisearch%2Fpyllms/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kagisearch","download_url":"https://codeload.github.com/kagisearch/pyllms/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254159772,"owners_count":22024564,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T16:02:37.743Z","updated_at":"2025-05-14T14:08:24.536Z","avatar_url":"https://github.com/kagisearch.png","language":"Python","funding_links":[],"categories":["Python","NLP","Industry Strength Natural Language Processing"],"sub_categories":[],"readme":"# PyLLMs\n\n[![PyPI version](https://badge.fury.io/py/pyllms.svg)](https://badge.fury.io/py/pyllms)\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/license/mit/)\n[![](https://dcbadge.vercel.app/api/server/aDNg6E9szy?compact=true\u0026style=flat)](https://discord.gg/aDNg6E9szy)\n[![Twitter](https://img.shields.io/twitter/follow/KagiHQ?style=social)](https://twitter.com/KagiHQ)\n\nPyLLMs is a minimal Python library to connect to various Language Models (LLMs) with a built-in model performance benchmark.\n\n## Table of Contents\n\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Usage](#usage)\n  - [Basic Usage](#basic-usage)\n  - [Multi-model Usage](#multi-model-usage)\n  - [Async Support](#async-support)\n  - [Streaming Support](#streaming-support)\n  - [Chat History and System Message](#chat-history-and-system-message)\n  - [Other Methods](#other-methods)\n- [Configuration](#configuration)\n- [Model Benchmarks](#model-benchmarks)\n- [Supported Models](#supported-models)\n- [Advanced Usage](#advanced-usage)\n  - [Using OpenAI API on Azure](#using-openai-api-on-azure)\n  - [Using Google Vertex LLM models](#using-google-vertex-llm-models)\n  - [Using Local Ollama LLM models](#using-local-ollama-llm-models)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Features\n\n- Connect to top LLMs in a few lines of code\n- Response meta includes tokens processed, cost, and latency standardized across models\n- Multi-model support: Get completions from different models simultaneously\n- LLM benchmark: Evaluate models on quality, speed, and cost\n-  Async and streaming support for compatible models\n\n## Installation\n\nInstall the package using pip:\n\n```bash\npip install pyllms\n```\n\n## Quick Start\n\n```python\nimport llms\n\nmodel = llms.init('gpt-4o')\nresult = model.complete(\"What is 5+5?\")\n\nprint(result.text)\n```\n\n## Usage\n\n### Basic Usage\n\n```python\nimport llms\n\nmodel = llms.init('gpt-4o')\nresult = model.complete(\n    \"What is the capital of the country where Mozart was born?\",\n    temperature=0.1,\n    max_tokens=200\n)\n\nprint(result.text)\nprint(result.meta)\n```\n\n### Multi-model Usage\n\n```python\nmodels = llms.init(model=['gpt-3.5-turbo', 'claude-instant-v1'])\nresult = models.complete('What is the capital of the country where Mozart was born?')\n\nprint(result.text)\nprint(result.meta)\n```\n\n### Async Support\n\n```python\nresult = await model.acomplete(\"What is the capital of the country where Mozart was born?\")\n```\n\n### Streaming Support\n\n```python\nmodel = llms.init('claude-v1')\nresult = model.complete_stream(\"Write an essay on the Civil War\")\nfor chunk in result.stream:\n   if chunk is not None:\n      print(chunk, end='')\n```\n\n### Chat History and System Message\n\n```python\nhistory = []\nhistory.append({\"role\": \"user\", \"content\": user_input})\nhistory.append({\"role\": \"assistant\", \"content\": result.text})\n\nmodel.complete(prompt=prompt, history=history)\n\n# For OpenAI chat models\nmodel.complete(prompt=prompt, system_message=system, history=history)\n```\n\n### Other Methods\n\n```python\ncount = model.count_tokens('The quick brown fox jumped over the lazy dog')\n```\n\n## Configuration\n\nPyLLMs will attempt to read API keys and the default model from environment variables. You can set them like this:\n\n```bash\nexport OPENAI_API_KEY=\"your_api_key_here\"\nexport ANTHROPIC_API_KEY=\"your_api_key_here\"\nexport AI21_API_KEY=\"your_api_key_here\"\nexport COHERE_API_KEY=\"your_api_key_here\"\nexport ALEPHALPHA_API_KEY=\"your_api_key_here\"\nexport HUGGINFACEHUB_API_KEY=\"your_api_key_here\"\nexport GOOGLE_API_KEY=\"your_api_key_here\"\nexport MISTRAL_API_KEY=\"your_api_key_here\"\nexport REKA_API_KEY=\"your_api_key_here\"\nexport TOGETHER_API_KEY=\"your_api_key_here\"\nexport GROQ_API_KEY=\"your_api_key_here\"\nexport DEEPSEEK_API_KEY=\"your_api_key_here\"\n\nexport LLMS_DEFAULT_MODEL=\"gpt-3.5-turbo\"\n```\n\nAlternatively, you can pass initialization values to the `init()` method:\n\n```python\nmodel = llms.init(openai_api_key='your_api_key_here', model='gpt-4')\n```\n\n## Model Benchmarks\n\nPyLLMs includes an automated benchmark system. The quality of models is evaluated using a powerful model (e.g., GPT-4) on a range of predefined questions, or you can supply your own.\n\n```python\nmodel = llms.init(model=['claude-3-haiku-20240307', 'gpt-4o-mini', 'claude-3-5-sonnet-20240620', 'gpt-4o', 'mistral-large-latest', 'open-mistral-nemo', 'gpt-4', 'gpt-3.5-turbo', 'deepseek-coder', 'deepseek-chat', 'llama-3.1-8b-instant', 'llama-3.1-70b-versatile'])\n\ngpt4 = llms.init('gpt-4o')\n\nmodels.benchmark(evaluator=gpt4)\n```\n\nCheck [Kagi LLM Benchmarking Project](https://help.kagi.com/kagi/ai/llm-benchmark.html) for the latest benchmarks!\n\nTo evaluate models on your own prompts:\n\n```python\nmodels.benchmark(prompts=[(\"What is the capital of Finland?\", \"Helsinki\")], evaluator=gpt4)\n```\n\n## Supported Models\n\nTo get a full list of supported models:\n\n```python\nmodel = llms.init()\nmodel.list() # list all models\n\nmodel.list(\"gpt\")  # lists only models with 'gpt' in name/provider name\n```\nCurrently supported models (may be outdated):\n\n| **Provider**              | **Models**                                                                                               |\n|---------------------------|---------------------------------------------------------------------------------------------------------|\n| OpenAIProvider            | gpt-3.5-turbo, gpt-3.5-turbo-1106, gpt-3.5-turbo-instruct, gpt-4, gpt-4-1106-preview, gpt-4-turbo-preview, gpt-4-turbo, gpt-4o, gpt-4o-mini, gpt-4o-2024-08-06, o1-preview, o1-mini, o1 |\n| AnthropicProvider         | claude-instant-v1.1, claude-instant-v1, claude-v1, claude-v1-100k, claude-instant-1, claude-instant-1.2, claude-2.1, claude-3-haiku-20240307, claude-3-sonnet-20240229, claude-3-opus-20240229, claude-3-5-sonnet-20240620, claude-3-5-sonnet-20241022 |\n| BedrockAnthropicProvider  | anthropic.claude-instant-v1, anthropic.claude-v1, anthropic.claude-v2, anthropic.claude-3-haiku-20240307-v1:0, anthropic.claude-3-sonnet-20240229-v1:0, anthropic.claude-3-5-sonnet-20240620-v1:0 |\n| AI21Provider              | j2-grande-instruct, j2-jumbo-instruct |\n| CohereProvider            | command, command-nightly |\n| AlephAlphaProvider        | luminous-base, luminous-extended, luminous-supreme, luminous-supreme-control |\n| HuggingfaceHubProvider    | hf_pythia, hf_falcon40b, hf_falcon7b, hf_mptinstruct, hf_mptchat, hf_llava, hf_dolly, hf_vicuna |\n| GoogleGenAIProvider       | chat-bison-genai, text-bison-genai, gemini-1.5-pro, gemini-1.5-pro-latest, gemini-1.5-flash, gemini-1.5-flash-latest, gemini-1.5-pro-exp-0801 |\n| GoogleProvider            | chat-bison, text-bison, text-bison-32k, code-bison, code-bison-32k, codechat-bison, codechat-bison-32k, gemini-pro, gemini-1.5-pro-preview-0514, gemini-1.5-flash-preview-0514 |\n| OllamaProvider            | vanilj/Phi-4:latest, falcon3:10b, smollm2:latest, llama3.2:3b-instruct-q8_0, qwen2:1.5b, mistral:7b-instruct-v0.2-q4_K_S, phi3:latest, phi3:3.8b, phi:latest, tinyllama:latest, magicoder:latest, deepseek-coder:6.7b, deepseek-coder:latest, dolphin-phi:latest, stablelm-zephyr:latest |\n| DeepSeekProvider          | deepseek-chat, deepseek-coder |\n| GroqProvider              | llama-3.1-405b-reasoning, llama-3.1-70b-versatile, llama-3.1-8b-instant, gemma2-9b-it |\n| RekaProvider              | reka-edge, reka-flash, reka-core |\n| TogetherProvider          | meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo |\n| OpenRouterProvider        | nvidia/llama-3.1-nemotron-70b-instruct, x-ai/grok-2, nousresearch/hermes-3-llama-3.1-405b:free, google/gemini-flash-1.5-exp, liquid/lfm-40b, mistralai/ministral-8b, qwen/qwen-2.5-72b-instruct |\n| MistralProvider           | mistral-tiny, open-mistral-7b, mistral-small, open-mixtral-8x7b, mistral-small-latest, mistral-medium-latest, mistral-large-latest, open-mistral-nemo |\n\n\n\n## Advanced Usage\n\n### Using OpenAI API on Azure\n\n```python\nimport llms\nAZURE_API_BASE = \"{insert here}\"\nAZURE_API_KEY = \"{insert here}\"\n\nmodel = llms.init('gpt-4')\n\nazure_args = {\n    \"engine\": \"gpt-4\",  # Azure deployment_id\n    \"api_base\": AZURE_API_BASE,\n    \"api_type\": \"azure\",\n    \"api_version\": \"2023-05-15\",\n    \"api_key\": AZURE_API_KEY,\n}\n\nazure_result = model.complete(\"What is 5+5?\", **azure_args)\n```\n\n### Using Google Vertex LLM models\n\n1. Set up a GCP account and create a project\n2. Enable Vertex AI APIs in your GCP project\n3. Install gcloud CLI tool\n4. Set up Application Default Credentials\n\nThen:\n\n```python\nmodel = llms.init('chat-bison')\nresult = model.complete(\"Hello!\")\n```\n\n### Using Local Ollama LLM models\n\n1. Ensure Ollama is running and you've pulled the desired model\n2. Get the name of the LLM you want to use\n3. Initialize PyLLMs:\n\n```python\nmodel = llms.init(\"tinyllama:latest\")\nresult = model.complete(\"Hello!\")\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkagisearch%2Fpyllms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkagisearch%2Fpyllms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkagisearch%2Fpyllms/lists"}