{"id":23387074,"url":"https://github.com/log10-io/log10","last_synced_at":"2025-04-11T05:08:37.023Z","repository":{"id":143353004,"uuid":"614655497","full_name":"log10-io/log10","owner":"log10-io","description":"Python client library for improving your LLM app accuracy","archived":false,"fork":false,"pushed_at":"2025-02-11T20:43:32.000Z","size":17362,"stargazers_count":97,"open_issues_count":10,"forks_count":11,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-11T05:08:29.571Z","etag":null,"topics":["agents","ai","anthropic","artificial-intelligence","autonomous-agents","debugging","evaluations","feedback","fine-tuning","llmops","llms","logging","monitoring","openai","python","rlhf"],"latest_commit_sha":null,"homepage":"https://log10.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/log10-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-16T03:14:47.000Z","updated_at":"2025-03-17T23:36:54.000Z","dependencies_parsed_at":"2023-10-30T18:24:36.612Z","dependency_job_id":"534ef3e6-4a62-4359-800a-83bbffc4bb66","html_url":"https://github.com/log10-io/log10","commit_stats":null,"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/log10-io%2Flog10","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/log10-io%2Flog10/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/log10-io%2Flog10/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/log10-io%2Flog10/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/log10-io","download_url":"https://codeload.github.com/log10-io/log10/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248345266,"owners_count":21088244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","ai","anthropic","artificial-intelligence","autonomous-agents","debugging","evaluations","feedback","fine-tuning","llmops","llms","logging","monitoring","openai","python","rlhf"],"created_at":"2024-12-22T01:14:28.807Z","updated_at":"2025-04-11T05:08:36.990Z","avatar_url":"https://github.com/log10-io.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# log10\n\n⚡ Unified LLM data management to drive accuracy at scale ⚡\n\n[![pypi](https://github.com/log10-io/log10/actions/workflows/release.yml/badge.svg)](https://github.com/log10-io/log10/actions/workflows/release.yml)\n[![](https://dcbadge.vercel.app/api/server/CZQvnuRV94?compact=true\u0026style=flat)](https://discord.gg/CZQvnuRV94)\n\n## Quick Install\n\n`pip install log10-io`\n\n## 🤔 What is this?\n\n\u003cimg width=\"800\" alt=\"Log10 stack\" src=\"https://github.com/user-attachments/assets/8a790f82-6d75-4aa0-905b-7d3693815414\"\u003e\n\nA one-line Python integration to manage your LLM data.\n\n```python\nimport openai\nfrom log10.load import log10\n\nlog10(openai)\n# all your openai calls are now logged - including 3rd party libs using openai\n```\nFor OpenAI v1, use `from log10.load import OpenAI` instead of `from openai import OpenAI`\n```python\nfrom log10.load import OpenAI\n\nclient = OpenAI()\n```\nAccess your LLM data at [log10.io](https://log10.io)\n\n\n## 🚀 What can this help with?\n\n### 📝📊 Logging\n\nUse Log10 to log both closed and open-source LLM calls, e.g. OpenAI, Anthropic, Google Gemini, Llama, Mistral, etc. It helps you:\n- Compare and identify the best models and prompts (try [playground](https://log10.io/docs/observability/playgrounds) and [llmeval](https://log10.io/docs/evaluation/installation))\n- Store feedback for fine-tuning\n- Collect performance metrics such as latency and usage\n- Perform analytics and monitor compliance for LLM powered applications\n\nLog10 offers various integration methods, including a python LLM library wrapper, the Log10 LLM abstraction, and callbacks, to facilitate its use in both existing production environments and new projects.\nPick the one that works best for you.\n\n#### OpenAI\n| log10 ver| openai v0 | openai v1 |\n|----------|----------|----------|\n| 0.4 | `log10(openai)` ✅ | ❌ |\n| 0.5+ | `log10(openai)` ✅ | `from log10.load import OpenAI` ✅ |\n\n**OpenAI v0** - Use library wrapper `log10(openai)`. Check out `examples/logging` in log10 version `0.4.6`.\n```python\nimport openai\nfrom log10.load import log10\n\nlog10(openai)\n# openai calls are now logged - including 3rd party libs using openai such as magentic or langchain\n```\n\n**OpenAI v1**\n\u003e NOTE: We added OpenAI v1 API support in log10 `0.5.0` release. `load.log10(openai)` still works for openai v1. This also enables logging LLM completions from providers which support OpenAI API, such as [Ollama](https://github.com/ollama/ollama/blob/main/docs/openai.md).\n\n```python\nfrom log10.load import OpenAI\n# from openai import OpenAI\n\nclient = OpenAI()\ncompletion = client.completions.create(model=\"gpt-3.5-turbo-instruct\", prompt=\"Once upon a time\")\n# All completions.create and chat.completions.create calls will be logged\n```\nFull script [here](examples/logging/completion.py).\n\n\n**Use Log10 LLM abstraction**\n\n```python\nfrom log10.openai import OpenAI\n\nllm = OpenAI({\"model\": \"gpt-3.5-turbo\"}, log10_config=Log10Config())\n```\nopenai v1+ lib required. Full script [here](examples/logging/llm_abstraction.py#6-#14).\n\n#### Anthropic\nUse library wrapper `log10(anthropic)`.\nFull script [here](/examples/logging/anthropic_completion.py).\n```python\nimport anthropic\nfrom log10.load import log10\n\nlog10(anthropic)\n# anthropic calls are now logged\n```\n\nUse Log10 LLM abstraction.\nFull script [here](examples/logging/llm_abstraction.py#16-#19).\n```python\nfrom log10.anthropic import Anthropic\n\nllm = Anthropic({\"model\": \"claude-2\"}, log10_config=Log10Config())\n```\n\n#### Asynchronous LLM calls\nWe support OpenAI and Anthropic Async-client (e.g. AsyncOpenAI and AsyncAnthropic client) in their Python SDK\nYou could use the same code `log10(openai)` or `log10(anthropic)` and then call the async-client to start logging asynchronous mode (including streaming).\n\nRelease `0.9.0` includes significant improvements in how we handle concurrency while using LLM in asynchronous streaming mode.\nThis update is designed to ensure that logging at steady state incurs no overhead (previously up to 1-2 seconds), providing a smoother and more efficient experience in latency critical settings.\n\n__Important Considerations for Short-Lived Scripts__:\n\u003e 💡For short-lived scripts using asynchronous streaming, it's important to note that you may need to wait until all logging requests have been completed before terminating your script.\nWe have provided a convenient method called `finalize()` to handle this.\nHere's how you can implement this in your code:\n\n``` python\nfrom log10._httpx_utils import finalize\n\n...\n\nawait finalize()\n```\nEnsure `finalize()` is called once, at the very end of your event loop to guarantee that all pending logging requests are processed before the script exits.\n\n\nFor more details, check [async logging examples](./examples/logging/).\n\n#### Open-source LLMs\nLog open-source LLM calls, e.g. Llama, Mistral, etc from providers.\nCurrently we support inference endpoints on Together.AI and MosaicML (ranked on the top based on our [benchmarking](https://arjunbansal.substack.com/p/which-llama-2-inference-api-should-i-use) on Llama-2 inference providers).\nAdding other providers is on the roadmap.\n\nIf the providers support OpenAI API (e.g. [Groq](https://console.groq.com/docs/openai), [vLLM](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html), [Together](https://docs.together.ai/docs/openai-api-compatibility)), you can easily starting logging using `log10(openai)`.\n\n**MosaicML** with LLM abstraction. Full script [here](/examples/logging/mosaicml_completion.py).\n```python\nfrom log10.mosaicml import MosaicML\n\nllm = MosaicML({\"model\": \"llama2-70b-chat/v1\"}, log10_config=Log10Config())\n```\n\n**Together** with LLM abstraction. Full script [here](/examples/logging/together_completion.py).\n```python\nfrom log10.together import Together\n\nllm = Together({\"model\": \"togethercomputer/llama-2-70b-chat\"}, log10_config=Log10Config())\n```\n\n#### Other LLM frameworks\nUse Log10 callbacks if you use LangChain's LLM abstraction. Full script [here](/examples/logging/langchain_model_logger.py).\n\n```python\nfrom langchain.chat_models import ChatOpenAI\nfrom langchain.schema import HumanMessage\n\nfrom log10.langchain import Log10Callback\nfrom log10.llm import Log10Config\n\nlog10_callback = Log10Callback(log10_config=Log10Config())\n\nmessages = [\n    HumanMessage(content=\"You are a ping pong machine\"),\n    HumanMessage(content=\"Ping?\"),\n]\n\nllm = ChatOpenAI(model_name=\"gpt-3.5-turbo\", callbacks=[log10_callback])\n```\n\nRead more here for options for logging using library wrapper, langchain callback logger and how to apply log10 tags [here](./logging.md).\n\n### 🤖👷 Prompt engineering copilot\n\nOptimizing prompts requires a lot of manual effort. Log10 provides a copilot that can help you with suggestions on how to [optimize your prompt](https://log10.io/docs/prompt_engineering/auto_prompt#how-to-use-auto-prompting-in-log10-python-library).\n\n### 👷🔢 Feedback\n\nAdd feedback to your completions. Checkout the Python [example](/examples/feedback/simple_feedback.py)\nor use CLI `log10 feedback-task create` and `log10 feedback create`. Please check our [doc](https://log10.io/docs/feedback) for more details.\n\n#### AutoFeedback\nLeverage your current feedback and AI by using our AutoFeedback feature to generate feedback automatically. Here’s a quick guide:\n\n* Summary feedback: Use [TLDR summary feedback](/log10/feedback/_summary_feedback_utils.py) rubics to rate summarization. E.g. `log10 feedback predict --task_id $FEEDBACK_TASK_ID --content '{\"prompt\": \"this is article\", \"response\": \"summary of the article.\"}'`.\n  * You can pass a file containing the context with `--file` or pass a completion from your Log10 logs with `--completion_id`.\n* Custom Feedback Rubrics: Integrate your own feedback criteria for personalized assessments.\n* Getting Started: To explore all options and usage details, use CLI `log10 feedback predict --help`.\n\nFeel free to integrate AutoFeedback into your workflow to enhance the feedback and evaluation process.\n\n### ⚖️📊 Model Comparison\nEasily benchmark your logged completions using LLM models from OpenAI, Anthropic, Mistral, Meta, etc., by using the `log10 completions benchmark_models` command in the log10 CLI.\nGenerate detailed reports and gain insights to enhance your model's performance and cost.\nPlease refer to the [cli doc](cli_docs.md#log10-completions-benchmark_models) or the [demo video](https://www.loom.com/share/6d088f9f079f4e65962741f58344d77e?sid=1d2c51e6-8978-4422-af5b-d39ebe561b83) for details.\n\n### 🔍🐞 Prompt chain debugging\n\nPrompt chains such as those in [Langchain](https://github.com/hwchase17/langchain) can be difficult to debug. Log10 provides prompt provenance, session tracking and call stack functionality to help debug chains.\n\n### 🧠🔁 Readiness for RLHF \u0026 self hosting\n\nUse your data and feedback from users to fine-tune custom models with RLHF with the option of building and deploying more reliable, accurate and efficient self-hosted models.\n\n### 👥🤝 Collaboration\n\nCreate flexible groups to share and collaborate over all of the above features\n\n## ⚙️ Setup\n\n1. Create a free account at [log10.io](https://log10.io)\n2. Set the following environment variables:\n- `LOG10_URL=https://log10.io`\n- `LOG10_TOKEN`: From the Settings tab in log10.io\n- `LOG10_ORG_ID`: From the Organization tab in log10.io\n- `OPENAI_API_KEY`: OpenAI API key\n- `ANTHROPIC_API_KEY`: Anthropic API key\n\n### ✅ Run examples and tests\n\nYou can find and run examples under folder `examples`, e.g. run a logging example:\n```\npython examples/logging/chatcompletion.py\n```\n\nAlso you can run some end-to-end tests with [`xdocttest`](https://github.com/Erotemic/xdoctest) installed (`pip install xdoctest`).\n\n```\n# list all tests\npython -m xdoctest log10 list\n\n# run all tests\npython -m xdoctest log10 all\n\n# run a single test, e.g.\npython -m xdoctest /Users/wenzhe/dev/log10/log10/load.py log10:0\n```\n\n### Logging\nFew options to enable debug logging:\n1. set environment varible `export LOG10_DEBUG=1`\n1. set `log10.load.log10(DEBUG_=True)` when using `log10.load`\n1. set `log10_config(DEBUG=True)` when using llm abstraction classes or callback.\n\n### 💿🧩 Flexible data store\n\nlog10 provides a managed data store, but if you'd prefer to manage data in your own environment, you can use data stores like google big query.\n\nInstall the big query client library with:\n\n`pip install log10-io[bigquery]`\n\nAnd provide the following configuration in either a `.env` file, or as environment variables:\n\n| Name | Description |\n|------|-------------|\n| `LOG10_DATA_STORE`  |  Either `log10` or `bigquery` |\n| `LOG10_BQ_PROJECT_ID`   | Your google cloud project id      |\n| `LOG10_BQ_DATASET_ID`  |  The big query dataset id  |\n| `LOG10_BQ_COMPLETIONS_TABLE_ID` | The name of the table to store completions in |\n\n**Note** that your environment should have been setup with google cloud credentials. Read more [here](https://cloud.google.com/sdk/gcloud/reference/auth/login) about authenticating.\n\n## CLI for completions and feedback\nWe add CLI to manage your completions and feedback. Read more [here](cli_docs.md).\n\n## 💬 Community\n\nWe welcome community participation and feedback. Please leave an issue, submit a PR or join our [Discord](https://discord.gg/CZQvnuRV94). For enterprise use cases, please [contact us](mailto:support@log10.io) to set up a shared slack channel.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flog10-io%2Flog10","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flog10-io%2Flog10","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flog10-io%2Flog10/lists"}