https://github.com/comet-ml/comet-llm-legacy
This repo was replaced by opik: https://github.com/comet-ml/opik
https://github.com/comet-ml/comet-llm-legacy
Last synced: 7 months ago
JSON representation
This repo was replaced by opik: https://github.com/comet-ml/opik
- Host: GitHub
- URL: https://github.com/comet-ml/comet-llm-legacy
- Owner: comet-ml
- License: mit
- Created: 2024-09-01T14:17:02.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-09-02T10:39:01.000Z (over 1 year ago)
- Last Synced: 2025-06-06T05:46:43.716Z (12 months ago)
- Language: Python
- Size: 1.26 MB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
CometLLM was replaced by opik - see https://github.com/comet-ml/opik
CometLLM is a tool to log and visualize your LLM prompts and chains. Use CometLLM to identify effective prompt strategies, streamline your troubleshooting, and ensure reproducible workflows!
## ⚡️ Quickstart
Install `comet_llm` Python library with pip:
```bash
pip install comet_llm
```
If you don't have already, [create your free Comet account](https://www.comet.com/signup/?utm_source=comet_llm&utm_medium=referral&utm_content=github&framework=llm) and grab your API Key from the account settings page.
Now you are all set to log your first prompt and response:
```python
import comet_llm
comet_llm.log_prompt(
prompt="What is your name?",
output=" My name is Alex.",
api_key="",
)
```
## 🎯 Features
- [x] Log your prompts and responses, including prompt template, variables, timestamps and duration and any metadata that you need.
- [x] Visualize your prompts and responses in the UI.
- [x] Log your chain execution down to the level of granularity that you need.
- [x] Visualize your chain execution in the UI.
- [x] Automatically tracks your prompts when using the OpenAI chat models.
- [x] Track and analyze user feedback.
- [ ] Diff your prompts and chain execution in the UI.
## 👀 Examples
To log a single LLM call as an individual prompt, use `comet_llm.log_prompt`. If you require more granularity, you can log a chain of executions that may include more than one LLM call, context retrieval, or data pre- or post-processing with `comet_llm.start_chain`.
### Log a full prompt and response
```python
import comet_llm
comet_llm.log_prompt(
prompt="Answer the question and if the question can't be answered, say \"I don't know\"\n\n---\n\nQuestion: What is your name?\nAnswer:",
prompt_template="Answer the question and if the question can't be answered, say \"I don't know\"\n\n---\n\nQuestion: {{question}}?\nAnswer:",
prompt_template_variables={"question": "What is your name?"},
metadata= {
"usage.prompt_tokens": 7,
"usage.completion_tokens": 5,
"usage.total_tokens": 12,
},
output=" My name is Alex.",
duration=16.598,
)
```
[Read the full documentation for more details about logging a prompt](https://www.comet.com/docs/v2/guides/large-language-models/llm-project/#logging-prompts-to-llm-projects).
### Log a LLM chain
```python
from comet_llm import Span, end_chain, start_chain
import datetime
from time import sleep
def retrieve_context(user_question):
if "open" in user_question:
return "Opening hours: 08:00 to 17:00 all days"
def llm_answering(user_question, current_time, context):
prompt_template = """You are a helpful chatbot. You have access to the following context:
{context}
The current time is: {current_time}
Analyze the following user question and decide if you can answer it, if the question can't be answered, say \"I don't know\":
{user_question}
"""
prompt = prompt_template.format(
user_question=user_question, current_time=current_time, context=context
)
with Span(
category="llm-call",
inputs={"prompt_template": prompt_template, "prompt": prompt},
) as span:
# Call your LLM model here
sleep(0.1)
result = "Yes we are currently open"
usage = {"prompt_tokens": 52, "completion_tokens": 12, "total_tokens": 64}
span.set_outputs(outputs={"result": result}, metadata={"usage": usage})
return result
def main(user_question, current_time):
start_chain(inputs={"user_question": user_question, "current_time": current_time})
with Span(
category="context-retrieval",
name="Retrieve Context",
inputs={"user_question": user_question},
) as span:
context = retrieve_context(user_question)
span.set_outputs(outputs={"context": context})
with Span(
category="llm-reasoning",
inputs={
"user_question": user_question,
"current_time": current_time,
"context": context,
},
) as span:
result = llm_answering(user_question, current_time, context)
span.set_outputs(outputs={"result": result})
end_chain(outputs={"result": result})
main("Are you open?", str(datetime.datetime.now().time()))
```
[Read the full documentation for more details about logging a chain](https://www.comet.com/docs/v2/guides/large-language-models/llm-project/#logging-chains-to-llm-projects).
## ⚙️ Configuration
You can configure your Comet credentials and where you are logging data to:
| Name | Python parameter name | Environment variable name |
| -------------------- | --------------------- | ------------------------- |
| Comet API KEY | api_key | COMET_API_KEY |
| Comet Workspace name | workspace | COMET_WORKSPACE |
| Comet Project name | project | COMET_PROJECT_NAME |
## 📝 License
Copyright (c) [Comet](https://www.comet.com/site/) 2023-present. `cometLLM` is free and open-source software licensed under the [MIT License](https://github.com/comet-ml/comet-llm/blob/master/LICENSE).
CometLLM was replaced by opik - see https://github.com/comet-ml/opik