https://github.com/hunkim/solar-as-judge
https://github.com/hunkim/solar-as-judge
Last synced: 9 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/hunkim/solar-as-judge
- Owner: hunkim
- Created: 2024-06-15T13:25:19.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-20T04:50:54.000Z (almost 2 years ago)
- Last Synced: 2025-08-16T22:10:25.278Z (10 months ago)
- Language: Python
- Size: 8.79 KB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# solar-as-judge
## How to use
Set the UPSTAGE_API_KEY environment variable. Obtain your key from the Upstage console at .
```bash
pip install solar-as-judge
```
```python
import solar_as_judge as saj
# test prompt with an optional ground truth.
prompt = "Please extract one keyword from this text: I love you so much"
ground_truth = "love"
# The outcome of the A and B language models (AB testing).
A_answer = "love"
B_answer = "so much"
# Check the scores and the winner.
# If they are consistent, then determine the final score.
a_score, b_score = saj.judge(prompt, A_answer, B_answer, ground_truth)
print(a_score, b_score)
```
## Get detailed scores
```python
import solar_as_judge as saj
# test prompt with an optional ground truth.
prompt = "Please extract one keyword from this text: I love you so much"
ground_truth = "love"
# The outcome of the A and B language models (AB testing).
A_answer = "love"
B_answer = "so much"
# Get scores separately.
A_score = saj.get_judge_score(prompt, A_answer, ground_truth)
B_score = saj.get_judge_score(prompt, B_answer, ground_truth)
# Determine the winner.
winner = saj.get_winner(prompt, A_answer, B_answer, ground_truth)
print(A_score, B_score, winner)
```