https://github.com/aatakansalar/llm-prompt-semantic-diff
A CLI tool for managing and comparing LLM prompts using semantic diffing instead of traditional text-based comparison.
https://github.com/aatakansalar/llm-prompt-semantic-diff
Last synced: 3 months ago
JSON representation
A CLI tool for managing and comparing LLM prompts using semantic diffing instead of traditional text-based comparison.
- Host: GitHub
- URL: https://github.com/aatakansalar/llm-prompt-semantic-diff
- Owner: aatakansalar
- License: mit
- Created: 2025-06-27T19:56:58.000Z (about 1 year ago)
- Default Branch: master
- Last Pushed: 2025-07-01T20:32:15.000Z (about 1 year ago)
- Last Synced: 2026-03-28T03:47:03.135Z (3 months ago)
- Language: Python
- Size: 15.6 KB
- Stars: 6
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-cli-apps-in-a-csv - LLM Prompt Semantic Diff - A CLI tool for managing and comparing LLM prompts using semantic diffing instead of traditional text-based comparison. (<a name="diff"></a>Diff)
- awesome-cli-apps - LLM Prompt Semantic Diff - A CLI tool for managing and comparing LLM prompts using semantic diffing instead of traditional text-based comparison. (<a name="diff"></a>Diff)
README
# LLM Prompt Semantic Diff
A CLI tool for managing and comparing LLM prompts using semantic diffing instead of traditional text-based comparison.
[](LICENSE)
[](https://python.org)
## Overview
LLM Prompt Semantic Diff delivers a lightweight command‑line workflow for managing, packaging, and *semantic* diffing of Large Language Model prompts. It addresses the blind spot where ordinary text‑based `git diff` fails to reveal meaning‑level changes that materially affect model behaviour.
Read More: https://medium.com/@aatakansalar/catching-prompt-regressions-before-they-ship-semantic-diffing-for-llm-workflows-feb3014ccac3
## Key Features
- **F-1**: `prompt init` - Generates skeletal prompt files and default manifests
- **F-2**: `prompt pack` - Embeds prompts into `.pp.json` with semantic versioning
- **F-3**: `prompt diff` - Semantic comparison with percentage scores and exit codes
- **F-4**: Dual embedding providers (OpenAI cloud + SentenceTransformers local)
- **F-5**: JSON output for CI/CD integration
- **F-6**: Schema validation for all manifests
- **F-7**: Comprehensive test suite with >75% coverage
## Installation
### Install from PyPI (Recommended)
```bash
pip install llm-prompt-semantic-diff
```
### Install from source
```bash
git clone https://github.com/aatakansalar/llm-prompt-semantic-diff
cd llm-prompt-semantic-diff
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -e .
```
## Quick Start
### 1. Initialize a New Prompt
```bash
prompt init my-greeting
```
Creates `my-greeting.prompt` and `my-greeting.pp.json` with default structure.
### 2. Package an Existing Prompt
```bash
prompt pack my-prompt.prompt
```
Generates embeddings and creates a versioned manifest.
### 3. Compare Prompt Versions
```bash
# Human-readable output
prompt diff v1.pp.json v2.pp.json --threshold 0.8
# JSON output for CI/CD
prompt diff v1.pp.json v2.pp.json --json --threshold 0.8
```
Returns exit code 1 if similarity below threshold.
### 4. Validate Manifest Schema
```bash
prompt validate my-prompt.pp.json
```
## Embedding Providers
### Local (Default)
Uses SentenceTransformers with `all-MiniLM-L6-v2` model:
```bash
prompt pack my-prompt.prompt --provider sentence-transformers
```
### Cloud (OpenAI)
Requires `OPENAI_API_KEY` environment variable:
```bash
export OPENAI_API_KEY="your-api-key"
prompt pack my-prompt.prompt --provider openai
```
## CI/CD Integration
Use `--json` flag for machine-readable output:
```yaml
- name: Check prompt changes
run: |
prompt diff main.pp.json feature.pp.json --json --threshold 0.8
if [ $? -eq 1 ]; then
echo "Prompt changes exceed threshold - review required"
exit 1
fi
```
## Manifest Format
Prompts are packaged into `.pp.json` files:
```json
{
"content": "Your prompt text here...",
"version": "0.1.0",
"embeddings": [0.1, -0.2, 0.3, ...],
"description": "Optional description",
"tags": ["category", "type"],
"model": "gpt-4"
}
```
## Example Workflow
```bash
# Create new prompt
prompt init greeting
# Edit greeting.prompt file
# ... make changes ...
# Package with embeddings
prompt pack greeting.prompt
# Create modified version
cp greeting.prompt greeting-v2.prompt
# ... make more changes ...
prompt pack greeting-v2.prompt
# Compare versions
prompt diff greeting.pp.json greeting-v2.pp.json
# Output:
# Semantic similarity: 85.2%
# Threshold: 80.0%
# Above threshold: Yes
# Version A: 0.1.0
# Version B: 0.1.0
```
## Security & Privacy
- **Local-first**: No data leaves your machine unless OpenAI provider is explicitly selected
- **API keys**: Only read from environment variables (`OPENAI_API_KEY`)
- **No telemetry**: No analytics, tracking, or hidden network calls
## Development
```bash
git clone https://github.com/aatakansalar/llm-prompt-semantic-diff
cd llm-prompt-semantic-diff
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -e ".[dev]"
pytest tests/ -v
```
## License
Licensed under the MIT License. See [LICENSE](LICENSE) for details.