https://github.com/19pine-ai/principal-loyalty
Whose Side Is Your Agent On? PrincipalBench — a multi-turn benchmark and post-training methods (prompt scaffold + per-token-KL distillation) for multi-party principal loyalty in LLM agents.
https://github.com/19pine-ai/principal-loyalty
agent-safety ai-alignment ai-safety benchmark knowledge-distillation large-language-models llm-agents llm-evaluation nlp principal-agent reinforcement-learning
Last synced: 16 days ago
JSON representation
Whose Side Is Your Agent On? PrincipalBench — a multi-turn benchmark and post-training methods (prompt scaffold + per-token-KL distillation) for multi-party principal loyalty in LLM agents.
- Host: GitHub
- URL: https://github.com/19pine-ai/principal-loyalty
- Owner: 19PINE-AI
- License: mit
- Created: 2026-06-20T05:49:03.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2026-06-30T02:21:56.000Z (about 1 month ago)
- Last Synced: 2026-06-30T04:11:13.146Z (about 1 month ago)
- Topics: agent-safety, ai-alignment, ai-safety, benchmark, knowledge-distillation, large-language-models, llm-agents, llm-evaluation, nlp, principal-agent, reinforcement-learning
- Language: Python
- Homepage: https://01.me/research/principal-loyalty
- Size: 64 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Principal Loyalty
### Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents
[](https://arxiv.org/abs/2606.30383)
[](LICENSE)
Most LLM agents today operate in a digital world: they call APIs, browse, and
serve the user who invoked them. A rapidly growing class of deployments is
**multi-party** — the agent represents a **principal** (who briefs the agent,
sends follow-up instructions, and receives results) while *also* conversing in a
separate channel with a **counterparty** whose interests may diverge:
negotiating with a vendor, screening inbound requests for an owner, mediating
between employees. Here *"help whoever you are talking to"* is the wrong
objective; the agent must stay **loyal to the principal it represents** without
over-refusing the principal's own cooperative requests.
This repository contains the benchmark, training recipes, evaluation harness,
analysis scripts, and companion website for our study of the **multi-party
loyalty problem**.
---
## What's here
| Contribution | Where |
|---|---|
| **PrincipalBench** — a 75-item multi-turn measurement instrument with leak probes, dual judges, and an integrity-audit gate | [`items/`](items/), [`src/`](src/) |
| **(M1) Prompt-time loyalty scaffold** — seven prioritized rules open-coded from 50+ failure trajectories | [`src/loyalty.py`](src/loyalty.py), [`src/agent.py`](src/agent.py) |
| **(M2) Per-token-KL distillation** — an on-policy recipe transferring a prompted Qwen3-32B teacher into 8B Qwen3 / Llama-3.1 students | [`scripts/pertoken_kl_collect.py`](scripts/pertoken_kl_collect.py), [`scripts/run_iter4_pertoken_kl.sh`](scripts/) |
| **Reproducibility pipeline** — every paper number regenerates from raw trajectories | [`scripts/recompute_all.py`](scripts/recompute_all.py) |
| **Companion website** — browse items, models, runs, and sample conversations | [`website/`](website/) |
| **Paper** | [arXiv:2606.30383](https://arxiv.org/abs/2606.30383) · [`paper_arxiv.tex`](paper_arxiv.tex), [`paper_arxiv.pdf`](paper_arxiv.pdf) |
---
## Key findings
- **A calibrated / over-refuse split.** Running PrincipalBench across 13 frontier
models exposes a sharp split — **≤20% harm** for calibrated models vs.
**53.6–75.3% harm** for over-refusers — that is invisible to single-turn
safety evaluations.
- **The scaffold works.** Under the seven-rule loyalty scaffold, Claude-Sonnet
reaches 21/108 (19.4%) harm, and all nine calibrated frontier models hold
≤20% harm.
- **Distillation transfers.** Per-token-KL distillation is the strongest
open-weight recipe we measure, moving an 8B student to harm 33 / leak 13.
- **A structural floor.** Both mechanisms move *along* a common
leak/over-refusal trade-off rather than crossing it — a floor that single-knob
RL (DAPO) also fails to break.
---
## Repository layout
```
principal-loyalty/
├── src/ # the benchmark harness — see docs/architecture.md
│ ├── harness.py # trajectory runner: (subject, arm, item) → JSONL
│ ├── items.py # item schema + loader
│ ├── agent.py # agent under test: plain / prompted / scaffolded arms
│ ├── counterparty.py # counterparty simulator (pressure strategies)
│ ├── loyalty.py # externalized LoyaltyState (the M1 scaffold)
│ ├── probe.py # leak-detection probe (lexical + LLM)
│ ├── scorer.py # per-item / per-cell metric aggregation
│ ├── reward.py # fast proxy reward for DAPO training
│ └── vendors.py # unified vendor layer (Anthropic/OpenAI/Google/OpenRouter)
├── items/ # benchmark items (JSON) — see docs/benchmark.md
│ ├── v0/ # 50 training items
│ ├── v0_75/ # 25 held-out items
│ └── v0_5/ # superseded dev-intermediate set (not part of release)
├── data/ # SFT / DPO / on-policy distillation datasets — see docs/training.md
├── scripts/ # eval, training, analysis, figure, release scripts (see Key scripts below)
├── figures/ # paper figures (PDF), regenerated by scripts/make_figs_arxiv.py
├── website/ # static React companion site — see docs/website.md
├── docs/ # extended documentation — see the Documentation section below
├── paper_arxiv.{tex,pdf,bib,bbl}, arxiv.sty, abstract.txt # the paper
├── RELEASE.md # building the clean public release tree
└── requirements.txt
```
> **Note on large artifacts.** Raw run trajectories (`runs/`, 150+ GB) and Hydra
> training outputs (`outputs/`) are git-ignored. The repository ships the items,
> code, datasets, figures, and paper; trajectories are regenerated by re-running
> the harness (see [docs/reproduce.md](docs/reproduce.md)).
### Key scripts
`scripts/` holds the full research history; these are the entry points you'll
actually use:
| Script | What it does |
|---|---|
| [`run_traj_only.py`](scripts/run_traj_only.py) · [`multi_rollout_eval.py`](scripts/multi_rollout_eval.py) | Generate trajectories / run a multi-seed grid eval |
| [`recompute_all.py`](scripts/recompute_all.py) | Recompute **every** paper number from `runs/` |
| [`paired_seed_test.py`](scripts/paired_seed_test.py) · [`per_arm_xvendor_wilcoxon.py`](scripts/per_arm_xvendor_wilcoxon.py) | The paper's statistical tests |
| [`make_figs_arxiv.py`](scripts/make_figs_arxiv.py) | Regenerate all paper figures |
| [`pertoken_kl_collect.py`](scripts/pertoken_kl_collect.py) · [`run_iter4_pertoken_kl.sh`](scripts/run_iter4_pertoken_kl.sh) | Per-token-KL distillation (mechanism M2) |
| [`run_dapo.sh`](scripts/run_dapo.sh) · [`run_full_pipeline.sh`](scripts/run_full_pipeline.sh) | DAPO baseline / end-to-end Llama pipeline |
| [`build_website_data.py`](scripts/build_website_data.py) · [`build_release.py`](scripts/build_release.py) | Refresh the website data / build the clean release tree |
---
## Quick start
```bash
# 1. install (the harness itself needs only anthropic + openai)
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# 2. set provider keys (only those you intend to use)
export ANTHROPIC_API_KEY=...
export OPENAI_API_KEY=...
export OPENROUTER_API_KEY=...
# 3. generate trajectories for a subject across all arms on the v0 items
python3 scripts/run_traj_only.py --seed-dirs runs/smoke_seed1 \
--subject claude-sonnet --counterparty claude-sonnet \
--items items/v0 --parallel 4
# 4. multi-seed grid eval (trajectories + scoring + aggregation)
python3 scripts/multi_rollout_eval.py --base runs/smoke \
--seeds 5 --counterparty claude-sonnet --parallel 4
```
The harness (`src/`) is a library, not a CLI — the scripts in `scripts/` drive
it (see [docs/architecture.md](docs/architecture.md)). For the full evaluation
sweep, training recipes, and figure regeneration, see the documentation below.
---
## Documentation
| Doc | Contents |
|---|---|
| [docs/benchmark.md](docs/benchmark.md) | Item schema, the six cells, the three arms, leak probes, dual judges, the integrity-audit gate |
| [docs/architecture.md](docs/architecture.md) | How the harness, scorer, and vendor layer fit together |
| [docs/reproduce.md](docs/reproduce.md) | Reproducing every number and figure in the paper |
| [docs/training.md](docs/training.md) | The per-token-KL distillation recipe and DAPO baselines |
| [docs/website.md](docs/website.md) | Building and refreshing the companion site |
| [RELEASE.md](RELEASE.md) | Building the clean public release tree |
---
## Citation
```bibtex
@misc{principalbench2026,
title = {Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents},
author = {Li, Bojie and Shi, Noah},
year = {2026},
eprint = {2606.30383},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
note = {PrincipalBench},
url = {https://arxiv.org/abs/2606.30383}
}
```
## License
Released under the [MIT License](LICENSE).