{"id":51572014,"url":"https://github.com/accenture/contextecho","last_synced_at":"2026-07-10T20:02:39.891Z","repository":{"id":362576279,"uuid":"1259609252","full_name":"Accenture/ContextEcho","owner":"Accenture","description":"ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions","archived":false,"fork":false,"pushed_at":"2026-07-06T03:33:36.000Z","size":17575,"stargazers_count":12,"open_issues_count":0,"forks_count":4,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-06T05:07:04.171Z","etag":null,"topics":["agentic-coding","benchmark","llm","llm-evaluation","long-context","persona-drift"],"latest_commit_sha":null,"homepage":"https://accenture.github.io/ContextEcho/donate/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Accenture.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-04T17:18:10.000Z","updated_at":"2026-07-06T03:33:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Accenture/ContextEcho","commit_stats":null,"previous_names":["accenture/contextecho"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Accenture/ContextEcho","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Accenture%2FContextEcho","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Accenture%2FContextEcho/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Accenture%2FContextEcho/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Accenture%2FContextEcho/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Accenture","download_url":"https://codeload.github.com/Accenture/ContextEcho/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Accenture%2FContextEcho/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35341768,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic-coding","benchmark","llm","llm-evaluation","long-context","persona-drift"],"created_at":"2026-07-10T20:02:39.196Z","updated_at":"2026-07-10T20:02:39.884Z","avatar_url":"https://github.com/Accenture.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ContextEcho\n\n[![arXiv](https://img.shields.io/badge/arXiv-2605.24279-b31b1b.svg)](https://arxiv.org/abs/2605.24279)\n[![Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-contextecho2026-yellow)](https://huggingface.co/datasets/contextecho2026/persona-drift-contextecho)\n[![Donate a session](https://img.shields.io/badge/Donate%20a%20session-ContextEcho-167843)](https://accenture.github.io/ContextEcho/donate/)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/)\n\nCode release for:\n\n\u003e **ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions**\n\u003e Xianzhong Ding, Yangyang Yu, Changwei Liu, Bill Zhao. arXiv:2605.24279, 2026.\n\n## News\n\n- **June 2026** — ContextEcho is released alongside our [arXiv preprint](https://arxiv.org/abs/2605.24279), with the full harness, three donated sessions, and the per-cell evaluation corpus on [Hugging Face](https://huggingface.co/datasets/contextecho2026/persona-drift-contextecho).\n\nContextEcho measures whether a frontier LLM's trained Assistant persona\nsurvives long agentic-coding sessions (thousands of tool-using turns,\nhours of continuous use). It is a **25-probe identity suite + harness**\nthat snapshots a real Claude Code session prefix, forks the\nconversation state, and probes any chat-completions API target on the\nforked branch — without perturbing the main session.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/context_echo_intro.png\" alt=\"ContextEcho framework overview\" width=\"100%\" /\u003e\n\u003c/p\u003e\n\n## Key findings\n\nMeasured across **24 frontier models from 10 organizations** on three\nanonymized real Claude Code sessions (3,746–9,716 turns):\n\n| # | Finding | Takeaway |\n|---|---------|----------|\n| 1 | **Drift is general, not family-specific** | Persona drift appears across organizations, not just one model family. |\n| 2 | **Compaction does not reliably reset it** | In-session context compaction fails to restore the trained register. |\n| 3 | **A single-shot anchor restores the persona** | One ~110-token anchor turn recovers the trained register across measured targets, persisting 20+ turns. |\n| 4 | **Downstream effects are mode-dependent** | Drift can aid tool-using continuation, but in tool-free chat it breaks format contracts and inflates output length. |\n\nSee the [paper](https://arxiv.org/abs/2605.24279) for the full results and\nper-target tables, and [`REPRODUCE.md`](REPRODUCE.md) for the\nclaim-by-claim reproduction.\n\nThis repository contains the **runtime, experiment runners, analysis,\nand plotting code**. The donated session transcripts and the ~42K\nper-cell JSON responses are released separately as the\n[ContextEcho dataset](#the-released-dataset).\n\n---\n\n## Demo\n\n**Drift in action** — the same model, asked the same question, answers very\ndifferently late in a long session (left arm = real session context, right arm\n= length-matched neutral control):\n\n**Left: drift (no anchor) — Right: mitigation (A-anchor applied)**\n\nhttps://github.com/user-attachments/assets/5ee5629d-ea0a-4e3f-bf67-0cc72652c148\n\n**Try it live.** The repository ships an interactive, token-streamed\nside-by-side demo — type any probe and watch the two arms diverge in real time,\nwith a live drift score. See [`demo_live/`](demo_live/):\n\n```bash\npython -m demo_live.server   # then open http://localhost:8765\n```\n\n---\n\n## Reproducing the paper\n\nThe fastest path is the [`REPRODUCE.md`](REPRODUCE.md) document and the\n[`Makefile`](Makefile). Every paper figure has a one-command reproduction:\n\n```bash\n# Install dependencies\nmake setup\n\n# Verify the released data is PII-clean (audit greps for every redaction surface form)\nmake verify-pii\n\n# Render the headline forest plot (Fig. 2)\nmake fig2-forest\n\n# All body figures\nmake figs-body\n\n# All appendix figures\nmake figs-app\n\n# Show every available target\nmake help\n```\n\nFor the claim-by-claim reproduction table, see [REPRODUCE.md](REPRODUCE.md).\nFor each command's data dependencies, expected output, and approximate\nre-collection cost, see the same document.\n\n---\n\n## The released dataset\n\nThe per-cell JSONs and 3 donated session transcripts are hosted\nseparately on Hugging Face:\n**[contextecho2026/persona-drift-contextecho](https://huggingface.co/datasets/contextecho2026/persona-drift-contextecho)**.\n\nTo use this code with the released data, point it at the dataset:\n\n```bash\n# Option 1: symlink the released data tree into the code repo\nln -s /path/to/data_archive_release/results results\nln -s /path/to/data_archive_release/data    data\n\n# Option 2: download from Hugging Face\nhuggingface-cli download contextecho2026/persona-drift-contextecho \\\n    --repo-type=dataset --local-dir data_archive_release\nln -s data_archive_release/results results\nln -s data_archive_release/data    data\n```\n\nThe released dataset includes:\n\n- **3 redacted donor sessions** under `data/sessions/` (310 MB)\n- **41,921 per-cell JSON evaluations** under `results/` (705 MB) covering\n  the headline cross-compaction trajectory, the 23-target panel at $P_5$,\n  the 25-probe × 12-position panel-extension, A-anchor mitigation,\n  cross-judge audit, drift-onset sweep, stressor-surface compliance,\n  SWE-Bench-style continuation, and TerminalBench fresh-task null\n- **`DATASHEET.md`** (Datasheets-for-Datasets format)\n- **[`DATASET_CARD.md`](DATASET_CARD.md)** (public release and donation\n  pipeline summary generated from release metadata)\n- **`croissant.json`** (ML Commons Croissant 1.0 metadata)\n- **`LICENSE-DATA`** (CC-BY-SA-4.0) and **`LICENSE-CODE`** (Apache-2.0\n  reference)\n\nPII redaction was verified via a 13-pattern grep audit returning 0 hits;\nsee `data_archive_release/DATASHEET.md` §8 and `make verify-pii`.\n\n---\n\n## Repository layout\n\n| Directory | Contents |\n|---|---|\n| `harness/` | snapshot-then-probe runtime, multi-provider clients, judge, scorer, probe definitions, cost tracking |\n| `experiments/` | per-experiment runners (`run.py` per directory, e.g. `e08_cross_compaction/run.py`) |\n| `analysis/` | aggregation, statistical tests, paper-claim auditors |\n| `plotting/` | one `.py` per paper figure (e.g. `fig2_forest_panelwide.py`) |\n| `scripts/` | utilities (anonymizer, Croissant generator, per-experiment runner wrappers) |\n| `donate/` | local browser/terminal wizard for donating a redacted coding-agent session |\n| `schemas/` | donation manifest and public-ledger JSON schemas |\n| `archive/` | consent template + pre-registration documents |\n| `Makefile` | reproduction targets (`make help` for the full list) |\n| `REPRODUCE.md` | claim-by-claim reproduction table |\n| `DONOR_PRIVACY.md` | donor-facing privacy tiers and guarantees |\n| `DATA_USE_POLICY.md` | allowed and disallowed uses for donated data |\n| `requirements.txt` | Python dependencies |\n| `requirements-donate.txt` | minimal dependencies for the donation wizard |\n\n---\n\n## Re-collecting cells from scratch\n\nIf you have provider API access and want to re-run an experiment from\nzero (instead of using the released JSONs), each experiment is a self-\ncontained runner. From the repository root:\n\n```bash\nexport ANTHROPIC_API_KEY=...    # or whichever provider you're targeting\npython3 experiments/e08_cross_compaction/run.py\n```\n\nRunners are **idempotent** — they pick up where they stopped, so a\nkilled run can be resumed. See `REPRODUCE.md` §\"Re-running an experiment\nfrom scratch\" for the full inventory of runners with cost and wall-time\nestimates.\n\n---\n\n## Contributing\n\nContextEcho is a **living benchmark** — its value grows with every real coding\nsession it covers. We welcome contributions, and we credit them.\n\n- **Run it on your own session** for a free persona-drift report (yours to keep,\n  whether or not you donate).\n- **Donate a session or annotation** to join the\n  [contributor leaderboard](CONTRIBUTORS.md).\n- **Clear the contribution threshold** for **co-authorship on the next dataset\n  release** (rolling re-authorship — missing the first paper doesn't close the\n  door).\n\nFastest donor path:\n\nOpen the hosted donor page:\n\n```text\nhttps://accenture.github.io/ContextEcho/donate/\n```\n\nOr start the local scanner directly on macOS/Linux:\n\n```bash\ncurl -Ls https://github.com/Accenture/ContextEcho/raw/main/scripts/run-donate.sh | bash\n```\n\nOn Windows, use PowerShell:\n\n```powershell\npowershell -ExecutionPolicy Bypass -NoProfile -Command \"irm https://github.com/Accenture/ContextEcho/raw/main/scripts/run-donate.ps1 | iex\"\n```\n\nThese commands bootstrap a private `uv` runner with Python if needed, then launch\nthe local browser wizard from the GitHub repo with a supported Python 3.10-3.13 runtime. Verified\ndonations upload through the official ContextEcho relay:\n\n```text\nhttps://contextecho2026-context-echo-donation-relay.hf.space\n```\n\nThe hosted donor landing page lives at [`docs/donate/index.html`](docs/donate/index.html)\nand can be served by GitHub Pages or any static web host. It points donors to\nthe one-line local scanner command; the scanner opens the browser wizard,\ndiscovers local Claude Code/Codex sessions automatically, and keeps raw session\nhistory on the donor machine.\n\nThe hosted maintainer console lives at\n[`docs/maintainer/index.html`](docs/maintainer/index.html). It shows relay\nstatus, pending staging submissions, relay duplicate/lineage records, backfill\ncontrols, and one-record reset actions. Admin actions require the relay admin\ntoken, which is stored only in the browser tab session.\n\nSupported donor platforms: macOS, Linux, Windows PowerShell, and Windows via\nWSL/Git Bash.\n\nIf you already have `pipx` and Python 3.10+, you can run the package directly:\n\n```bash\nCONTEXTECHO_RELAY_URL=https://contextecho2026-context-echo-donation-relay.hf.space \\\npipx run --no-cache --spec git+https://github.com/Accenture/ContextEcho.git contextecho-donate\n```\n\nOr donate from this cloned repository with the local browser wizard:\n\n```bash\nmake setup-donate\npython3 -m donate --web\n```\n\nThe wizard discovers local Claude Code/Codex sessions, redacts and verifies\nthe selected session on your machine, writes `session.redacted.jsonl`,\n`manifest.json`, and `CONSENT.md`, then submits only those redacted artifacts\nto private maintainer review.\n\nPublic collection routes uploads through the server-side relay so the Hugging\nFace staging token is never shipped to donors. See\n[`DONATION_RELAY.md`](DONATION_RELAY.md).\n\n\u003e **Donor privacy.** ContextEcho analyzes **assistant behavior**, not\n\u003e **donor personality**. The default donation mode is **full redacted**, which\n\u003e removes **PII, secrets, paths, and custom scrub terms** while preserving task\n\u003e flow. Donors can choose **user-minimized** mode to selectively mask sensitive\n\u003e donor-authored spans after redaction. See [`DONOR_PRIVACY.md`](DONOR_PRIVACY.md) and\n\u003e [`DATA_USE_POLICY.md`](DATA_USE_POLICY.md).\n\n\u003e **Public credit choice.** Donors can provide name/email/institute for\n\u003e maintainer review while choosing to appear publicly as an anonymous donor.\n\u003e Public leaderboard rank still counts, but the released leaderboard and\n\u003e acknowledgments use an anonymous donor label. Maintainers can still see the\n\u003e submitted identity fields for review, support, and duplicate checks.\n\nMaintainers convert accepted private staging submissions into the next public\ndataset candidate with one command:\n\n```bash\nmake maintainer-intake\n```\n\nThis bootstraps a Python 3.14 maintainer environment, downloads staged\nsubmissions, checks duplicate/session lineage, runs technical review and quick\nvalidation, promotes accepted sessions, and regenerates/checks public metadata.\nAccepted sessions are copied into `data_archive_release_v2/data/sessions/`,\nwith consent, manifests, review reports, and a donation ledger under\n`data_archive_release_v2/data/donations/`. Use lower-level Make targets only\nfor debugging individual stages; promotion enforces quick validation.\nAfter approving contributor metadata changes in the maintainer console, run\n`make update-release-metadata`; it first syncs approved relay updates into the\nlocal release ledger/manifests, then regenerates `CONTRIBUTORS.md` and\n`DATASET_CARD.md`.\nSee [`MAINTAINER_DONATION_WORKFLOW.md`](MAINTAINER_DONATION_WORKFLOW.md) for\nthe full donor-to-ledger workflow and maintainer checklist.\n\nSee [`CONTRIBUTING.md`](CONTRIBUTING.md) for the points scale, the local-first\nredaction pipeline, session scoring rule, confidentiality rules, and exactly\nwhat you get.\n\n---\n\n## Citation\n\n```bibtex\n@article{ding2026contextecho,\n  title={ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions},\n  author={Ding, Xianzhong and Yu, Yangyang and Liu, Changwei and Zhao, Bill},\n  journal={arXiv preprint arXiv:2605.24279},\n  year={2026}\n}\n```\n\n---\n\n## License\n\n- **Code** (this repository): Apache-2.0\n- **Data** (the released dataset, separate host): CC-BY-SA-4.0 per donor\n  consent template\n\nThe dual license is standard for ML benchmarks that bundle data and\nsoftware — Apache-2.0 is the appropriate license for source code,\nwhile CC-BY-SA-4.0 is the appropriate license for the dataset.\n\n---\n\n## Acknowledgments\n\n- The three anonymized session donors, who consented to the release of\n  their Claude Code transcripts under the project's donor consent terms.\n- The maintainers of [SWE-bench](https://www.swebench.com/) and\n  [Terminal-Bench](https://www.tbench.ai/), whose task suites the\n  downstream-cost experiments build on.\n- The model providers whose chat-completions APIs are evaluated as\n  targets in the panel.\n\n---\n\n## Star History\n\n\u003ca href=\"https://star-history.com/#Accenture/ContextEcho\u0026Date\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://api.star-history.com/svg?repos=Accenture/ContextEcho\u0026type=Date\u0026theme=dark\" /\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://api.star-history.com/svg?repos=Accenture/ContextEcho\u0026type=Date\" /\u003e\n    \u003cimg alt=\"Star History Chart\" src=\"https://api.star-history.com/svg?repos=Accenture/ContextEcho\u0026type=Date\" /\u003e\n  \u003c/picture\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faccenture%2Fcontextecho","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faccenture%2Fcontextecho","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faccenture%2Fcontextecho/lists"}