{"id":51879682,"url":"https://github.com/19pine-ai/interaction-scaling","last_synced_at":"2026-07-25T11:01:27.731Z","repository":{"id":371219635,"uuid":"1274994247","full_name":"19PINE-AI/interaction-scaling","owner":"19PINE-AI","description":"Interaction Scaling: a third inference-time compute axis grounded in environment feedback — proposer–reviewer harness, grounded evaluation, and the paper source.","archived":false,"fork":false,"pushed_at":"2026-07-14T01:57:18.000Z","size":24745,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-14T03:28:26.013Z","etag":null,"topics":["ai-research","benchmark","code-generation","inference-scaling","large-language-models","llm","llm-agents","reasoning","reproducible-research","self-refinement","test-time-compute"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/19PINE-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-20T05:52:35.000Z","updated_at":"2026-07-14T01:57:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/19PINE-AI/interaction-scaling","commit_stats":null,"previous_names":["19pine-ai/interaction-scaling"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/19PINE-AI/interaction-scaling","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Finteraction-scaling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Finteraction-scaling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Finteraction-scaling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Finteraction-scaling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/19PINE-AI","download_url":"https://codeload.github.com/19PINE-AI/interaction-scaling/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Finteraction-scaling/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35877013,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-25T02:00:06.922Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-research","benchmark","code-generation","inference-scaling","large-language-models","llm","llm-agents","reasoning","reproducible-research","self-refinement","test-time-compute"],"created_at":"2026-07-25T11:01:27.071Z","updated_at":"2026-07-25T11:01:27.723Z","avatar_url":"https://github.com/19PINE-AI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Interaction Scaling\n\n**A third test-time compute axis, grounded in environment feedback.**\n\nThis repository contains the full research codebase, experiment harnesses, and paper\nsource for *Interaction Scaling* — a study of how agents improve their outputs by\niteratively interacting with an external environment rather than only reasoning longer\nor sampling more.\n\n\u003e **Paper:** *Grounding the Loop on Both Sides: Interaction as a Third Test-Time Compute\n\u003e Axis, and Why Its Gains Are Invisible Without Grounded Evaluation*\n\u003e Bojie Li, Pine AI (`boj@19pine.ai`) and Noah Shi, University of Washington.\n\u003e **arXiv: [2607.11598](https://arxiv.org/abs/2607.11598)** · source in [`paper/`](paper/).\n\n---\n\n## TL;DR\n\nTest-time compute is dominated by two axes that both operate inside a model's own token\nspace: **reasoning** (longer chains of thought) and **sampling** (best-of-*N*). Both are\nbounded — by the data-processing inequality (DPI) — by the information already in the\nweights and the prompt.\n\nWe study a third axis: **interaction** with an environment that returns *grounded*\nfeedback (execution results, rendered layout geometry, fact verification). Because that\nfeedback imports information from outside the model, it can exceed the reasoning/sampling\nceiling. The central claim is that a single variable — **grounding** — governs this axis,\nand it must hold on *both* sides of the loop:\n\n- **Grounded feedback** drives improvement past the reasoning/sampling ceiling.\n- **Grounded evaluation** is required even to *measure* that improvement. A default VLM\n  judge is structurally blind to the geometric defects the loop fixes (it rates 14 of 15\n  single-shot academic figures \"perfect\" when only 3 are actually clean); a deterministic\n  DOM-geometry instrument reveals the real gains.\n\n## Headline results\n\n| Setup | Result |\n|---|---|\n| Code harness (Sonnet 4, 3-run mean) | 66.7 ± 6.7% → **100.0 ± 0.0%** (+33.3 pp) |\n| Academic figures (DOM-geometry defects) | **−74%** (paired sign-test *p* \u003c 0.002) |\n| Dense slides | **−73%** |\n| Responsive web pages | **−47%** |\n| SVG/CSS animations | **−40%** |\n| Reasoning-only at matched 20K-token budget (code) | saturates at 73.3% / best-of-*N* 86.7% |\n| Every interaction strategy at 20K budget | **≥ 97.8%**; proposer–reviewer wins on token cost, zero seed variance |\n| Cross-model replication (3 seeds each) | Sonnet +33.3 pp / Qwen3-235B +22.2 pp / GPT-5 +20.0 pp (reviewed ceiling has zero seed variance) |\n| Held-out code suite (32 tasks, zero overlap) | 90.6% → 100.0% |\n| Distillation into 8B student | partial transfer of harness behavior |\n\nA grounded VLM **reviewer** can even make visual layouts *worse* — only the deterministic\ngeometric instrument both fixes and measures the defect. See the paper for the full\naccount.\n\n---\n\n## The framework in one table\n\nThe grounded-feedback taxonomy that organizes the whole study:\n\n| Type | Feedback source | Grounded? | Example |\n|---|---|---|---|\n| **0** | Self-review (same model re-reads its output) | No | Re-reading your own essay |\n| **1** | LLM cross-review (different model/prompt) | No | Re-reading it wearing a different hat |\n| **2** | Static tools (lint, type-check, structural validators) | Yes (pre-execution) | A copy editor checking grammar |\n| **3a** | Execution feedback (tests, errors, runtime) | Yes | Running the experiment |\n| **3b** | Visual rendering feedback (rendered layout, geometry) | Yes | Looking at the rendered page |\n| **3c** | Temporal feedback (video / animation keyframes) | Yes | Watching the playback |\n| **3d** | Factual verification (search, knowledge base) | Yes | Checking a claim against evidence |\n\nTypes 0–1 form a Markov chain `T → A → F` and are DPI-bounded. Types 2–3 introduce an\nexternal variable `E`, breaking the chain and importing new, reliable information.\n\n---\n\n## Repository layout\n\n```\ninteraction-scaling/\n├── src/                      # Core library\n│   ├── agents/               # Proposer, reviewer, single-agent, meta-controller\n│   ├── feedback/             # Type 0–3d feedback providers (the taxonomy above)\n│   ├── budget/               # Token-budget allocator + tracker (think/do/review)\n│   ├── benchmarks/           # HumanEval / MBPP loaders\n│   ├── evaluation/           # Code eval, geometric checker, VLM/Gemini judges, rubrics\n│   ├── rendering/            # Headless-browser rendering (Playwright)\n│   ├── training/             # Distillation: trace collection, SFT, GRPO/RFT, students\n│   ├── experiments/          # Ablations, scaling curves, hard-benchmark runners\n│   ├── analysis/             # Tables + plots\n│   └── utils/                # LLM client, code utilities\n├── scripts/                  # Runnable entry points (sweeps, rescoring, figures, study)\n├── data/                     # Benchmark tasks + (small) training splits\n│   ├── hard_benchmarks/      # Hardened, de-saturated task suites\n│   └── training/             # SFT / GRPO data splits (large dumps are gitignored)\n├── notes/                    # Per-phase findings logs (the running research record)\n├── paper/                    # LaTeX source, figures, refs, built PDF\n├── study/                    # Human-preference study kit (protocol + static site)\n├── website/                  # Results website (Vite)\n├── results/                  # Eval outputs (small summaries in git; large dumps via Git LFS)\n└── docs/                     # Architecture \u0026 usage documentation\n```\n\n\u003e **Note on artifacts.** Model checkpoints and large training dumps remain\n\u003e **not** version-controlled (see [`.gitignore`](.gitignore)). The eval **results** behind\n\u003e every table and figure *are* tracked: small summary outputs live in normal git, while the\n\u003e large per-run dumps (`results/hard_benchmarks/`, `results/phase5/`, `results/phase6/`, and\n\u003e rendered result images) are stored via **Git LFS** — see [Getting the data](#getting-the-data-git-lfs).\n\nSee [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for a deeper tour of the modules and the\nproposer–reviewer loop, [`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md) for how to reproduce the\nmain results, and [`docs/RESEARCH_PLAN.md`](docs/RESEARCH_PLAN.md) for the original research\nplan the study grew out of. The running research record — one file per phase — lives in\n[`notes/`](notes/) (see [`notes/README.md`](notes/README.md) for an index).\n\n---\n\n## Getting the data (Git LFS)\n\nThe eval results behind every paper table and figure are committed to the repository. The\nlarge per-run dumps (`results/hard_benchmarks/`, `results/phase5/`, `results/phase6/`, and\nrendered result images, ~560 MB) are stored with [Git LFS](https://git-lfs.com); the small\nsummary outputs are in normal git and need no special handling.\n\nTo fetch the full data, install Git LFS **once per machine**, then clone or pull:\n\n```bash\n# 1. Install git-lfs\n#    macOS:  brew install git-lfs\n#    Ubuntu: sudo apt-get install git-lfs\ngit lfs install\n\n# 2a. Fresh clone — LFS objects download automatically\ngit clone git@github.com:19PINE-AI/interaction-scaling.git\n\n# 2b. Already cloned (or cloned without git-lfs installed)? Fetch the large objects:\ngit lfs pull\n```\n\nWithout Git LFS the large files appear as small text *pointer stubs* instead of the real\ndata; `git lfs pull` replaces them with the actual result dumps. The code, task suites,\npaper source, and small summary results all work fine without LFS.\n\n---\n\n## Quickstart\n\nThe project uses [`uv`](https://github.com/astral-sh/uv) and Python ≥ 3.11.\n\n```bash\n# Install dependencies (also installs the local `src` package, so scripts can\n# `import src` regardless of which directory they are launched from)\nuv sync\n\n# Install the headless browser used for visual rendering\nuv run playwright install chromium\n\n# Provide API keys for the model providers you intend to use\nexport ANTHROPIC_API_KEY=...    # Claude proposers/reviewers\nexport OPENAI_API_KEY=...       # GPT / cross-model baselines\nexport GEMINI_API_KEY=...       # Gemini video / visual rubric judges\n```\n\nA minimal end-to-end run of the proposer–reviewer code harness:\n\n```bash\n# Scaling-curve comparison (reasoning vs sampling vs interaction) on hard code tasks\nuv run python scripts/run_scaling_curves_code.py\n\n# Feedback-type controls (isolate Type 3a execution from extra reviewing)\nuv run python scripts/run_feedback_type_controls.py\n\n# Budget allocation sweep across think/do/review\nuv run python scripts/run_allocation_sweep.py\n```\n\nSee [`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md) for the full catalog of runnable scripts\nand what each one reproduces.\n\n---\n\n## Building the paper\n\n```bash\ncd paper\nlatexmk -pdf main.tex     # or: pdflatex main \u0026\u0026 bibtex main \u0026\u0026 pdflatex main \u0026\u0026 pdflatex main\n```\n\n`paper/main.pdf` is the built output. See [`paper/README.md`](paper/README.md).\n\n---\n\n## Citation\n\n```bibtex\n@misc{li2026interactionscaling,\n  title         = {Grounding the Loop on Both Sides: Interaction as a Third Test-Time\n                   Compute Axis, and Why Its Gains Are Invisible Without Grounded Evaluation},\n  author        = {Li, Bojie and Shi, Noah},\n  year          = {2026},\n  eprint        = {2607.11598},\n  archivePrefix = {arXiv},\n  primaryClass  = {cs.LG},\n  url           = {https://arxiv.org/abs/2607.11598},\n}\n```\n\n## License\n\nReleased under the [MIT License](LICENSE). The paper text and figures in\n[`paper/`](paper/) are © 2026 the authors.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F19pine-ai%2Finteraction-scaling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F19pine-ai%2Finteraction-scaling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F19pine-ai%2Finteraction-scaling/lists"}