{"id":51879686,"url":"https://github.com/19pine-ai/latent-bridge-games","last_synced_at":"2026-07-25T11:01:29.095Z","repository":{"id":364319447,"uuid":"1239605582","full_name":"19PINE-AI/latent-bridge-games","owner":"19PINE-AI","description":"A learned continuous slow→fast latent channel for real-time game agents: frozen MiniCPM-o 4.5 (fast) + Qwen3-VL-8B-Thinking (slow), 33M-param bridge. The latent helps iff slow reasoning helps (T\u003eF). Paper: arXiv:2606.24470","archived":false,"fork":false,"pushed_at":"2026-06-29T09:32:31.000Z","size":80366,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-29T11:17:25.443Z","etag":null,"topics":["agents","arcade-learning-environment","atari","deep-learning","fast-slow","game-ai","llava","llm","machine-learning","multimodal","pytorch","real-time","reasoning","reinforcement-learning","vlm"],"latest_commit_sha":null,"homepage":"https://01.me/research/latent-bridge-games","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/19PINE-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-15T08:57:47.000Z","updated_at":"2026-06-29T09:32:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/19PINE-AI/latent-bridge-games","commit_stats":null,"previous_names":["bojieli/latent-bridge-games","19pine-ai/latent-bridge-games"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/19PINE-AI/latent-bridge-games","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Flatent-bridge-games","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Flatent-bridge-games/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Flatent-bridge-games/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Flatent-bridge-games/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/19PINE-AI","download_url":"https://codeload.github.com/19PINE-AI/latent-bridge-games/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Flatent-bridge-games/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35877013,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-25T02:00:06.922Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","arcade-learning-environment","atari","deep-learning","fast-slow","game-ai","llava","llm","machine-learning","multimodal","pytorch","real-time","reasoning","reinforcement-learning","vlm"],"created_at":"2026-07-25T11:01:28.412Z","updated_at":"2026-07-25T11:01:29.049Z","avatar_url":"https://github.com/19PINE-AI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The Latent Bridge: A Continuous Slow–Fast Channel for Real-Time Game Agents\n\n\u003e 🎬 **Deliverable: a working demo.** The fast/slow architecture is the means; the\n\u003e deliverable is a real-time agent you can watch play and replay. Recorded MP4 + live\n\u003e web playback + interactive website are the primary outputs; the experiments in\n\u003e [`docs/06_results.md`](docs/06_results.md) ground the demo's claims (roadmap:\n\u003e [`docs/07_next_steps.md`](docs/07_next_steps.md)).\n\u003e\n\u003e **Paper:** [arXiv:2606.24470](https://arxiv.org/abs/2606.24470) · **Website:** \u003chttps://01.me/research/latent-bridge-games\u003e\n\nWe want agents that operate a computer like a person — read the screen, issue inputs, close\nthe loop — and **real-time games are the hardest case**: the agent must act every few tens of\nmilliseconds while pursuing a goal that needs planning over seconds. No single open multimodal\nLLM does both: a **reasoning** VLM (Qwen3-VL-8B-Thinking, 8B) is ~1.5 s too slow for the ~15 Hz\ncontrol loop, while a **reactive** VLM (MiniCPM-o 4.5, 9B) has no deliberation. The fast/slow\nsplit is the fix — Thinking Machines' [Interaction Models](https://thinkingmachines.ai/blog/interaction-models/)\nmake it explicit via shared text/context; we test an **open** alternative.\n\nThis project investigates whether a learned continuous-valued **latent bridge** — project the\nslow model's residuals into the fast model's input-embedding space, LLaVA-style — beats the\nstandard **text channel** (the slow model writes a prompt suffix the fast model reads) between\ntwo **frozen** models at matched ~8–9 B scale, so the *channel*, not a capability gap, is the\nload-bearing variable. The 33 M-param bridge is the only trained component.\n\n**Headline finding:** the latent bridge helps *if and only if* slow reasoning helps the task\n(**T \u003e F**) — `L−F` tracks `T−F` at **Pearson r = 0.93**. Tuned per channel, the latent is never\nsignificantly worse than the text bridge and significantly better on **2 of 7** games; combining\nboth channels *interferes*, so couple via exactly **one**. Details below.\n\n## 🎯 Per-game scores — 8 Atari games, 12 episodes per cell (fixed-greedy view)\n\nThe best L vs T result per game (using whichever Stage A — bare or robust —\ngave the higher L score):\n\n| Game | F | T | **L** | L vs T |\n|---|---|---|---|---|\n| MsPacman | 256 ± 24 | 408 ± 88 | **628 ± 341** | **+54 %** |\n| Seaquest | 42 ± 19 | 63 ± 11 | **80 ± 0** | **+26 %** |\n| RoadRunner | 0 ± 0 | 475 ± 160 | **608 ± 29** | **+28 %** |\n| River Raid (robust SA) | 1033 ± 19 | 337 ± 77 | **612 ± 297** | **+82 %** |\n| SpaceInvaders (robust SA) | 107 ± 60 | 18 ± 18 | 15 ± 0 | recovered from 0 |\n| Enduro (robust SA) | 0.8 ± 1.0 | 4.9 ± 5.6 | **5.8 ± 2.5** | +18 % |\n| Q*bert (robust SA) | 25 ± 0 | **125 ± 0** | 50 ± 0 | T \u003e L (greedy; tie under tuned decoders) |\n| Pong | −21 ± 0 | −21 ± 0 | −21 ± 0 | reactive floor |\n\n\u003e ⚠️ **These are fixed-*greedy*-decoder numbers.** The latent's advantage over\n\u003e text is decoder-specific: a full decoder sweep (greedy, τ∈{0.3…1.5}) shows it\n\u003e vanishes at every fixed sampling temperature. The honest comparison tunes the\n\u003e action decoder per channel on held-out seeds (*best-achievable*): there the latent\n\u003e is **never significantly worse than text and significantly better on 2 of 7**\n\u003e (MsPacman, RoadRunner); the other 5 are ties. See the [paper](https://arxiv.org/abs/2606.24470)\n\u003e for the decoder-robust tables. The continuous-vs-categorical hypothesis these\n\u003e per-game scores once motivated is **retired** — emission statistics do not predict\n\u003e sign(L−T) (lexical-diversity r=+0.05, n.s.).\n\n**Current claim (decoder-robust)**: the latent bridge helps *if and only if* slow\nreasoning helps the task (T \u003e F) — `L−F` tracks `T−F` at **Pearson r = 0.93** across\n7 Atari games + MetaDrive (the controlled negative). Whether to couple is a property\nof the task, not the channel; if you couple, use exactly one channel (text+latent\ntogether *interferes*, −96 % on RoadRunner).\n\nThe largest *greedy* L−T gap (+82 %) is on **River Raid** after robust Stage A (a tie\nunder tuned decoders). The cleanest qualitative demo is **RoadRunner**: F=0 vs L=608\n(reproducible; an earlier run scored 967, but the F=0 baseline makes the magnitude\nrun-to-run-unstable — the L\u003eT direction is robust).\n\n### Stage A robustness recipe (the second-order finding)\n\nStage A trained on bare prompts becomes out-of-distribution when T appends a\ntext suffix or L prepends bridge tokens. We diagnosed this through three\nSpaceInvaders interventions (random-T, expert-T, aggressive-prompt all gave\nT=L=0) and confirmed by fixing it: `--suffix-prob=0.5` Stage A retraining\nbreaks the collapse.\n\n**Targeted, not universal**: applying robust SA to games where L \u003e T already\nworked (MsPacman, Seaquest) *hurt* — the slight bare-prompt accuracy drop\ndominated the suffix-robustness gain. The recipe is:\n- Use robust SA when T/L collapse to ~0 (SI, RR-bare, Q*bert, Enduro)\n- Don't use it when T/L already win (MsPacman, Seaquest)\n- Surprise: RoadRunner F=0 under bare SA was overfitting, not policy\n  stuckness — robust SA recovered F to 958 and all three strategies tie (~925-1000)\n\n### Detailed SpaceInvaders breakdown (diagnostic chain)\n| Strategy | bare Stage A | robust Stage A |\n|---|---|---|\n| F | 105 ± 0   | 107 ± 60 |\n| T | **0 ± 0** | **18 ± 18** |\n| L | **0 ± 0** | **15 ± 0** |\n\nThe L=T=0 collapse was diagnosed across four interventions (random-T,\nexpert-T, aggressive-prompt, all gave 0; robust-Stage-A retry recovered both\nT and L to nonzero). Bridge MI under expert-T was +0.024 nats — the bridge\n*did* learn structure — but the deployed policy still collapsed because of\nthe action head's OOD-brittleness, not the bridge itself.\n\n- L \u003e T claim: under fixed greedy decoding, 4-of-7 games; under the\n  decoder-robust best-achievable comparison (tune decoder per channel on\n  held-out seeds), the latent significantly wins 2-of-7 (MsPacman, RoadRunner),\n  ties the other 5, and never significantly loses.\n- SpaceInvaders diagnosis end-to-end validated: L=T=0 under bare Stage A;\n  L=15, T=18 under robust Stage A. The bridge mechanism was never broken;\n  Stage A OOD-brittleness was. PPO under deployment distribution\n  is the next step to close the F-L gap.\n\n### Slow-only S baseline (MsPacman, n=3)\n| Strategy | Score | Comment |\n|---|---|---|\n| **S (slow only, ~1 Hz)** | 113 ± 24 | Just use the big model: 4 s/decision, too slow |\n| F (fast only, 15 Hz) | 256 ± 24 | Reactive, no planning |\n| T (text bridge) | 408 ± 88 | Slow guides fast via text |\n| **L (latent bridge)** | **628 ± 341** | Slow guides fast via latents |\n\nOn MsPacman the ordering is **S \u003c F \u003c T \u003c L** — slow-only is too slow for real-time,\nfast-only lacks planning, and both bridges help (the latent most, under greedy decoding).\nThis is *not* a universal law: whether either bridge beats fast-only is task-dependent\n(the **T \u003e F** predictor), and the latent's edge over text is decoder-specific (see the\ncaveat above).\n\nThe full story (v1 cross-attn → v2 LLaVA-style redesign) is in\n[`docs/06_results.md`](docs/06_results.md).\n\n## What works (v2)\n\nThe latent bridge is **LLaVA-style**: the slow model produces N=8 latent tokens in the\nfast model's 4096-d input embedding space; they are **prepended to the fast model's\ninput sequence** so all 36 LLM layers attend over them through the standard causal\nattention path. This is the same architectural pattern that LLaVA, BLIP-2, Flamingo,\nand MiniCPM-o itself use for multimodal coupling.\n\n```\nslow model (Qwen3-VL-8B-Thinking)\n   └─ residuals at layer 24, last N=8 positions\n       └─ ThoughtProjection (4096 → 4096 → 4096 + LayerNorm, ~33M params trainable)\n            └─ N=8 latent tokens in fast model's embedding space\n                 └─ PREPENDED to fast model's input embedding sequence\n                      └─ fast model (MiniCPM-o 4.5) — all 36 LLM layers attend\n                           └─ action_head on last hidden state → action logits\n```\n\n## What didn't work (v1)\n\nThe original design tried mid-layer cross-attention into a 256-d ring buffer at LLM\ndepths 12 \u0026 24, with 71M trainable fast-side params. This **converged to KL=0.004 on\ntraining data but failed at deployment** (L=225 vs F=256, bimodal with 4/12 catastrophic\nepisodes). Three architectural variants (ungated / gated / gated+head-tune) all failed.\n\nWhy v1 failed:\n1. **No inductive bias** for arbitrary 256-d vectors — the LLM had no pretraining for\n   that format and ~5K Stage C samples wasn't enough to learn one from scratch.\n2. **Information bottleneck** — only 2 of 36 layers saw the bridge; the rest had to\n   propagate it via the residual stream alone.\n\nv2 solves both by matching the LLM's input-embedding pattern (text-like inductive bias)\nand using the full attention stack.\n\n## Hypotheses\n\n- **H1**: Latent bridge ≥ text bridge on games needing both reflex + planning.\n  **✅ Decoder-robust**: tuned per channel (held-out decoder selection), the latent is\n  never significantly worse than text and significantly better on 2/7 (MsPacman,\n  RoadRunner). The original greedy \"L \u003e T on 4/7\" over-credited the latent — the\n  advantage is greedy-specific (see the decoder-sensitivity note above).\n- **H2**: Latent-vs-text gap *grows* with strategic complexity. **❌ Refuted** — emission\n  statistics don't predict sign(L−T) (lexical-diversity *r* = +0.05, n.s.; the\n  continuous-vs-categorical hypothesis is retired). What *does* gate the bridge is the\n  behavioral predictor: it pays off iff slow reasoning beats reaction on the task (**T \u003e F**).\n- **H3**: A frozen base + a small trained channel recovers most of a unified upper bound.\n  **✅ Confirmed**: only the ~33 M-param slow-projection bridge trains; both base models are\n  frozen. (The latent's ceiling is its text teacher — Stage C distills L toward T.)\n\n## Repo layout\n\n```\nlatent-bridge-games/\n├── README.md\n├── docs/\n│   ├── 01_framing.md            # research thesis + scope\n│   ├── 02_related_work.md       # surveyed prior art\n│   ├── 03_experiment_plan.md    # experiment plan (updated for v2)\n│   ├── 04_architecture.md       # v2 LLaVA-style bridge spec\n│   ├── 05_status.md             # what was built + key empirical findings\n│   ├── 06_results.md            # paper-style results summary\n│   └── 07_next_steps.md         # roadmap\n├── paper/                          # LaTeX source + generated figures (main.pdf)\n├── web-react/                      # interactive website source (Vite/React)\n├── src/\n│   ├── env/atari_wrapper.py     # ALE wrapper + MsPacman/Frostbite/Seaquest RAM decoders\n│   ├── models/fast_model.py     # MiniCPM-o + v2 bridge-token prepend + vision cache\n│   ├── models/slow_model.py     # Qwen3-VL-8B-Thinking + trainable ThoughtProjection\n│   ├── bridge/ring_buffer.py    # (v1 legacy; unused in v2)\n│   ├── training/\n│   │   ├── stage_a_behavioral.py     # Stage A imitation (frozen base + action_head only)\n│   │   ├── stage_c_v2.py             # v2 Stage C: trainable slow ThoughtProjection only\n│   │   ├── stage_c_bridge.py         # (v1 legacy: cross-attn KL training)\n│   │   ├── prompts.py                # per-game Stage B text prompts\n│   │   └── imitation_data.py         # global 18-way action space + per-game maps\n│   └── eval/\n│       ├── benchmark.py              # multi-strategy multi-game eval harness\n│       └── mi_diagnostic.py          # bridge information-content diagnostic\n├── configs/                          # YAML run configs\n├── scripts/\n│   ├── collect_trajectories.py       # SB3-expert trajectory collection (CPU)\n│   ├── run_text_bridge_baseline.py   # T-trajectory collection (saves v2 raw residuals)\n│   ├── aggregate_results.py          # multi-eval comparison table\n│   ├── make_figures.py               # paper-quality matplotlib figures\n│   ├── \u003cgame\u003e_pipeline.sh            # per-game A→B→C→eval chains\n│   ├── README.md                     # guide to every script + canonical entry points\n│   └── ...\n├── tests/                            # 45 unit + integration tests (CPU-runnable)\n├── results/                          # per-condition raw eval outputs\n└── checkpoints/                      # Stage A + Stage C trained checkpoints\n```\n\n## Setup\n\n```bash\ngit clone https://github.com/19PINE-AI/latent-bridge-games\ncd latent-bridge-games\npython -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -e .                 # core: Atari repro (experts, training, eval, figures)\n# optional extras:\npip install -e \".[demo]\"         # narrated-demo build + Flask/SSE live server\npip install -e \".[metadrive]\"    # driving domain (the controlled negative)\n```\n\nThe two frozen base models are **not** in the repo. Point the harness at local\nweights via `LB_FAST_MODEL_PATH` / `LB_SLOW_MODEL_PATH` (or let `transformers`\npull `openbmb/MiniCPM-o-4_5` and `Qwen/Qwen3-VL-8B-Thinking` from the HF cache);\nset `HF_HUB_OFFLINE=1` to force the local copies. See `scripts/README.md` for a\nguide to every script and the canonical entry points.\n\n## Reproducing the headline number\n\n\u003e Trained checkpoints and `results/` are **not shipped** (both are gitignored). Train the\n\u003e Stage A behavioral-cloning policy first (`python -m src.training.stage_a_behavioral ...`;\n\u003e the per-game `scripts/*_pipeline.sh` run the full A→C→eval chain end-to-end). The steps\n\u003e below assume `checkpoints/stage_a/` already exists.\n\n```bash\n# 1. Collect T-trajectories (45 min on GPU)\nHF_HUB_OFFLINE=1 python scripts/run_text_bridge_baseline.py \\\n    --game MsPacman --episodes 10 --ticks 750 \\\n    --out-dir results/t_trajectories_v2\n\n# 2. Train v2 Stage C bridge (12 min on GPU)\nHF_HUB_OFFLINE=1 python -m src.training.stage_c_v2 \\\n    --trace 'results/t_trajectories_v2/MsPacman_seed*.pt' \\\n    --stage-a-ckpt checkpoints/stage_a/mspacman_sb3dqn_v2.pt \\\n    --out checkpoints/stage_c/v2_mspacman.pt\n\n# 3. F/T/L head-to-head eval (~60 min for 36 episodes)\nHF_HUB_OFFLINE=1 python -m src.eval.benchmark \\\n    --strategies F T L --games MsPacman --seeds 0 1 2 --episodes 4 \\\n    --fast-ckpt checkpoints/stage_a/mspacman_sb3dqn_v2.pt \\\n    --bridge-ckpt checkpoints/stage_c/v2_mspacman.pt \\\n    --out results/eval_v2_mspacman.json\n\n# 4. Aggregate + plot\npython scripts/aggregate_results.py\npython scripts/make_figures.py\n```\n\n## Hardware\n\nNVIDIA RTX Pro 6000, 96GB VRAM. Joint inference of MiniCPM-o 4.5 (bf16, 18GB) and\nQwen3-VL-8B-Thinking (bf16, 17GB) leaves ~60GB headroom for training/PPO batches.\nA single scaling ablation with Qwen3-30B-A3B-Thinking (~60GB) fits at inference with\n~10GB headroom; would need activation checkpointing or AWQ-4bit for joint training.\n\n## Status\n\n- [x] Joint inference validation (34GB VRAM, ~270ms cold tick)\n- [x] Stage A behavioral cloning + the **v2 Stage C latent bridge** (the core channel)\n- [x] Stage B text-bridge baseline (T = +59 % over F on MsPacman)\n- [x] Cross-game sweep: 7 Atari games + MetaDrive (the controlled negative)\n- [x] Stage A robustness recipe (`--suffix-prob=0.5`) — validates the OOD-brittleness diagnosis\n- [x] Decoder-sensitivity sweep + held-out **best-achievable** per-channel selection (2-of-7 sig. wins)\n- [x] Combined-channel (T+L) experiment — *interferes*; couple via exactly one channel\n- [x] **Behavioral predictor**: L−F tracks T−F, r = 0.93 (0.96 over all 16 cells)\n- [x] Bridge-replacement control (learned content tracks T \u003e F)\n- [x] 30B-A3B cross-scale ablation (more slow-model capacity does **not** widen L−T)\n- [x] Latent token-count (N) ablation N=4/8/16 (deploy-only N=16 best — no capacity ceiling)\n- [x] MI diagnostic; vision-token cache (latency option)\n- [x] Recorded MP4 demos + interactive website (`web-react` → `web-dist`)\n- [ ] Stage D PPO (online RL; will it recover SpaceInvaders?)\n- [ ] Slow-only S + Oracle O baselines tabulated for all games\n- [ ] Scale to the motivating target: real-time computer-use / game agents on phone \u0026 desktop\n\n## License\n\nMIT License — see [`LICENSE`](LICENSE). Copyright (c) 2026 Pine AI.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F19pine-ai%2Flatent-bridge-games","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F19pine-ai%2Flatent-bridge-games","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F19pine-ai%2Flatent-bridge-games/lists"}