{"id":50980568,"url":"https://github.com/Visual-AI/SPoT","last_synced_at":"2026-07-07T11:00:31.382Z","repository":{"id":365304797,"uuid":"1168674614","full_name":"Visual-AI/SPoT","owner":"Visual-AI","description":"Official code for paper \"Surgical Post-Training: Cutting Errors, Keeping Knowledge\"","archived":false,"fork":false,"pushed_at":"2026-06-16T18:34:38.000Z","size":113,"stargazers_count":18,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-16T20:23:37.568Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Visual-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-27T17:04:10.000Z","updated_at":"2026-06-16T18:34:43.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Visual-AI/SPoT","commit_stats":null,"previous_names":["visual-ai/spot"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/Visual-AI/SPoT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Visual-AI%2FSPoT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Visual-AI%2FSPoT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Visual-AI%2FSPoT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Visual-AI%2FSPoT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Visual-AI","download_url":"https://codeload.github.com/Visual-AI/SPoT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Visual-AI%2FSPoT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35225029,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-19T14:00:18.565Z","updated_at":"2026-07-07T11:00:31.373Z","avatar_url":"https://github.com/Visual-AI.png","language":"Python","funding_links":[],"categories":["🎭 OPD with Black-Box / Outcome-Based Teachers"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention\n\n\n[![arXiv](https://img.shields.io/badge/arXiv-2603.01683-b31b1b.svg)](https://arxiv.org/abs/2603.01683)\n[![Model](https://img.shields.io/badge/🤗%20Model-Qwen3--8B--SPoT-blue)](https://huggingface.co/linius/Qwen3-8B-SPoT)\n[![Model](https://img.shields.io/badge/🤗%20Model-Llama--3.1--8B--Instruct--SPoT-blue)](https://huggingface.co/linius/Llama3.1-8B-SPoT)\n[![Dataset](https://img.shields.io/badge/🤗%20Dataset-connect4-blue)](https://huggingface.co/datasets/linius/connect4)\n\n*SPoT is a proximal on-policy distillation framework that uses a black-box Oracle to minimally correct student failures, improving reasoning while preserving prior knowledge through a reward-based binary optimization objective.*\n\n\u003c/div\u003e\n\n\n## 📰 News\n\n- **[2026-06-17]** Our SPoT-tuned Llama-3.1-8B-Instruct checkpoint is live on Hugging Face: [linius/Llama3.1-8B-SPoT](https://huggingface.co/linius/Llama3.1-8B-SPoT).\n- **[2026-05-15]** Paper updated to v2 with the proximal on-policy distillation framing and knowledge-retention analysis.\n- **[2026-03-05]** Our SPoT-tuned Qwen3-8B checkpoint is live on HuggingFace — try it yourself! [linius/Qwen3-8B-SPoT](https://huggingface.co/linius/Qwen3-8B-SPoT)\n- **[2026-03-04]** The Connect4 OOD reasoning evaluation dataset is now publicly available: [linius/connect4](https://huggingface.co/datasets/linius/connect4).\n\n\n## 📊 Main Results\n\nSPoT consistently outperforms all baselines across in-domain reasoning, OOD reasoning, and general instruction following. Crucially, while every competing method trades off at least one capability, **SPoT is the only approach that improves all three simultaneously**.\n\n### Qwen3-8B\n\n| Method | In-domain Avg | OOD Avg | IFEval | Overall Avg |\n|---|---|---|---|---|\n| Qwen3-8B (base) | 46.8 | 29.9 | 83.0 | 47.1 |\n| + SFT | 41.0 (-5.8) | 25.5 (-4.4) | 79.6 (-3.4) | 41.8 (-5.3) |\n| + RFT | 47.3 (+0.5) | 26.1 (-3.8) | 81.5 (-1.5) | 46.4 (-0.7) |\n| + SFT+ | 50.5 (+3.7) | 30.7 (+0.8) | 80.0 (-3.0) | 49.4 (+2.3) |\n| **+ SPoT (ours)** | **52.1 (+5.3)** | **41.4 (+11.5)** | **84.8 (+1.8)** | **53.3 (+6.2)** |\n\n### Llama-3.1-8B-Instruct\n\n| Method | In-domain Avg | OOD Avg | IFEval | Overall Avg |\n|---|---|---|---|---|\n| Llama-3.1-8B-Instruct (base) | 18.6 | 16.8 | 73.6 | 24.3 |\n| + SFT | 18.0 (-0.6) | 15.7 (-1.1) | 62.1 (-11.5) | 22.4 (-1.9) |\n| + RFT | 18.0 (-0.6) | 17.2 (+0.4) | 71.2 (-2.4) | 23.7 (-0.6) |\n| + SFT+ | 19.9 (+1.3) | 16.7 (-0.1) | 68.6 (-5.0) | 24.6 (+0.3) |\n| **+ SPoT (ours)** | **20.7 (+2.1)** | **18.5 (+1.7)** | 73.2 (-0.4) | **26.0 (+1.7)** |\n\nBenchmarks: AIME24/25, AMC23, MATH-500, Minerva, OlympiadBench (in-domain); GPQA-Diamond, Connect4 (OOD); IFEval (instruction following). The OOD gain on Connect4 alone is **+25.1 points** for Qwen3-8B (10.9 → 36.0).\n\n\n## 🔧 Data Pipeline\n\nThe pipeline generates proximal on-policy contrastive pairs `(x, y⁻, y⁺)` where `y⁺` is a minimally-edited correction of the model's wrong response `y⁻`.\n\n```\nRaw Dataset  →  Error Elicitation  →  Oracle Rectification  →  Contrastive Pairs\n  (DAPO)          (Model Inference)      (Gemini 2.5 Pro)        (x, y⁻, y⁺)\n```\n\n### Step 1: Error Elicitation\n\nRun inference on the DAPO-Math dataset to collect model failures:\n\n```bash\nbash scripts/run_dapo_inference.sh\n```\n\nOr with custom options:\n\n```bash\npython scripts/parallel_inference_dapo.py \\\n    --model Qwen/Qwen3-8B \\\n    --data_path data/dapo_5k/data.jsonl \\\n    --output_dir output/dapo_5k_inference \\\n    --num_gpus 8 \\\n    --temperature 0.7 \\\n    --top_p 0.8 \\\n    --max_tokens 32768\n```\n\nOutputs `errors.jsonl` (incorrect predictions) and `all_results.jsonl`.\n\n### Step 2: Oracle Rectification\n\nUse a black-box Oracle such as Gemini 2.5 Pro to surgically correct the errors (supports resuming):\n\n```bash\n# Correction mode: correct student errors while preserving style\npython scripts/correct_errors_parallel.py \\\n    --input output/dapo_5k_inference/errors.jsonl \\\n    --output data/gemini_corrected.jsonl \\\n    --workers 200\n```\n\n\n## 🏋️ Training\n\nSPoT uses a reward-based binary cross-entropy objective (implemented as `bco_pair`) over the proximal contrastive pairs `(x, y⁻, y⁺)` produced by the data pipeline. Training is full finetune (no LoRA) on Qwen3-8B with DeepSpeed ZeRO-2.\n\n### Install\n\n```bash\npip install transformers==4.53.3 trl==0.20.0 accelerate==1.10.0 \\\n            torch==2.7.0 deepspeed flash-attn wandb\n```\n\n### Run\n\n```bash\nbash train/dpo_sft_qwen3_accelerate.sh \u003coutput_ckpt\u003e \u003cdata.jsonl\u003e [base_model]\n```\n\nExample:\n\n```bash\nbash train/dpo_sft_qwen3_accelerate.sh \\\n    ckpts/qwen3_spot \\\n    data/gemini_corrected.jsonl \\\n    Qwen/Qwen3-8B\n```\n\nThe launcher autodetects GPU count via `nvidia-smi -L` and dispatches with `accelerate launch` using `train/accelerate_config.yaml` (DeepSpeed ZeRO-2, bf16). After training, the saved checkpoint is re-serialized in bf16 to halve disk usage.\n\n### Data format\n\nEach line of the input JSONL must contain:\n\n| Key | Role |\n|---|---|\n| `question` (or `prompt`) | User query |\n| `gemini_corrected_answer` | Chosen response `y⁺` (Oracle correction) |\n| `original_answer` | Rejected response `y⁻` (model's wrong output) |\n\n### Default hyperparameters\n\n| | |\n|---|---|\n| Loss | `bco_pair`, β=0.1 |\n| Learning rate | 1e-6, cosine schedule, warmup 0.05 |\n| Epochs | 2 |\n| Batch | 2 per GPU × 2 grad accum |\n| Max sequence length | 8192 |\n| Precision | bf16, FlashAttention-2, gradient checkpointing |\n\nOverride any of the above by editing `train/dpo_sft_qwen3_accelerate.sh`.\n\n### WandB\n\nLogging defaults to project `dpo-sft-qwen3` with no entity. Set `WANDB_ENTITY=\u003cyour-team\u003e` before launching, or pass `--report_to none` to disable.\n\n\n## 🧪 Evaluation\n\n### Supported Benchmarks\n\n| Task string | Benchmark | Type |\n|---|---|---|\n| `custom\\|aime24\\|0\\|0` | AIME 2024 | Math |\n| `custom\\|aime25\\|0\\|0` | AIME 2025 | Math |\n| `custom\\|amc23\\|0\\|0` | AMC 2023 | Math |\n| `custom\\|math_500\\|0\\|0` | MATH-500 | Math |\n| `custom\\|minerva\\|0\\|0` | Minerva Math | Math |\n| `custom\\|olympiadbench\\|0\\|0` | OlympiadBench | Math |\n| `custom\\|gpqa:diamond\\|0\\|0` | GPQA-Diamond | Science |\n| `custom\\|ifeval_no_thinking\\|0\\|0` | IFEval | Instruction Following |\n| `connect4` | Connect4 (OOD) | Game Reasoning |\n\nAll benchmarks except Connect4 use `sober_eval/main.py`. Connect4 uses `eval/eval_connect4.py`.\n\n\u003e **Connect4** serves as an OOD reasoning benchmark with verifiable intermediate steps. Game states are dynamically generated via [GAMEBoT](https://github.com/Visual-AI/GAMEBoT) to prevent data contamination.\n\n### Multi-Trial Evaluation\n\nLoads the model once and duplicates the dataset N times — more efficient than re-running per seed:\n\n```bash\nbash run_evaluation_multi_trial_duplicated_data.sh /path/to/model --disable-thinking\n```\n\nResults are saved to `evaluation_results/aggregated_results.json`.\n\n\n## 📖 Citation\n\nIf you find this work useful, please cite:\n\n```bibtex\n@article{lin2026surgical,\n      title={Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention},\n      author={Wenye Lin and Kai Han},\n      year={2026},\n      journal={arXiv preprint arXiv:2603.01683}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVisual-AI%2FSPoT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FVisual-AI%2FSPoT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVisual-AI%2FSPoT/lists"}