{"id":49215908,"url":"https://github.com/lightseekorg/torchspec","last_synced_at":"2026-04-24T00:02:28.512Z","repository":{"id":340758329,"uuid":"1163082957","full_name":"lightseekorg/TorchSpec","owner":"lightseekorg","description":"A PyTorch native library for training speculative decoding models","archived":false,"fork":false,"pushed_at":"2026-04-19T05:29:43.000Z","size":709,"stargazers_count":84,"open_issues_count":9,"forks_count":12,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-19T07:27:54.262Z","etag":null,"topics":["eagle3","fsdp","lightseek","llm","mooncake","pytorch","sglang","vllm"],"latest_commit_sha":null,"homepage":"https://pytorch.org/blog/torchspec-speculative-decoding-training-at-scale","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lightseekorg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-21T04:18:35.000Z","updated_at":"2026-04-19T05:29:46.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lightseekorg/TorchSpec","commit_stats":null,"previous_names":["torchspec-project/torchspec","lightseekorg/torchspec"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lightseekorg/TorchSpec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightseekorg%2FTorchSpec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightseekorg%2FTorchSpec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightseekorg%2FTorchSpec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightseekorg%2FTorchSpec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lightseekorg","download_url":"https://codeload.github.com/lightseekorg/TorchSpec/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightseekorg%2FTorchSpec/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32203362,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-23T20:19:26.138Z","status":"ssl_error","status_checked_at":"2026-04-23T20:19:23.520Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["eagle3","fsdp","lightseek","llm","mooncake","pytorch","sglang","vllm"],"created_at":"2026-04-24T00:02:27.786Z","updated_at":"2026-04-24T00:02:28.505Z","avatar_url":"https://github.com/lightseekorg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TorchSpec\n\nTorchSpec is a torch-native speculative decoding training framework. We introduce a disaggregated way of training speculative decoding draft models where inference and training are fully decoupled and stream hidden states directly from inference engine groups to distributed training workers via [Mooncake](https://github.com/kvcache-ai/Mooncake) store, allowing each side to scale independently.\n\nTorchSpec currently includes training flows and examples for:\n\n- Kimi-K2.5\n- MiniMax-M2.5\n- Qwen3-Coder-Next\n\n## 🚀 Blogs\n\n- PyTorch blog: [TorchSpec: Speculative Decoding Training at Scale](https://pytorch.org/blog/torchspec-speculative-decoding-training-at-scale/)\n- Release blog: [TorchSpec: Speculative Decoding Training at Scale](https://lightseek.org/blog/torchspec-speculative-decoding-training-at-scale.html)\n- Released draft model: [lightseekorg/kimi-k2.5-eagle3](https://huggingface.co/lightseekorg/kimi-k2.5-eagle3)\n\n## Table of Contents\n\n- [Architecture Overview](#architecture-overview)\n- [Quick Start](#quick-start)\n- [Setup](#setup)\n- [Examples](#examples)\n- [Training Modes](#training-modes)\n- [Checkpoint Conversion](#checkpoint-conversion)\n- [Metrics Reporting](#metrics-reporting)\n- [Troubleshooting](#troubleshooting)\n\n## Architecture Overview\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/torchspec_architecture.png\" alt=\"TorchSpec Architecture\" width=\"100%\"\u003e\n\u003c/p\u003e\n\nTorchSpec is built around a disaggregated training pipeline:\n\n- **Inference engines** generate target-model hidden states with either vLLM or SGLang.\n- **Mooncake store** transfers tensors between inference and training without materializing them on disk.\n- **Training workers** consume streamed hidden states to train speculative decoding draft models.\n\nThis separation keeps the training side focused on optimization while letting the inference side scale for hidden-state generation throughput.\n\n## Quick Start\n\nTrain an Eagle3 draft model for Qwen3-8B on a single node with 4 GPUs (2 for training and 2 for inference):\n\n```bash\n./examples/qwen3-8b-single-node/run.sh\n```\n\nOverride config values directly from the CLI:\n\n```bash\n./examples/qwen3-8b-single-node/run.sh training.learning_rate=5e-5 training.num_train_steps=500\n```\n\n## Setup\n\n### Quick Setup\n\n```bash\n# Install with vLLM\n./tools/build_conda.sh 1 vllm\nmicromamba activate torchspec\n\n# Or install with SGLang\n./tools/build_conda.sh\nmicromamba activate torchspec\n```\n\nTo install into your current environment instead:\n\n```bash\n./tools/build_conda.sh current sglang  # or 'vllm' or 'both'\n```\n\nOptional: install Flash Attention support:\n\n```bash\npip install -e \".[fa]\"\n```\n\n### Backend-Specific Usage\n\n**vLLM**\n\n```bash\n./examples/qwen3-8b-single-node/run.sh --config configs/vllm_qwen3_8b.yaml\n```\n\n**SGLang**\n\n```bash\n./examples/qwen3-8b-single-node/run.sh\n```\n\nTorchSpec uses vLLM's **Worker Extension** mechanism to hook into the model forward pass and capture hidden states directly inside worker processes, which avoids RPC serialization overhead during extraction. For SGLang, TorchSpec applies a patch to the existing codebase to enable hidden-state extraction.\n\n## Examples\n\n| Example | Backend | Model |\n|---------|---------|-------|\n| [hf-quickstart](examples/hf-quickstart/) | HuggingFace | Qwen3-8B |\n| [qwen3-8b-single-node](examples/qwen3-8b-single-node/) | Inference engine | Qwen3-8B |\n| [kimi-k25-2node-h200](examples/kimi-k25-2node-h200/) | Inference engine | Kimi-K2.5 |\n| [kimi-k25-3node-h100](examples/kimi-k25-3node-h100/) | Inference engine | Kimi-K2.5 |\n| [minimax-m25-5node-h200](examples/minimax-m25-5node-h200/) | Inference engine | MiniMax-M2.5 |\n\nSee [examples/README.md](examples/README.md) for more details about each example.\n\n## Training Modes\n\n### Resume vs. Continual Training\n\nBoth modes use `training.load_path`, but they restore different states:\n\n| Goal | `training.load_path` | `training.continual_training` | What gets restored |\n|------|----------------------|-------------------------------|--------------------|\n| Resume an interrupted run | Required | `false` (default) | Model, optimizer, LR scheduler, RNG, and step metadata |\n| Start a new run from existing weights | Required | `true` | Model weights only |\n\nResume the same run:\n\n```yaml\ntraining:\n  load_path: /path/to/old_run/checkpoints\n\noutput_dir: /path/to/old_run\n```\n\nStart a new run from existing weights:\n\n```yaml\ntraining:\n  load_path: /path/to/old_run/checkpoints\n  continual_training: true\n  learning_rate: 1e-5\n  warmup_ratio: 0.01\n  num_epochs: 1\n\noutput_dir: /path/to/new_run\n```\n\n## Checkpoint Conversion\n\nConvert an FSDP checkpoint to HuggingFace format:\n\n```bash\npython tools/convert_to_hf.py --input-dir ./outputs/my_experiment/iter_0010000/\n```\n\nVocabulary pruning, which reduces the draft model `lm_head` to a smaller token set and emits `d2t` and `t2d` mappings, can be applied either during training or at conversion time.\n\n- **Pre-pruning**: set `draft_vocab_size` in your training config. The checkpoint already contains the pruned `lm_head` and `d2t`/`t2d` buffers, so the basic conversion command is enough.\n- **Post-pruning**: train with the full vocabulary, then pass `--prune-vocab` at conversion time together with a representative dataset to compute token frequencies.\n\n```bash\npython tools/convert_to_hf.py \\\n    --input-dir ./outputs/my_experiment/iter_0010000/ \\\n    --prune-vocab \\\n    --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \\\n    --draft-vocab-size 32000 \\\n    --tokenizer Qwen/Qwen3-8B \\\n    --chat-template qwen \\\n    --prompt-key conversations\n```\n\nPass `--cache-dir ./cache` to reuse the tokenized dataset cache from training.\n\n## Metrics Reporting\n\nW\u0026B logging is disabled by default with `report_to: none`. To enable it, set `report_to: wandb` in your config and provide your API key.\n\n## Troubleshooting\n\nSet `TORCHSPEC_LOG_LEVEL=DEBUG` for more verbose logging when diagnosing issues:\n\n```bash\nTORCHSPEC_LOG_LEVEL=DEBUG ./examples/qwen3-8b-single-node/run.sh\n```\n\n### Per-Rank File Logging\n\nSet `TORCHSPEC_LOG_DIR` to an absolute path on a shared filesystem (NFS) to enable per-rank log files for every Ray actor on both training and inference:\n\n```bash\nexport TORCHSPEC_LOG_DIR=/my_project/running_logs\n```\n\nThis creates a structured directory with one file per actor, organized by role and node:\n\n```text\nrunning_logs/\n  training/\n    10.0.0.1/\n      training_g0_rank0_20260301_080012.log\n      training_g0_rank1_20260301_080012.log\n    10.0.0.2/\n      training_g0_rank2_20260301_080013.log\n  inference/\n    10.0.0.1/\n      inference_g0_rank0_20260301_080014.log\n    10.0.0.2/\n      inference_g0_rank1_20260301_080015.log\n```\n\nThe path must be absolute and writable from all nodes. If `TORCHSPEC_LOG_DIR` is unset or not writable, per-rank file logging stays disabled and Ray falls back to stdout/stderr capture.\n\n| Issue | Reference |\n|-------|-----------|\n| Stuck or failing distributed runs, Ray actor errors | [docs/debugging_ray_jobs.md](docs/debugging_ray_jobs.md) |\n| Ray cluster setup, actor hierarchy, placement groups | [docs/ray.md](docs/ray.md) |\n| Pipeline bottlenecks, slow steps, throughput analysis | [docs/performance_metrics.md](docs/performance_metrics.md) |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightseekorg%2Ftorchspec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flightseekorg%2Ftorchspec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightseekorg%2Ftorchspec/lists"}