{"id":50331728,"url":"https://github.com/tsugiai/tsugi-mend","last_synced_at":"2026-05-29T10:00:46.818Z","repository":{"id":360659333,"uuid":"1251112087","full_name":"tsugiai/tsugi-mend","owner":"tsugiai","description":"Cross-rack distributed-training reducer for PyTorch. Apache-2.0, patent-independent. Part of the unified pip install tsugi surface.","archived":false,"fork":false,"pushed_at":"2026-05-27T10:54:21.000Z","size":86,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-27T11:23:45.099Z","etag":null,"topics":["cross-rack","diloco","distributed-training","fault-tolerance","fsdp","gpu","llm-training","pytorch"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tsugiai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-27T08:54:41.000Z","updated_at":"2026-05-27T10:55:33.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tsugiai/tsugi-mend","commit_stats":null,"previous_names":["tsugiai/tsugi-mend"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/tsugiai/tsugi-mend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tsugiai%2Ftsugi-mend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tsugiai%2Ftsugi-mend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tsugiai%2Ftsugi-mend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tsugiai%2Ftsugi-mend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tsugiai","download_url":"https://codeload.github.com/tsugiai/tsugi-mend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tsugiai%2Ftsugi-mend/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33646428,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cross-rack","diloco","distributed-training","fault-tolerance","fsdp","gpu","llm-training","pytorch"],"created_at":"2026-05-29T10:00:29.796Z","updated_at":"2026-05-29T10:00:46.812Z","avatar_url":"https://github.com/tsugiai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tsugi-mend\n\n[![PyPI version](https://img.shields.io/pypi/v/tsugi-mend.svg)](https://pypi.org/project/tsugi-mend/)\n[![Python versions](https://img.shields.io/pypi/pyversions/tsugi-mend.svg)](https://pypi.org/project/tsugi-mend/)\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)\n[![CI](https://github.com/tsugiai/tsugi-mend/actions/workflows/ci.yml/badge.svg)](https://github.com/tsugiai/tsugi-mend/actions/workflows/ci.yml)\n\nCross-rack reducer toolkit for PyTorch training loops.\n\n`tsugi-mend` is a software-only component toolkit for wiring\nDecoupled-DiLoCo-style periodic merges and concurrent outer-step overlap into\na training loop. It is not a transparent 0.1.x drop-in that intercepts DDP or\nFSDP collectives by itself. The caller drives the integration points at\nouter-step boundaries, supplies parameter-delta fragments, collects merged\ndeltas, and applies them at the same lag as the synchronous-reducer path. The\nexamples and benchmark driver are the worked integrations.\n\nPublic-art references and 0.1.x implementation status:\n\n- **Decoupled DiLoCo for Resilient Distributed Pre-training** (Arthur Douillard et al., arXiv:2604.21428, April 2026): the reducer implements minimum quorum, adaptive grace window, and token-weighted merge.\n- **Concurrent outer-step overlap**: the `ConcurrentOuterStep` orchestrator is wired when `concurrent_outer_step=True`, so the training thread can overlap the grace-window wait with inner-step compute.\n- **DES-LOC / Local Adam** (Iacob et al., arXiv:2505.22549, May 2025; ICLR 2026): desynchronized synchronization-period components are present, but moment synchronization is not automatically wired into `mend_init` in 0.1.x.\n- **Async tensor parallelism** (PyTorch / TorchTitan, September 2024): treated as an integration component/configuration point, not automatically installed by `mend_init` in 0.1.x.\n- **FALCON fail-slow detection** (arXiv:2410.12588, October 2024): the runtime observes step times and can emit detection diagnostics; FALCON-style quorum exclusion/mitigation is not wired in 0.1.x.\n- **Gradient compression** (`none`, `int8`, `powersgd`): primitives and config validation are present; the default path is lossless `none`, and compression is not invoked by the 0.1.x runtime outer-step path.\n\nThe SDK keeps intra-rack TP / CP / PP / FSDP collectives unchanged. In 0.1.x,\nthe public runtime exercises the reducer plus concurrent outer-step overlap;\nthe other mechanisms above are components or integration points.\n\n## Install\n\n```bash\npip install tsugi-mend\n```\n\nOr install the unified surface that bundles this SDK with the companion patent-aligned SDK:\n\n```bash\npip install tsugi   # exposes tsugi.mend and tsugi.kpool\n```\n\nFor local development:\n\n```bash\npip install -e \".[dev]\"\n```\n\n## License and IP posture\n\nThis SDK is licensed under **Apache-2.0** with its full automatic patent grant. The SDK is patent-independent by deliberate construction: it does not exercise the K-Pool LoRA (US App. 64/060,315) or Infinity (US App. 64/055,093) patent estates that the companion SDK (`tsugi-kpool`, also Apache-2.0) does. Read the preamble at the top of the `LICENSE` file for the full posture explanation.\n\nThe companion patent-aligned SDK at [`github.com/tsugiai/tsugi-kpool`](https://github.com/tsugiai/tsugi-kpool) is the software embodiment of those two TsugiCinema patent estates. The two SDKs share zero code.\n\n## Measurements\n\nThe numbers below are first-party internal benchmark measurements taken\nunder the reproduction contract in [`docs/benchmark_protocol.md`](docs/benchmark_protocol.md)\n(same workload / checkpoint / hardware, baseline vs SDK, paired runs,\nbootstrap 95% CI). The raw per-run results logs are internal; the\nprotocol document is the public reproduction pointer, and the headline\ncells below can be re-derived by anyone who runs the protocol on the\nstated hardware. We report point estimates with their 95% CI where\navailable and flag single-seed (n=1) cells explicitly.\n\n### Production-grounded results\n\nThe robust headline is **bit-exact loss equivalence in default mode**. It is\npreserved across every paired run, every seed, every fabric condition we have\nmeasured.\nThroughput uplift on real cross-network is **jitter-conditional**: the SDK's\noverlap mechanisms hide cross-rack latency when it exists, so the magnitude of\nthe uplift depends on the fabric jitter present at measurement time.\n\n| Workload | Hardware | Measurement |\n|---|---|---|\n| **Real cross-network 2-node 8xV100 (synchronous reducer)** | Lambda Labs commodity Ethernet, SmolLM-135M, 500 paired steps × 7 seeds | **Bit-exact loss PASS on every seed** (max\\|loss diff\\| = 0.0); uplift **mean +3.4%, CI95 [-5%, +12%]**, per-seed range **[-10%, +15%]** (n=7). Details: [`benchmarks/results/real_8xv100_2node/`](benchmarks/results/real_8xv100_2node/) |\n| Production-realistic multi-GPU FSDP + 7B model (realistic floor, 3-seed CI) | Modal 8xH100 FSDP FULL_SHARD, Qwen-2.5-7B + simulated 2-rack, 4 delays × 3 seeds | +6.37% ± 1.31% at 2000ms (n=3) |\n| H100 Hopper single-instance (synchronous reducer baseline) | Modal 8x H100 SXM5, Llama-3-8B, 2000 paired steps × 3 seeds | -0.97% ± 1.5% (predicted null; Hopper NVLink absorbs the synchronous-path cross-rack tax) |\n\nHow to read the production-grounded numbers honestly:\n\n- **Bit-exact loss equivalence is the load-bearing result.** Every cross-network\n  paired run preserves loss to IEEE-754 equality vs the synchronous-reducer\n  baseline: both paths apply the same Decoupled-DiLoCo-style merged delta at\n  the same lag, and the concurrent path only moves the merge wait off the\n  training thread. This is not a claim that either path is numerically equal to\n  a vanilla DDP/FSDP all-reduce run.\n- **Throughput uplift on real cross-network is jitter-conditional, not a fixed\n  magnitude.** On the 2-node 8xV100 commodity-Ethernet cell, `n=7` re-measurement\n  under [`docs/benchmark_protocol.md`](docs/benchmark_protocol.md) shows mean\n  **+3.4%** with CI95 **[-5%, +12%]** and a per-seed range of **[-10%, +15%]**.\n  Baseline tok/s itself varies ~18% seed-to-seed (1473-1735), and that\n  fabric-side variance dominates the SDK signal over the ~3-4 outer rounds a\n  500-step run at sync_period 128 contains. A prior single-run measurement on\n  the same setup produced **+28.58%** during a higher-jitter Lambda Ethernet\n  session; that point estimate sits in the high tail of the measured envelope\n  and is **not representative of the mean** under n\u003e=3 protocol. Report any\n  cross-network uplift number with a range or CI, per the protocol's \"never a\n  bare point estimate\" rule.\n- Production-realistic multi-GPU FSDP yields a smaller honest floor (**+6.37%\n  ± 1.31%, n=3**) at injected 2000ms delay because 8-rank NCCL pipelining\n  absorbs some of the simulated delay.\n- **Protocol-incomplete single-seed note.** The real-fabric Hopper 2-pod\n  InfiniBand / RoCE result is not comparable to the n\u003e=3 rows above yet:\n  RunPod 2x 8x H100 SXM5 over real InfiniBand / RoCE v2 3.2 Tbps, Llama-3-8B,\n  500 paired steps × 1 seed, measured +1.42% tps with +0.18% loss delta. The\n  **n=1 caveat is load-bearing** because the point estimate is the same order\n  of magnitude as baseline-only seed variance; n=3 CI is pending.\n\n### Ceiling-case / simulated-delay results\n\nEvery cell in this subsection uses an injected simulated grace-window\ndelay on a single instance or simulated two-rack setup, not a real\ncross-network measurement. These are ceiling-case stress tests for the\noverlap mechanism rather than production numbers.\n\n| Workload | Hardware | Measurement |\n|---|---|---|\n| Statistical-confidence ceiling case (Hopper 3-seed CI) | Modal H100:1, Qwen-2.5-1.5B, 200 steps × 3 seeds at 2000ms grace window | **+71.49% ± 2.83% (95% CI, n=3)** throughput uplift |\n| Cross-rack grace-window overlap on Hopper at 7B scale | Modal H100:1, Qwen-2.5-7B, 200 steps × 5 delays | +76.58% at 2000ms; +39.72% at 1000ms; +19.20% at 500ms |\n| Intermediate model-scale (3B) confirmation (Hopper 3-seed CI) | Modal H100:1, Qwen-2.5-3B, 200 steps × 4 delays × 3 seeds | +41.31% ± 0.29% at 2000ms (n=3); +20.40% ± 0.03% at 1000ms; +9.75% ± 0.04% at 500ms |\n| Cross-rack grace-window overlap at 1.5B scale | Modal H100:1, Qwen-2.5-1.5B, 200 steps × 7 delays | +70.64% at 2000ms; +34.73% at 1000ms; +16.86% at 500ms |\n| Cross-rack grace-window overlap on A10G | Modal A10G, SmolLM-135M, 200 steps × 7 delays | +52.75% at 2000ms (constant); +11.61% at 500ms; -0.06% overhead at 0ms; bit-exact loss preserved across all cells |\n\nHow to read the ceiling-case numbers honestly:\n\n- The **+71.49% ± 2.83% (n=3) Hopper result** is a single-instance measurement with an injected simulated grace-window delay, not a real cross-network result. Read it as a ceiling-case for the overlap mechanism.\n- The orchestrator's uplift is governed by `N · T_step / G` (sync-period steps × per-step compute time vs grace-window ms). Apparent non-monotonicity with model size (Qwen-3B +41.31% below both Qwen-1.5B and Qwen-7B) is explained by the Qwen-7B measurement using 1/8 the tokens-per-step (seq_len 1024 / mbs 1 vs 2048 / 4); at fixed tokens-per-step, uplift is monotonically decreasing in model size.\n- Constant-delay headlines (e.g. +52.75% A10G at 2000ms) are ceiling-case stress tests. The FALCON paper documents cross-rack inter-node RDMA variance (CoV=0.29) but does not characterize the per-iteration latency distribution shape; the delay sweep is a stress test, not a literal FALCON replay.\n\nAt every scale the concurrent path's throughput is rock-solid across delays (Qwen-7B single-process: 4,300 ± 80 tok/s; Qwen-3B Hopper: 18,153 ± 4 tok/s; Qwen-1.5B Hopper: 30,840 ± 35 tok/s; SmolLM-135M A10G: 23,610 ± 80 tok/s) while the synchronous baseline collapses linearly with delay.\n\n### Run it multi-node\n\nSee [`docs/multinode.md`](docs/multinode.md) for the multi-node launch walkthrough.\n\n## Status\n\n**Pre-Alpha (0.1.1).** APIs are stabilizing and may change before v1.0. Published to PyPI as `tsugi-mend`; also reachable through the unified `tsugi` meta-package as `tsugi.mend`. The staged validation (Stage A unit/integration through cross-network production-fabric runs) all passed under the protocol above; the real-fabric Hopper cross-network result is point-estimate closed (n=1), with an n=3 CI pending.\n\n## Quickstart\n\n```python\nfrom tsugi_mend import MendConfig, mend_init, mend_shutdown\nfrom tsugi_mend.runtime import get_runtime\n\nconfig = MendConfig(\n    quorum_min_learners=4,\n    grace_window_ms=2000,\n    token_weighted_merge=True,\n    sync_period_steps=128,\n    # Orchestrator overlaps the cross-rack outer-step wait with inner-step\n    # compute. Default True.\n    concurrent_outer_step=True,\n    diagnostics_dir=\"./results/mend_diag\",\n)\n\nmend_init(model, config)\nruntime = get_runtime(model)\n\nfor step, batch in enumerate(loader):\n    runtime.step_begin(step)\n    loss = train_one_step(model, optimizer, batch)\n\n    sched = runtime.schedule_for(step)\n    if sched.should_sync_params and not runtime.outer_step_in_flight():\n        runtime.outer_step_begin(\n            round_id=step,\n            fragment_provider=make_fragment_provider(...),\n        )\n\n    result = runtime.outer_step_collect()\n    if result is not None:\n        apply_merged_delta(model, result.merged_delta)\n\n    runtime.step_end(step)\n\nmend_shutdown(model)\n```\n\nThe snippet above is the integration shape, not a complete program. Use the\nexamples below for runnable wiring, and the benchmark driver for the fuller\nfragment gather and merge-application path.\n\nTwo runnable, CPU-only integration examples (no GPU or multi-node required):\n\n- [`examples/minimal_single_process.py`](examples/minimal_single_process.py) - smallest end-to-end use on a toy `nn.Module`.\n- [`examples/concurrent_orchestrator.py`](examples/concurrent_orchestrator.py) - wiring the `ConcurrentOuterStep` orchestrator into a training loop with a synthetic single-rank fragment provider.\n\n```bash\npython examples/minimal_single_process.py\npython examples/concurrent_orchestrator.py\n```\n\n## Layout\n\n```\nsrc/tsugi_mend/   SDK source\ntests/            unit and integration tests (CPU-only)\ndocs/             architecture, benchmark protocol, convergence-equivalence sketch\nexamples/         minimal CPU-only training-loop integration examples\n```\n\n## Companion SDK\n\nFor LoRA-adapter-granularity productization that exercises the K-Pool LoRA and Infinity patent estates, see [`tsugi-kpool`](https://github.com/tsugiai/tsugi-kpool). The two SDKs share zero code and can be installed and used independently, or together via the unified [`tsugi`](https://github.com/tsugiai/tsugi) meta-package.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftsugiai%2Ftsugi-mend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftsugiai%2Ftsugi-mend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftsugiai%2Ftsugi-mend/lists"}