{"id":50297211,"url":"https://github.com/abdulvahapmutlu/mhc-slm","last_synced_at":"2026-05-28T09:34:53.301Z","repository":{"id":356488710,"uuid":"1232757695","full_name":"abdulvahapmutlu/mhc-slm","owner":"abdulvahapmutlu","description":"Official PyTorch implementation of mHC-SSM: manifold-constrained multi-stream residual mixing for State Space Language Models, extended with stream-specialized adapters for improved WikiText-2 perplexity.","archived":false,"fork":false,"pushed_at":"2026-05-08T10:35:18.000Z","size":17,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-08T10:37:33.855Z","etag":null,"topics":["adapters","deep-learning","hyper-connections","language-modeling","machine-learning","mhc","python","pytorch","sinkhorn","ssm","state-space-models"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abdulvahapmutlu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-08T08:36:41.000Z","updated_at":"2026-05-08T10:35:21.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/abdulvahapmutlu/mhc-slm","commit_stats":null,"previous_names":["abdulvahapmutlu/mhc-slm"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/abdulvahapmutlu/mhc-slm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdulvahapmutlu%2Fmhc-slm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdulvahapmutlu%2Fmhc-slm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdulvahapmutlu%2Fmhc-slm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdulvahapmutlu%2Fmhc-slm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abdulvahapmutlu","download_url":"https://codeload.github.com/abdulvahapmutlu/mhc-slm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdulvahapmutlu%2Fmhc-slm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33603475,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adapters","deep-learning","hyper-connections","language-modeling","machine-learning","mhc","python","pytorch","sinkhorn","ssm","state-space-models"],"created_at":"2026-05-28T09:34:48.696Z","updated_at":"2026-05-28T09:34:53.291Z","avatar_url":"https://github.com/abdulvahapmutlu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# mHC-SSM: Manifold-Constrained Hyper-Connections for State Space Language Models\n\nPyTorch research implementation of **mHC-SSM**, a State Space Language Model architecture that adapts **Manifold-Constrained Hyper-Connections (mHC)** to SSM-based language modeling and extends it with **Stream-Specialized Adapters**.\n\nThis project investigates whether stability-constrained multi-stream residual mixing can improve lightweight SSM language models, and whether adding compact stream-specific adapter capacity can further improve validation perplexity without modifying the SSM recurrence itself.\n\n---\n\n## Overview\n\nStandard residual SSM language models use a single residual stream:\n\n```\nx_{l+1} = x_l + SSM(x_l)\n````\n\nmHC-SSM expands the residual stream into multiple parallel streams:\n\n```\nX_l ∈ R^{B × T × n × d}\n```\n\nEach mHC-SSM block performs:\n\n1. Multi-stream residual representation\n2. Simplex-constrained pre-mixing into a single SSM input\n3. Lightweight diagonal SSM computation\n4. Simplex-constrained post-mixing back into streams\n5. Sinkhorn-projected residual stream mixing\n6. Optional stream-specialized adapters before and after the SSM core\n\nThe best-performing variant in this repository is:\n\n```\nmHC-SSM + Stream-Specialized Adapters\n```\n\n---\n\n## Main Contributions\n\nThis repository provides:\n\n* A baseline single-stream diagonal SSM language model.\n* A static mHC-SSM architecture with Sinkhorn-constrained multi-stream residual mixing.\n* A stream-specialized adapter extension using shared bottleneck projections and per-stream scaling.\n* Training and benchmarking scripts for WikiText-2.\n* Checkpoint-restored benchmarking to fairly measure perplexity after throughput timing.\n* Reproducible commands for baseline, static mHC-SSM, and mHC-SSM with adapters.\n\n---\n\n## Architecture\n\n```\ntokens\n  │\n  ▼\nEmbedding + Positional Encoding\n  │\n  ▼\nStream Expander\n  │\n  ▼\nStack of mHC-SSM Blocks\n  │\n  ├── optional stream-specialized pre-adapter\n  ├── simplex pre-mixing\n  ├── SSM core\n  ├── optional stream-specialized post-adapter\n  ├── simplex post-mixing\n  └── Sinkhorn-projected residual stream mixing\n  │\n  ▼\nStream Aggregator\n  │\n  ▼\nFinal Norm + Tied LM Head\n  │\n  ▼\nNext-token logits\n```\n\n---\n\n## Installation\n\nCreate a virtual environment:\n\n```\npython -m venv .venv\n```\n\nActivate it.\n\nWindows PowerShell:\n\n```\n.\\.venv\\Scripts\\Activate.ps1\n```\n\nLinux/macOS:\n\n```\nsource .venv/bin/activate\n```\n\nInstall dependencies:\n\n```\npip install -r requirements.txt\n```\n\n---\n\n## Requirements\n\n```\ntorch\u003e=2.1\ntransformers\u003e=4.41\ndatasets\u003e=2.19\ntqdm\u003e=4.66\neinops\u003e=0.8\nnumpy\u003e=1.26\n```\n\nCUDA is recommended. The experiments reported here were run with CUDA AMP enabled.\n\n---\n\n## Dataset\n\nThe experiments use **WikiText-2** through the Hugging Face `datasets` library.\n\nTokenization is performed with the GPT-2 tokenizer from Hugging Face Transformers.\n\nThe data loader packs the tokenized corpus into fixed-length autoregressive language modeling sequences:\n\n```\ninput  = tokens[t : t + seq_len]\ntarget = tokens[t + 1 : t + 1 + seq_len]\n```\n\n---\n\n## Training Setup\n\nAll reported models use the same core setup:\n\n| Setting             |           Value |\n| ------------------- | --------------: |\n| Dataset             |      WikiText-2 |\n| Tokenizer           | GPT-2 tokenizer |\n| Epochs              |              10 |\n| Sequence length     |             256 |\n| Batch size          |              16 |\n| Layers              |               8 |\n| Hidden dimension    |             512 |\n| Number of streams   |               4 |\n| Sinkhorn iterations |               8 |\n| Optimizer           |           AdamW |\n| Learning rate       |            3e-4 |\n| Weight decay        |             0.1 |\n| Gradient clipping   |             1.0 |\n| AMP                 | Enabled on CUDA |\n\n---\n\n## Training Commands\n\n### 1. Baseline SSM\n\n```\npython -m mhc_slm.train `\n  --model baseline `\n  --epochs 10 `\n  --seq_len 256 `\n  --batch_size 16 `\n  --layers 8 `\n  --dim 512 `\n  --amp\n```\n\n---\n\n### 2. Static mHC-SSM\n\n```\npython -m mhc_slm.train `\n  --model mhc `\n  --epochs 10 `\n  --seq_len 256 `\n  --batch_size 16 `\n  --layers 8 `\n  --dim 512 `\n  --n_streams 4 `\n  --sinkhorn_iters 8 `\n  --amp\n```\n\n---\n\n### 3. mHC-SSM + Stream-Specialized Adapters\n\n```\npython -m mhc_slm.train `\n  --model mhc `\n  --epochs 10 `\n  --seq_len 256 `\n  --batch_size 16 `\n  --layers 8 `\n  --dim 512 `\n  --n_streams 4 `\n  --sinkhorn_iters 8 `\n  --stream_adapters `\n  --adapter_rank 64 `\n  --adapter_dropout 0.1 `\n  --amp\n```\n\n---\n\n## Benchmarking\n\nThe benchmark script measures:\n\n* Training throughput in tokens/sec\n* Peak CUDA memory\n* Validation loss\n* Perplexity\n\nImportant: throughput timing uses synthetic optimizer steps, which normally modify model weights. The benchmark script saves and restores the checkpoint weights before perplexity evaluation, so the reported validation loss and perplexity are computed fairly from the original checkpoint.\n\n---\n\n### Baseline Benchmark\n\n```\npython -m mhc_slm.benchmark `\n  --model baseline `\n  --seq_len 256 `\n  --batch_size 8 `\n  --steps 50 `\n  --warmup 10 `\n  --amp `\n  --ppl_eval `\n  --ppl_batches 62 `\n  --ckpt \"runs\\mhc_slm\\baseline_...\\baseline_final.pt\"\n```\n\n---\n\n### Static mHC-SSM Benchmark\n\n```\npython -m mhc_slm.benchmark `\n  --model mhc `\n  --n_streams 4 `\n  --sinkhorn_iters 8 `\n  --seq_len 256 `\n  --batch_size 8 `\n  --steps 50 `\n  --warmup 10 `\n  --amp `\n  --ppl_eval `\n  --ppl_batches 62 `\n  --ckpt \"runs\\mhc_slm\\mhc_static_...\\mhc_final.pt\"\n```\n\n---\n\n### mHC-SSM + Stream Adapters Benchmark\n\n```\npython -m mhc_slm.benchmark `\n  --model mhc `\n  --n_streams 4 `\n  --sinkhorn_iters 8 `\n  --seq_len 256 `\n  --batch_size 8 `\n  --steps 50 `\n  --warmup 10 `\n  --amp `\n  --ppl_eval `\n  --ppl_batches 62 `\n  --stream_adapters `\n  --adapter_rank 64 `\n  --ckpt \"runs\\mhc_slm\\mhc_static_..._ad64_...\\mhc_final.pt\"\n```\n\n---\n\n## Results\n\n### Final Checkpoint Benchmark\n\n| Model                     | Validation Loss | Perplexity | Tokens/sec | Peak GPU Memory |\n| ------------------------- | --------------: | ---------: | ---------: | --------------: |\n| Baseline SSM              |          6.3507 |     572.91 |    1025.52 |         2365 MB |\n| Static mHC-SSM            |          6.2448 |     515.35 |     964.81 |         2568 MB |\n| mHC-SSM + Stream Adapters |      **6.1353** | **461.88** |     938.90 |         3092 MB |\n\n---\n\n## Result Interpretation\n\nStatic mHC-SSM improves over the baseline by introducing constrained multi-stream residual mixing.\n\nCompared with the baseline:\n\n```\nValidation loss: 6.3507 → 6.2448\nPerplexity:      572.91 → 515.35\n```\n\nAdding stream-specialized adapters improves the model further:\n\n```\nValidation loss: 6.2448 → 6.1353\nPerplexity:      515.35 → 461.88\n```\n\nOverall, the adapter-augmented model improves perplexity from **572.91** to **461.88**, while reducing throughput from **1025.52** to **938.90 tokens/sec** and increasing peak memory from **2365 MB** to **3092 MB**.\n\nThis suggests that stream-specialized adapters provide a useful extension point for mHC-style residual topologies in SSM language models.\n\n---\n\n## License\n\nThis project is released under the MIT License.\n\n---\n\n## Acknowledgements\n\nThis project builds on ideas from residual learning, state space sequence models, Hyper-Connections, Manifold-Constrained Hyper-Connections, Sinkhorn normalization, and adapter-based parameter-efficient modeling.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabdulvahapmutlu%2Fmhc-slm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabdulvahapmutlu%2Fmhc-slm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabdulvahapmutlu%2Fmhc-slm/lists"}