{"id":51321835,"url":"https://github.com/zeekmartin/sal-torch","last_synced_at":"2026-07-02T15:01:15.399Z","repository":{"id":367697702,"uuid":"1280460829","full_name":"zeekmartin/sal-torch","owner":"zeekmartin","description":"Structurally Adaptive Learning - training-time sparsification for PyTorch","archived":false,"fork":false,"pushed_at":"2026-06-29T20:32:07.000Z","size":231,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-07-01T14:34:31.591Z","etag":null,"topics":["machine-learning","model-compression","pruning","pytorch","sparsification","transformers"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/sal-torch/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zeekmartin.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-25T15:53:42.000Z","updated_at":"2026-06-29T20:32:12.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/zeekmartin/sal-torch","commit_stats":null,"previous_names":["zeekmartin/sal-torch"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/zeekmartin/sal-torch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeekmartin%2Fsal-torch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeekmartin%2Fsal-torch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeekmartin%2Fsal-torch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeekmartin%2Fsal-torch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zeekmartin","download_url":"https://codeload.github.com/zeekmartin/sal-torch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeekmartin%2Fsal-torch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35051884,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","model-compression","pruning","pytorch","sparsification","transformers"],"created_at":"2026-07-01T14:30:27.533Z","updated_at":"2026-07-02T15:01:15.348Z","avatar_url":"https://github.com/zeekmartin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sal-torch\n\n![CI](https://github.com/zeekmartin/sal-torch/actions/workflows/ci.yml/badge.svg) ![PyPI](https://img.shields.io/pypi/v/sal-torch) ![Python](https://img.shields.io/pypi/pyversions/sal-torch) ![Downloads](https://pepy.tech/badge/sal-torch) ![License](https://img.shields.io/badge/license-BSL%201.1-blue)\n\n\n**Structurally Adaptive Learning for PyTorch**\n\nTraining-time sparsification that makes neural networks structurally resilient to compression.\n\n## Install\n\n```bash\npip install sal-torch            # core\npip install sal-torch[hf]        # + HuggingFace Trainer\npip install sal-torch[all]       # everything\n```\n\n```python\nfrom sal import SALConfig, SALCallback\n\nconfig = SALConfig.auto(model)\ntrainer = Trainer(model=model, callbacks=[SALCallback(config)])\ntrainer.train()\n```\n\nThree lines. Any transformer. Compression-resilient.\n\n## Know your model before you touch it\n\n### PlasticityScanner — where can a model absorb compression?\n\nFI tells you how fragile a model *is*. `PlasticityScanner` tells you how much\nroom it has to *reorganize*, so you know where it is safe to compress. It scores\nthree complementary axes per layer — routing flexibility (attention entropy),\ninter-layer redundancy (linear CKA), and intra-layer redundancy (an MI proxy) —\nand folds them into an **absorption map** that labels each layer `ELASTIC`\n(safe), `SATURATED` (bottleneck), or `HUB` (compensates when others are pruned).\n\n```python\nfrom sal import PlasticityScanner\n\npmap = PlasticityScanner(model, probe_dataset).scan()\nprint(pmap.summary)              # \"3 elastic, 1 saturated, 2 hub | mean routing=0.61 ...\"\n\nrec = pmap.recommend(target_compression=0.33)\nrec.safe_to_prune                # [(layer, head), ...] — prune these first\nrec.never_touch                  # heads in hub layers — leave alone\nrec.expected_impact              # heuristic accuracy delta\n\npmap.save(\"plasticity.json\")     # raw scores\npmap.save(\"plasticity.pdf\")      # visual report (needs sal-torch[reports])\n```\n\n### sal.compare() — SAL vs. other pruning methods\n\nBenchmark SAL against post-hoc baselines at a matched compression level and see\nwhich keeps the most accuracy (or lowest loss) after heads are removed.\n\n```python\nfrom sal import compare\n\nresult = compare(model, train_dataset, eval_dataset,\n                 methods=[\"sal\", \"magnitude\", \"random_posthoc\"],\n                 compression=0.33, sal_epochs=3, metric=\"accuracy\")\nprint(result.table)              # method | score | pruned_heads | time\nprint(result.winner)\nresult.save(\"comparison.pdf\")    # bar chart + table\n\n# plug in your own method\ncompare.register_method(\"my_pruner\", lambda model, ds, eval_ds, ctx: my_score)\n```\n\n## Continual learning without replay buffers\n\n### StructuralGuard — protect what matters when you fine-tune\n\nWhen you fine-tune a trained model on a new task, it quietly overwrites the\nstructure that carried the old one. `StructuralGuard` reads the model's\nstructural map and **freezes the critical attention heads** (hub layers,\nstructural bottlenecks, and the functionally unique heads) while leaving the\nredundant heads free to absorb the new task. No EWC, no replay buffer, no\ndistillation — the topology itself decides what to protect.\n\n```python\nfrom sal import StructuralGuard\n\n# After training on task A, build a guard from the model's structure.\nguard = StructuralGuard.from_model(model, probe_dataset, protection_level=0.5)\n\nprint(guard.protected_heads)   # [(layer, head), ...] frozen during fine-tuning\nprint(guard.trainable_heads)   # [(layer, head), ...] free to absorb task B\nprint(guard.protection_map)    # {layer: [protected head indices]}\n\nguard.protect(model)           # zero gradients for protected heads (backward hooks)\ntrainer.train()                # fine-tune on task B with ANY training loop\nguard.release(model)\n\ndrift = guard.measure_drift(model, probe_dataset=probe_dataset)\nprint(drift.forgetting_score)      # 0 = nothing forgot, 1 = total reorganization\nprint(drift.protected_integrity)   # ~1.0 if the protected heads held\n\nguard.save(\"model_guard.json\")     # serialize; reload before task C, D, ...\nguard = StructuralGuard.load(\"model_guard.json\")\n```\n\nProtection is at the **head level** — some heads in a layer can be frozen while\nothers in the same layer keep learning. `protection_level` (0.0–1.0) sets the\nfraction of the most critical heads to protect.\n\nHuggingFace `Trainer`? Use the callback — it applies protection on\n`train_begin`, measures drift on `train_end`:\n\n```python\nfrom sal import StructuralGuardCallback\n\nguard = StructuralGuard.from_model(model, probe_dataset)\ncallback = StructuralGuardCallback(guard)\ntrainer = Trainer(model=model, callbacks=[callback])\ntrainer.train()\nprint(callback.drift_report.summary)\n```\n\n### DriftMonitor — measure structural forgetting after any fine-tuning\n\n`DriftMonitor` quantifies how much a model's structure moved, guarded or not.\nSnapshot before and after, then compare.\n\n```python\nfrom sal import DriftMonitor\n\nmonitor = DriftMonitor(model, probe_dataset)\nmonitor.snapshot(\"before_task_b\")\ntrainer.train()\nmonitor.snapshot(\"after_task_b\")\n\ndrift = monitor.compare(\"before_task_b\", \"after_task_b\")\nprint(drift.summary)\nprint(drift.layer_drift)             # per-layer activation retention (1 = identical)\nprint(drift.classification_changes)  # layers whose fragility class flipped\ndrift.save(\"drift_report.json\")\ndrift.save(\"drift_report.pdf\")       # visual before/after comparison\n```\n\nSnapshots are keyed, so you can track drift across many sequential tasks and\ncompare any pair.\n\n## Examples\n\n- [`examples/quickstart.py`](examples/quickstart.py) — 3-line SAL training on DistilBERT\n- [`examples/standalone_fi.py`](examples/standalone_fi.py) — Fragility Index scan, no training\n- [`examples/full_control.py`](examples/full_control.py) — manual config + standalone trainer\n- [`examples/compare_with_without_sal.py`](examples/compare_with_without_sal.py) — SAL vs. baseline under compression\n\nNew here? Start with [docs/getting_started.md](docs/getting_started.md).\n\n## License\n\nBSL 1.1 — free for research and evaluation. Commercial production requires a license.\n\nBuilt by [Cognitive Engineering](https://cognitive-engineering.dev) in Switzerland.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzeekmartin%2Fsal-torch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzeekmartin%2Fsal-torch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzeekmartin%2Fsal-torch/lists"}