{"id":49416207,"url":"https://github.com/chansigit/scgpt-modern","last_synced_at":"2026-04-29T03:07:10.156Z","repository":{"id":350562608,"uuid":"1207386053","full_name":"chansigit/scGPT-modern","owner":"chansigit","description":"A drop-in modernization of bowang-lab/scGPT for Python 3.12 + torch 2.6 + flash-attn 3 (H100 sm_90a native). Original pretrained weights load unmodified — compatible, more modern, faster.","archived":false,"fork":false,"pushed_at":"2026-04-10T22:44:28.000Z","size":35318,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-11T00:22:24.897Z","etag":null,"topics":["bioinformatics","deep-learning","flash-attention","foundation-model","genomics","h100","hopper","llm","pytorch","rna-seq","scgpt","single-cell","single-cell-genomics","transcriptomics"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chansigit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-10T22:12:15.000Z","updated_at":"2026-04-10T22:44:32.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/chansigit/scGPT-modern","commit_stats":null,"previous_names":["chansigit/scgpt-modern"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/chansigit/scGPT-modern","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chansigit%2FscGPT-modern","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chansigit%2FscGPT-modern/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chansigit%2FscGPT-modern/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chansigit%2FscGPT-modern/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chansigit","download_url":"https://codeload.github.com/chansigit/scGPT-modern/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chansigit%2FscGPT-modern/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32408467,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T02:37:21.628Z","status":"ssl_error","status_checked_at":"2026-04-29T02:36:50.947Z","response_time":110,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bioinformatics","deep-learning","flash-attention","foundation-model","genomics","h100","hopper","llm","pytorch","rna-seq","scgpt","single-cell","single-cell-genomics","transcriptomics"],"created_at":"2026-04-29T03:07:06.738Z","updated_at":"2026-04-29T03:07:10.149Z","avatar_url":"https://github.com/chansigit.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# scGPT-modern\n\n\u003e A **drop-in modernization** of [bowang-lab/scGPT](https://github.com/bowang-lab/scGPT)\n\u003e for **Python 3.12 + torch 2.6 + flash-attn 3** (H100 `sm_90a` native).\n\u003e **The original pretrained checkpoints load unmodified** — this is purely a\n\u003e runtime / dependency upgrade, not a new training run.\n\n## What's different from upstream\n\n| | upstream `bowang-lab/scGPT` | this repo `chansigit/scGPT-modern` |\n|---|---|---|\n| Python | 3.7 – 3.10 | **3.12** |\n| torch | 1.13.x + cu117 | **2.6.0 + cu124** |\n| numpy | locked `\u003c 2` | **≥ 2** (free) |\n| flash-attn | v1.0.4 (patched for sm_86 PTX) | **v3.0.0 (hopper)** native `sm_90a`, auto-falls back to v2 |\n| torchtext dependency | required (0.14 for torch 1.13) | **removed** (pure-Python shim) |\n| Target GPU | A100 / A40 / L40S | **H100** (primary), still runs on older arch via the v2 fallback |\n| scGPT source changes | — | **4 import-line edits** (see [`scgpt/_compat/`](scgpt/_compat/)) |\n\n## Why this exists\n\nThe upstream code has a hard `from flash_attn.flash_attention import FlashMHA`\nimport pinned to the legacy flash-attn 1.x API, and an `import torchtext.vocab`\nthat breaks under torch 2.5+ (ABI mismatch — `libtorchtext.so` fails to load\nwith undefined-symbol errors).\n\nRather than fork the model code, this repo adds a thin `scgpt/_compat/` shim\npackage that:\n\n1. Reproduces the old `FlashMHA` class with **the exact same submodule names**\n   (`Wqkv`, `out_proj`) so pretrained `state_dict`s load with **zero key\n   remapping**.\n2. Provides a ~190-line pure-Python replacement for `torchtext.vocab.Vocab` —\n   scGPT only touches 5 methods of it, no reason to keep the C++ dependency.\n\nThe shim auto-selects the best available attention backend at runtime:\n\n```\nflash-attn-4 (CuTeDSL, when NVIDIA ships working packaging)\n  ↓ fallback\nflash-attn-3 (hopper, sm_90a native — source-built from Dao-AILab/flash-attention hopper/)\n  ↓ fallback\nflash-attn-2 (pre-built wheel, covers Ampere / Ada / Hopper)\n  ↓ fallback\npure PyTorch SDPA (if none of the above import — via the upstream `flash_attn_available` gate)\n```\n\nFA3 removed the `dropout_p` kwarg, so the shim wraps it in a tiny adapter.\nfp32 inputs are auto-cast to fp16 inside `FlashMHA.forward()` and cast back\nbefore `out_proj` — this mirrors flash-attn v1's internal behavior, which the\n`scgpt.tasks.embed_data` pipeline relies on (it leaves the model unpromoted).\n\n## Verified\n\nPBMC3k zero-shot cell embedding with the **unmodified** `scGPT_continual_pretrained`\ncheckpoint (208 MB) produces results **numerically near-identical** to the\nlegacy flash-attn-1.0.4 run on L40S:\n\n| | this fork (FA3 on H100) | upstream env (FA1 on L40S) |\n|---|---|---|\n| `load_state_dict` missing keys on attention layers | **0** | 0 |\n| `load_state_dict` unexpected keys on attention layers | **0** | 0 |\n| embedding shape | (2638, 512) | (2638, 512) |\n| embedding `std` | 0.0442 | 0.0442 |\n| embedding `mean` | 0.0007 | 0.0007 |\n| per-cell cosine similarity vs legacy run | **0.9958** | — |\n| relative L2 error vs legacy run | **0.092** | — |\n\nThe tiny numerical drift comes from sm_86 PTX-JIT vs sm_90a WGMMA fp16\naccumulation order — it's well below the single-run Leiden clustering noise\nfloor (ARI ± 0.05 across random seeds) documented in the upstream zero-shot\ntutorials.\n\n## Quick start\n\n\u003e See [`docs/scGPT-modern-env-tutorial.md`](docs/scGPT-modern-env-tutorial.md)\n\u003e for the full setup guide: prerequisites, cluster pollution pitfalls, FA3\n\u003e build from source with trimmed scope, the compat shim architecture deep-dive,\n\u003e MC-TTA for stable benchmarks, per-GPU backend matrix, and a troubleshooting\n\u003e section for the failure modes we actually hit.\n\n```bash\n# 1. Create a Python 3.12 env (micromamba / conda / venv — your pick)\nmicromamba create -p /path/to/env -c conda-forge -y python=3.12 pip\n\n# 2. Install torch 2.6 + cu124 from pytorch.org\npip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124\n\n# 3. Install flash-attn 2 as a universal fallback (cp312 wheel from GitHub release)\npip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whl \\\n    --no-build-isolation\n\n# 4. (Recommended on H100) build flash-attn 3 from source — ~5 min with trimmed\n#    scope (sm_90a only; disable fp8 / hdim 96 / 192 / 256 / paged / local / cluster).\n#    See https://github.com/Dao-AILab/flash-attention/tree/main/hopper\n#\n#    The resulting wheel is tagged cp39-abi3 and is forward-compatible to any\n#    CPython \u003e= 3.9, so you only ever build it once even across Python upgrades.\n\n# 5. Scientific stack (all binary wheels)\npip install --only-binary=:all: \\\n    numpy scanpy anndata pandas scipy scikit-learn numba \\\n    einops ipython leidenalg python-igraph h5py pyarrow\n\n# 6. scGPT-modern itself (editable)\npip install poetry-core\ngit clone https://github.com/chansigit/scGPT-modern.git\ncd scGPT-modern\npip install -e . --no-deps --no-build-isolation\n\n# 7. Verify the backend resolver picks the fastest available kernel\npython -c \"\nfrom scgpt._compat.flash_attention import get_backend_name\nprint('FlashMHA backend:', get_backend_name())\n# expected on H100 with FA3 installed:      v3-hopper\n# expected on H100 with only FA2 installed: v2\n# expected with nothing importable:         none\n\"\n```\n\n## Usage\n\n**Identical to upstream.** The compat shim is completely transparent once the 4\nimport edits are in place. Every existing scGPT workflow — `embed_data`,\n`get_batch_cell_embeddings`, `TransformerModel`, `MultiomicModel`, the\nfine-tuning notebooks, the zero-shot tutorials — runs unchanged.\n\n```python\nfrom scgpt.tasks.cell_emb import embed_data\nimport scanpy as sc\n\nadata = sc.read_h5ad(\"path/to/raw_counts.h5ad\")\nout = embed_data(\n    adata,\n    model_dir=\"path/to/scGPT_continual_pretrained\",\n    gene_col=\"gene_name\",\n    use_fast_transformer=True,   # this triggers the shim backend resolver\n    return_new_adata=True,\n)\n# out.X is (n_cells, 512), L2-normalized, ready for sc.pp.neighbors(metric=\"cosine\")\n```\n\n## Relationship to upstream\n\nThis repo is a **standalone upgrade**, not a GitHub fork. The commit history\nbelow the two modernization commits is identical to upstream `bowang-lab/scGPT`\nat the branch point; an `upstream` git remote can be added locally for pulling\nfuture upstream changes:\n\n```bash\ngit remote add upstream https://github.com/bowang-lab/scGPT.git\ngit fetch upstream \u0026\u0026 git merge upstream/main\n```\n\n**Where to file issues:**\n- Questions about the scGPT model itself, training, checkpoints, published\n  results → [bowang-lab/scGPT](https://github.com/bowang-lab/scGPT)\n- Issues specific to the modernized stack (`_compat/` shim, torch 2.x build,\n  flash-attn 3, H100 deployment) → this repo\n\n## License\n\nSame as upstream scGPT (MIT). See [`LICENSE`](LICENSE).\n\n---\n\n# Original scGPT README\n\n_Everything below this line is preserved verbatim from the upstream project._\n\nThis is the official codebase for **scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI**.\n\n[![Preprint](https://img.shields.io/badge/preprint-available-brightgreen)](https://www.biorxiv.org/content/10.1101/2023.04.30.538439) \u0026nbsp;\n[![Documentation](https://img.shields.io/badge/docs-available-brightgreen)](https://scgpt.readthedocs.io/en/latest/) \u0026nbsp;\n[![PyPI version](https://badge.fury.io/py/scgpt.svg)](https://badge.fury.io/py/scgpt) \u0026nbsp;\n[![Downloads](https://pepy.tech/badge/scgpt)](https://pepy.tech/project/scgpt) \u0026nbsp;\n![Webapp](https://img.shields.io/website?url=https%3A%2F%2Fscgpthub.org\u0026up_color=chartreuse%20\u0026logo=gotomeeting\u0026logoColor=%23FFB3FF\u0026label=WebApp\u0026labelColor=%2300CBFF) \u0026nbsp;\n[![License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/username/repo/blob/main/LICENSE)\n\n**!UPDATE**: We have released several new pretrained scGPT checkpoints. Please see the [Pretrained scGPT checkpoints](#pretrained-scGPT-checkpoints) section for more details.\n\n**[2024.02.26]** We have provided a priliminary support for running the pretraining workflow with HuggingFace at the [integrate-huggingface-model](https://github.com/bowang-lab/scGPT/tree/integrate-huggingface-model) branch. We will conduct further testing and merge it to the main branch soon.\n\n**[2023.12.31]** New tutorials about zero-shot applications are now available! Please see find them in the [tutorials/zero-shot](tutorials/zero-shot) directory. We also provide a new continual pretrained model checkpoint for cell embedding related tasks. Please see the [notebook](tutorials/zero-shot/Tutorial_ZeroShot_Integration_Continual_Pretraining.ipynb) for more details.\n\n**[2023.11.07]** As requested by many, now we have made flash-attention an optional dependency. The pretrained weights can be loaded on pytorch CPU, GPU, and flash-attn backends using the same [load_pretrained](https://github.com/bowang-lab/scGPT/blob/f6097112fe5175cd4e221890ed2e2b1815f54010/scgpt/utils/util.py#L304) function, `load_pretrained(target_model, torch.load(\"path_to_ckpt.pt\"))`. An example usage is also [here](https://github.com/bowang-lab/scGPT/blob/f6097112fe5175cd4e221890ed2e2b1815f54010/scgpt/tasks/cell_emb.py#L258).\n\n**[2023.09.05]** We have release a new feature for reference mapping samples to a custom reference dataset or to all the millions of cells collected from CellXGene! With the help of the [faiss](https://github.com/facebookresearch/faiss) library, we achieved a great time and memory efficiency. The index of over 33 millions cells only takes less than 1GB of memory and the similarity search takes less than **1 second for 10,000 query cells on GPU**. Please see the [Reference mapping tutorial](https://github.com/bowang-lab/scGPT/blob/main/tutorials/Tutorial_Reference_Mapping.ipynb) for more details.\n\n### Online apps\n\nscGPT is now available at the following online apps as well, so you can get started simply with your browser!\n\n- Run the [reference mapping app](https://app.superbio.ai/apps/299?id=6548f339a9ed6f6e5560b07d), [cell annotation app](https://app.superbio.ai/apps/274?id=64d205cb980ff714de831ee0) and the [GRN inference app](https://app.superbio.ai/apps/270?id=64b804fb823bc93b64c10a76) with cloud gpus. Thanks to the [Superbio.ai](https://app.superbio.ai/) team for helping create and host the interactive tools.\n\n## Installation\n\nscGPT works with Python \u003e= 3.7.13 and R \u003e=3.6.1. Please make sure you have the correct version of Python and R installed pre-installation.\n\nscGPT is available on PyPI. To install scGPT, run the following command:\n\n```bash\npip install scgpt \"flash-attn\u003c1.0.5\"  # optional, recommended\n# As of 2023.09, pip install may not run with new versions of the google orbax package, if you encounter related issues, please use the following command instead:\n# pip install scgpt \"flash-attn\u003c1.0.5\" \"orbax\u003c0.1.8\"\n```\n\n[Optional] We recommend using [wandb](https://wandb.ai/) for logging and visualization.\n\n```bash\npip install wandb\n```\n\nThe poetry installation is out of sync. Please use pip install instead. ~~For developing, we are using the [Poetry](https://python-poetry.org/) package manager. To install Poetry, follow the instructions [here](https://python-poetry.org/docs/#installation).~~\n\n```bash\n$ git clone this-repo-url\n$ cd scGPT\n$ poetry install\n```\n\n**Note**: The `flash-attn` dependency usually requires specific GPU and CUDA version. If you encounter any issues, please refer to the [flash-attn](https://github.com/HazyResearch/flash-attention/tree/main) repository for installation instructions. For now, May 2023, we recommend using CUDA 11.7 and flash-attn\u003c1.0.5 due to various issues reported about installing new versions of flash-attn.\n\n## Pretrained scGPT Model Zoo\n\nHere is the list of pretrained models. Please find the links for downloading the checkpoint folders. We recommend using the `whole-human` model for most applications by default. If your fine-tuning dataset shares similar cell type context with the training data of the organ-specific models, these models can usually demonstrate competitive performance as well. A paired vocabulary file mapping gene names to ids is provided in each checkpoint folder. If ENSEMBL ids are needed, please find the conversion at [gene_info.csv](https://github.com/bowang-lab/scGPT/files/13243634/gene_info.csv).\n\n| Model name                | Description                                             | Download                                                                                     |\n| :------------------------ | :------------------------------------------------------ | :------------------------------------------------------------------------------------------- |\n| whole-human (recommended) | Pretrained on 33 million normal human cells.            | [link](https://drive.google.com/drive/folders/1oWh_-ZRdhtoGQ2Fw24HP41FgLoomVo-y?usp=sharing) |\n| continual pretrained      | For zero-shot cell embedding related tasks.             | [link](https://drive.google.com/drive/folders/1_GROJTzXiAV8HB4imruOTk6PEGuNOcgB?usp=sharing) |\n| brain                     | Pretrained on 13.2 million brain cells.                 | [link](https://drive.google.com/drive/folders/1vf1ijfQSk7rGdDGpBntR5bi5g6gNt-Gx?usp=sharing) |\n| blood                     | Pretrained on 10.3 million blood and bone marrow cells. | [link](https://drive.google.com/drive/folders/1kkug5C7NjvXIwQGGaGoqXTk_Lb_pDrBU?usp=sharing) |\n| heart                     | Pretrained on 1.8 million heart cells                   | [link](https://drive.google.com/drive/folders/1GcgXrd7apn6y4Ze_iSCncskX3UsWPY2r?usp=sharing) |\n| lung                      | Pretrained on 2.1 million lung cells                    | [link](https://drive.google.com/drive/folders/16A1DJ30PT6bodt4bWLa4hpS7gbWZQFBG?usp=sharing) |\n| kidney                    | Pretrained on 814 thousand kidney cells                 | [link](https://drive.google.com/drive/folders/1S-1AR65DF120kNFpEbWCvRHPhpkGK3kK?usp=sharing) |\n| pan-cancer                | Pretrained on 5.7 million cells of various cancer types | [link](https://drive.google.com/drive/folders/13QzLHilYUd0v3HTwa_9n4G4yEF-hdkqa?usp=sharing) |\n\n## Fine-tune scGPT for scRNA-seq integration\n\nPlease see our example code in [examples/finetune_integration.py](examples/finetune_integration.py). By default, the script assumes the scGPT checkpoint folder stored in the `examples/save` directory.\n\n## To-do-list\n\n- [x] Upload the pretrained model checkpoint\n- [x] Publish to pypi\n- [ ] Provide the pretraining code with generative attention masking\n- [ ] Finetuning examples for multi-omics integration, cell type annotation, perturbation prediction, cell generation\n- [x] Example code for Gene Regulatory Network analysis\n- [x] Documentation website with readthedocs\n- [x] Bump up to pytorch 2.0\n- [x] New pretraining on larger datasets\n- [x] Reference mapping example\n- [ ] Publish to huggingface model hub\n\n## Contributing\n\nWe greatly welcome contributions to scGPT. Please submit a pull request if you have any ideas or bug fixes. We also welcome any issues you encounter while using scGPT.\n\n## Acknowledgements\n\nWe sincerely thank the authors of following open-source projects:\n\n- [flash-attention](https://github.com/HazyResearch/flash-attention)\n- [scanpy](https://github.com/scverse/scanpy)\n- [scvi-tools](https://github.com/scverse/scvi-tools)\n- [scib](https://github.com/theislab/scib)\n- [datasets](https://github.com/huggingface/datasets)\n- [transformers](https://github.com/huggingface/transformers)\n\n## Citing scGPT\n\n```bibtex\n@article{cui2023scGPT,\ntitle={scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI},\nauthor={Cui, Haotian and Wang, Chloe and Maan, Hassaan and Pang, Kuan and Luo, Fengning and Wang, Bo},\njournal={bioRxiv},\nyear={2023},\npublisher={Cold Spring Harbor Laboratory}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchansigit%2Fscgpt-modern","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchansigit%2Fscgpt-modern","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchansigit%2Fscgpt-modern/lists"}