{"id":27949493,"url":"https://github.com/songlab-cal/gpn","last_synced_at":"2025-05-07T15:28:15.821Z","repository":{"id":58192027,"uuid":"462508804","full_name":"songlab-cal/gpn","owner":"songlab-cal","description":"Genomic Pre-trained Network","archived":false,"fork":false,"pushed_at":"2025-03-30T19:53:45.000Z","size":54580,"stargazers_count":254,"open_issues_count":2,"forks_count":39,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-30T20:28:57.546Z","etag":null,"topics":["dna","genomics","language-model","variant-effect-prediction"],"latest_commit_sha":null,"homepage":"https://doi.org/10.1073/pnas.2311219120","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/songlab-cal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-22T23:21:19.000Z","updated_at":"2025-03-30T19:53:43.000Z","dependencies_parsed_at":"2023-02-18T12:19:28.607Z","dependency_job_id":"914bedd4-074b-496a-a87e-a4b58125a51f","html_url":"https://github.com/songlab-cal/gpn","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/songlab-cal%2Fgpn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/songlab-cal%2Fgpn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/songlab-cal%2Fgpn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/songlab-cal%2Fgpn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/songlab-cal","download_url":"https://codeload.github.com/songlab-cal/gpn/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252904856,"owners_count":21822687,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dna","genomics","language-model","variant-effect-prediction"],"created_at":"2025-05-07T15:28:15.201Z","updated_at":"2025-05-07T15:28:15.809Z","avatar_url":"https://github.com/songlab-cal.png","language":"Jupyter Notebook","funding_links":[],"categories":["🔬 Domain-Specific Applications","genomic language models","Machine Learning Tasks and Models"],"sub_categories":["🧬 Biology \u0026 Medicine","Plant","Foundation Models"],"readme":"# GPN (Genomic Pre-trained Network)\n[![hgt_genome_392c4_a47ce0](https://github.com/user-attachments/assets/282b6204-156b-4b6d-83ff-2f4a53a9bb2e)](https://genome.ucsc.edu/s/gbenegas/gpn-arabidopsis)\n \nCode and resources from [GPN](https://doi.org/10.1073/pnas.2311219120) and related genomic language models.\n\n## Table of contents\n- [Installation](#installation)\n- [Quick start](#quick-start)\n- [Modeling frameworks](#modeling-frameworks)\n- [Applications of the models](#applications-of-the-models)\n- [GPN](#gpn)\n- [GPN-MSA](#gpn-msa)\n- [PhyloGPN](#phylogpn)\n- [Citation](#citation)\n\n## Installation\n```bash\npip install git+https://github.com/songlab-cal/gpn.git\n```\n\n## Quick start\n```python\nimport gpn.model\nfrom transformers import AutoModelForMaskedLM, AutoModel\n\nmodel = AutoModelForMaskedLM.from_pretrained(\"songlab/gpn-brassicales\")\n# or\nmodel = AutoModelForMaskedLM.from_pretrained(\"songlab/gpn-msa-sapiens\")\n# or\nmodel = AutoModel.from_pretrained(\"songlab/PhyloGPN\", trust_remote_code=True)\n```\n\n## Modeling frameworks\n| Model | Paper | Notes |\n| --------- | --- | ----------- |\n| GPN | [Benegas et al. 2023](https://doi.org/10.1073/pnas.2311219120) | Requires unaligned genomes | \n| GPN-MSA | [Benegas et al. 2025](https://www.nature.com/articles/s41587-024-02511-w) | Requires aligned genomes for both training and inference |\n| PhyloGPN | [Albors et al. 2025] | Uses an alignment during training, but does not require it for inference or fine-tuning |\n\n## Applications of the models\n| Paper |  Model | Dataset | Code | Resources on HuggingFace 🤗 |\n|  -- | --- | ------- | ---- | -------------- |\n| [Benegas et al. 2023](https://doi.org/10.1073/pnas.2311219120) | GPN | Arabidopsis and other Brassicale plants | [analysis/gpn_arabidopsis](analysis/gpn_arabidopsis) |  [Model, dataset, intermediate results](https://huggingface.co/collections/songlab/gpn-653191edcb0270ed05ad2c3e) |\n| [Benegas et al. 2025](https://www.nature.com/articles/s41587-024-02511-w) | GPN-MSA | Human and other vertebrates | [analysis/gpn-msa_human](analysis/gpn-msa_human) | [Model, dataset, benchmarks, predictions](https://huggingface.co/collections/songlab/gpn-msa-65319280c93c85e11c803887) |\n| [Benegas et al. 2025b](https://www.biorxiv.org/content/10.1101/2025.02.11.637758v1) | GPN | Animal promoters | [analysis/gpn_animal_promoter](analysis/gpn_animal_promoter) | [Model, dataset, benchmarks](https://huggingface.co/collections/songlab/traitgym-6796d4fbb825d5b94e65d30f) |\n \n## GPN\nCan also be called GPN-SS (single sequence).\n\n### Examples\n* Play with the model: `examples/ss/basic_example.ipynb` [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/songlab-cal/gpn/blob/main/examples/ss/basic_example.ipynb)\n\n### Training on your own data\n1. [Snakemake workflow to create a dataset](workflow/make_dataset)\n    - Can automatically download data from NCBI given a list of accessions, or use your own fasta files.\n2. Training\n    - Will automatically detect all available GPUs.\n    - Track metrics on [Weights \u0026 Biases](https://wandb.ai/)\n    - Implemented encoders: `convnet` (default), `roformer` (Transformer), `bytenet`\n    - Specify config overrides: e.g. `--config_overrides encoder=bytenet,num_hidden_layers=30`\n    - The number of steps that you can train without overfitting will be a function of the size and diversity of your dataset\n    - Example:\n```bash\nWANDB_PROJECT=your_project torchrun --nproc_per_node=$(echo $CUDA_VISIBLE_DEVICES | awk -F',' '{print NF}') -m gpn.ss.run_mlm --do_train --do_eval \\\n    --report_to wandb --prediction_loss_only True --remove_unused_columns False \\\n    --dataset_name results/dataset --tokenizer_name gonzalobenegas/tokenizer-dna-mlm \\\n    --soft_masked_loss_weight_train 0.1 --soft_masked_loss_weight_evaluation 0.0 \\\n    --weight_decay 0.01 --optim adamw_torch \\\n    --dataloader_num_workers 16 --seed 42 \\\n    --save_strategy steps --save_steps 10000 --evaluation_strategy steps \\\n    --eval_steps 10000 --logging_steps 10000 --max_steps 120000 --warmup_steps 1000 \\\n    --learning_rate 1e-3 --lr_scheduler_type constant_with_warmup \\\n    --run_name your_run --output_dir your_output_dir --model_type GPN \\\n    --per_device_train_batch_size 512 --per_device_eval_batch_size 512 --gradient_accumulation_steps 1 --total_batch_size 2048 \\ \n    --torch_compile \\\n    --ddp_find_unused_parameters False \\\n    --bf16 --bf16_full_eval \\\n```\n3. Extract embeddings\n    - Input file requires `chrom`, `start`, `end`\n    - Example:\n```bash\ntorchrun --nproc_per_node=$(echo $CUDA_VISIBLE_DEVICES | awk -F',' '{print NF}') -m gpn.ss.get_embeddings windows.parquet genome.fa.gz 100 your_output_dir \\\n    results.parquet --per_device_batch_size 4000 --is_file --dataloader_num_workers 16\n```\n4. Variant effect prediction\n    - Input file requires `chrom`, `pos`, `ref`, `alt`\n    - Example:\n```bash\ntorchrun --nproc_per_node=$(echo $CUDA_VISIBLE_DEVICES | awk -F',' '{print NF}') -m gpn.ss.run_vep variants.parquet genome.fa.gz 512 your_output_dir results.parquet \\\n    --per_device_batch-size 4000 --is_file --dataloader_num_workers 16\n```\n\n## GPN-MSA\n\n### Examples\n* Play with the model: `examples/msa/basic_example.ipynb`\n* Variant effect prediction: `examples/msa/vep.ipynb`\n* Training (human): `examples/msa/training.ipynb`\n\n### Training on other species (e.g. other vertebrates, plants)\n* See https://github.com/songlab-cal/gpn/issues/28, https://github.com/songlab-cal/gpn/discussions/40, https://github.com/songlab-cal/gpn/issues/44\n* Another source for plant alignments: https://plantregmap.gao-lab.org/download.php#alignment-conservation\n\n## PhyloGPN\nPhyloGPN is a convolutional neural network that takes encoded DNA sequences as input and outputs rate matrix parameters for [Felsenstein's 1981 model](https://en.wikipedia.org/wiki/Models_of_DNA_evolution#F81_model_(Felsenstein_1981)) (the F81 model, for short). It was trained to maximize the likelihood of columns in the [Zoonomia alignment](https://cglgenomics.ucsc.edu/november-2023-nature-zoonomia-with-expanded-primates-alignment/) given a phylogenetic tree. The stationary distribution of the substitution process described by the F81 model indicates the relative viability of each allele at any given locus. As a result, PhyloGPN is formally a (single-sequence) genomic language model. It can be used for transfer learning and zero-shot SNV deleteriousness prediction. It is especially useful for sequences that are not directly in the human reference genome.\n\n## Citation\n[GPN](https://doi.org/10.1073/pnas.2311219120):\n```bibtex\n@article{benegas2023dna,\n  title={DNA language models are powerful predictors of genome-wide variant effects},\n  author={Benegas, Gonzalo and Batra, Sanjit Singh and Song, Yun S},\n  journal={Proceedings of the National Academy of Sciences},\n  volume={120},\n  number={44},\n  pages={e2311219120},\n  year={2023},\n  publisher={National Acad Sciences}\n}\n```\n\n[GPN-MSA](https://www.nature.com/articles/s41587-024-02511-w):\n```bibtex\n@article{benegas2025dna,\n  title={A DNA language model based on multispecies alignment predicts the effects of genome-wide variants},\n  author={Benegas, Gonzalo and Albors, Carlos and Aw, Alan J and Ye, Chengzhong and Song, Yun S},\n  journal={Nature Biotechnology},\n  pages={1--6},\n  year={2025},\n  publisher={Nature Publishing Group US New York}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsonglab-cal%2Fgpn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsonglab-cal%2Fgpn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsonglab-cal%2Fgpn/lists"}