{"id":19482509,"url":"https://github.com/aqlaboratory/insilico_design_pipeline","last_synced_at":"2025-09-12T20:42:43.706Z","repository":{"id":242109055,"uuid":"808365044","full_name":"aqlaboratory/insilico_design_pipeline","owner":"aqlaboratory","description":"In-silico design pipeline for evaluating protein structure diffusion models.","archived":false,"fork":false,"pushed_at":"2024-06-25T08:41:12.000Z","size":128,"stargazers_count":27,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-09-09T09:36:34.129Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aqlaboratory.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-30T23:23:45.000Z","updated_at":"2025-06-27T02:34:06.000Z","dependencies_parsed_at":"2025-01-08T07:25:20.422Z","dependency_job_id":"974e3f74-e2f2-419f-b10d-d79d614943e1","html_url":"https://github.com/aqlaboratory/insilico_design_pipeline","commit_stats":null,"previous_names":["aqlaboratory/insilico_design_pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aqlaboratory/insilico_design_pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqlaboratory%2Finsilico_design_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqlaboratory%2Finsilico_design_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqlaboratory%2Finsilico_design_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqlaboratory%2Finsilico_design_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aqlaboratory","download_url":"https://codeload.github.com/aqlaboratory/insilico_design_pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqlaboratory%2Finsilico_design_pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274873613,"owners_count":25365824,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-12T02:00:09.324Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T20:10:50.824Z","updated_at":"2025-09-12T20:42:43.678Z","avatar_url":"https://github.com/aqlaboratory.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# In-silio Protein Design Pipeline\n\nThis repository contains the in-silico protein design and evaluation pipeline that we used for assessing [Genie 2](https://arxiv.org/abs/2405.15489). We set this up separately from [Genie 2 repository](https://github.com/aqlaboratory/genie2) to facilitate assessments of different structure-based protein diffusion models. The pipeline consists of:\n- assessment on designability through self-consistency pipeline\n- assessment on secondary diversity through P-SEA algorithm\n- assessment on tertiary diversity through hierarchical clustering\n- assessment on novelty\n\n## Set up\nAssume the environment has a cuda-compatiable PyTorch installed and Python \u003c= 3.9. For example, on our own machine, the environment is created and initialized by running.\n\n```\npython3.9 -m venv insilico_pipeline_venv\nsource insilico_pipeline_venv/bin/activate\nmodule load cuda11.8\npip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --index-url https://download.pytorch.org/whl/cu118\n```\n\nThe setup process consists of three parts:\n\n- Set up the pipeline package and additional packages (TMscore and TMalign) by running\n\n  ```\n  bash scripts/setup/setup.sh\n  ```\n\n- Set up an inverse folding model and its dependencies by running\n\n  ```\n  bash scripts/setup/inverse_folds/setup_[INVERSE_FOLD_MODEL_NAME].sh\n  ```\n\n  Our current pipeline supports ProteinMPNN (`proteinmpnn`) and we intend to extend this to include more inverse folding models.\n\n- Set up a folding model and its dependencies by running\n\n  ```\n  bash scripts/setup/folds/setup_[FOLD_MODEL_NAME].sh\n  ```\n  \n  Our current pipeline supports ESMFold (`esmfold`) and we intend to extend this to include more folding models.\n\n### Additional notes\n\nWhen setting up the environment for ESMFold, we install OpenFold v1.0.1 to ensure compatibility. One known [issue](https://github.com/aqlaboratory/openfold/issues/276) for this OpenFold installation is its compatibility with deepspeed. This raises `AttributeError: module 'deepspeed.utils' has no attribute 'is_initialized'` when running the pipeline and could be fixed by replacing all occurences of `deepspeed.utils.is_initialized()` with `deepspeed.comm.comm.is_initialized()`.\n\n## Pipelines\n\nOur design package consists of three separate pipelines:\n- standard pipeline for assessing designability and secondary diversity\n- diversity pipeline for assessing tertiary diversity\n- novelty pipeline for assessing novelty with respect to a reference dataset.\n\n### Standard pipeline (`pipeline/standard`)\n\nEvaluate a set of generated structures by running\n\n```\npython pipeline/standard/evaluate.py --version [VERSION] --rootdir [ROOTDIR]\n```\n\nOur standard pipeline currently supports evaluation of structures from unconditional generation (by setting version to `unconditional`) and motif scaffolding (`scaffold`). For both modes, we assume that the root directory contains a folder named `pdbs`, which contains the PDB files of generated structures to be evaluated. For motif scaffolding, we additionally assume that the root directory contains a folder named `motif_pdbs`, which contains the PDB files of the corresponding motif structures (with the same filename as the generated structure and residue index aligned). Note that for motif scaffolding, we also support evaluations of multiple problems at the same time. This means that the root directory could contain a list of subdirectories, each of which consists of a `pdbs` and `motif_pdbs` folder detailed above. When evaluating multiple motif scaffolding problems, our pipeline supports distribution of tasks across multiple GPUS by adding the following flags `--num_devices [NUM_GPUS] --num_processes [NUM_GPUS]`.\n\nEvaluation results are stored in the root directory, which contains:\n- a directory named `designs`, where each PDB file stores the fold model predicted structure that is most similar to the corresponding generated structure;\n- a csv file named `info.csv`, which contains evaluation statistics for the set of generated structures. Information on columns is summarized in the table below.\n\n  | Column | Description |\n  | :--- | :--------------------------- |\n  | `domain`                   | Name of generated structure |\n  | `seqlen`                   | Sequence length of generated structure |\n  | `scRMSD`                   | RMSD between the generated structure and the most \u003cbr\u003esimilar structure predicted by the specified fold model |\n  | `scTM`                     | TM score between the generated structure and the most \u003cbr\u003esimilar structure predicted by the specified fold model |\n  | `pLDDT`                    | Local confidence from the specified fold model, \u003cbr\u003eaveraged across all residues |\n  | `pAE`                      | Confidence from the specified fold model in the \u003cbr\u003erelative position of two residues, averaged across all \u003cbr\u003eresidue-residue pairs |\n  | `generated_pct_helix`      | Percentage of helices in the generated structure |\n  | `generated_pct_strand`     | Percentage of strands in the generated structure |\n  | `generated_pct_ss`         | Percentage of helices and strands in the \u003cbr\u003egenerated structure |\n  | `generated_pct_left_helix` | Percentage of left-handed helices in the \u003cbr\u003egenerated structure |\n  | `designed_pct_helix`       | Percentage of helices in the most similar structure \u003cbr\u003epredicted by the specific fold model |\n  | `designed_pct_strand`      | Percentage of strands in the most similar structure \u003cbr\u003epredicted by the specific fold model |\n  | `designed_pct_ss`          | Percentage of helices and strands in the most similar \u003cbr\u003estructure predicted by the specific fold model |\n  | `designed_pct_left_helix`  | Percentage of left-handed helices in the most similar \u003cbr\u003estructure predicted by the specific fold model |\n\nNote that for secondary structure evaluations, we use the P-SEA algorithm, which allows us to predict secondary structures based on Ca atoms only.\n\n### Diversity pipeline (`pipeline/diversity`)\n\nAssume that a set of generated structure is assessed by the above standard pipeline. Evaluate this set of generated structures on tertiary diversity by running\n\n```\npython pipeline/diversity/evaluate.py --rootdir [ROOTDIR] --num_cpus [NUM_CPUS]\n```\nOur default value of `num_cpus` is 1. We found this can be slow, so we recommend setting the value to the number of physical cores or the number of processes you want to run in parallel. \n\nResults are stored by updating `info.csv` in the root directory to include\n\n| Column | Description |\n| :------- | :--------- |\n| `single_cluster_idx`   | Index of cluster that the generated structure belongs \u003cbr\u003e(hierarchically clustered via single linkage) |\n| `complete_cluster_idx` | Index of cluster that the generated structure belongs \u003cbr\u003e(hierarchically clustered via complete linkage) |\n| `average_cluster_idx`  | Index of cluster that the generated structure belongs \u003cbr\u003e(hierarchically clustered via average linkage) |\n\nNote that for hierarchical clustering, we use TMalign to compute pairwise TM scores among all generated structures and a TM score threshold of 0.6 in the clustering process.\n\n### Novelty pipeline (`pipeline/novelty`)\n\nAssume that a set of generated structure is assessed by the above standard pipeline. Evaluate this set of generated structures on novelty by running\n\n```\npython pipeline/novelty/evaluate.py --rootdir [ROOTDIR] --dataset [DATASET] --datadir [DATADIR] --num_cpus [NUM_CPUS]\n```\n\nwhere `DATASET` is the name of the reference dataset and `DATADIR` is the directory for the reference dataset (with each reference structure stored in a PDB format). Results are stored by updating `info.csv` in the root directory to include\n\n| Column | Description |\n| :------- | :--------- |\n| `max_[DATASET]_name` | Name of structure in the dataset that is most \u003cbr\u003esimilar to the generated structure |\n| `max_[DATASET]_tm`   | TM score between the generated structure and the \u003cbr\u003emost similar structure in the dataset |\n\n## Examples\n\nIn the `examples` directory, we provide three examples (together with their correponding outputs) to demonstrate the input and output to our evaluation pipeline. Examples include:\n-\t`unconditional`: evaluation of a unconditionally generated structure\n-\t`scaffold_single`: evaluation of a conditionally generated structure, whose generation is conditioned on a single functional motif\n-\t`scaffold_multi`: evaluation of a conditionally generated structure, whose generation is conditioned on multiple functional motifs\n\n## Profiling\n\n### Unconditional generation\n\nAssume that the standard (designability) and diversity pipelines are run. To show the evaluation metrics on the set of generated structures, run\n\n```\npython scripts/analysis/profile_unconditional.py --rootdir [ROOTDIR]\n```\n\nThis reports designability, diversity and F1 score on the set of generated structures. It also reports PDB novelty and/or AFDB novelty, provided that the corresponding novelty pipeline is run. Details on these evaluation metrics are found in the [Genie 2](https://arxiv.org/abs/2405.15489) paper.\n\n### Motif scaffolding\n\nAssume that the standard (designability) and diversity pipelines are run. To show the evaluation metrics on the set of generated structures, run\n\n```\npython scripts/analysis/profile_scaffold.py --rootdir [ROOTDIR]\n```\n\nThis reports the number of solved motif scaffolding problems and the total number of unique clusters, aggregated across all problems. Details on these evaluation metrics are found in the [Genie 2](https://arxiv.org/abs/2405.15489) paper. Here, we assume that the root directory contains a set of subdirectories, where each subdirectory starts with a prefix of `motif=` and contains inputs and outputs for a motif scaffolding problem (check out `examples/scaffold_single` and `examples/scaffold_multi` for detailed examples).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faqlaboratory%2Finsilico_design_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faqlaboratory%2Finsilico_design_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faqlaboratory%2Finsilico_design_pipeline/lists"}