{"id":13685182,"url":"https://github.com/hpcaitech/FastFold","last_synced_at":"2025-05-01T01:31:02.128Z","repository":{"id":38073414,"uuid":"464065100","full_name":"hpcaitech/FastFold","owner":"hpcaitech","description":"Optimizing AlphaFold Training and Inference on GPU Clusters","archived":false,"fork":false,"pushed_at":"2024-07-16T06:14:42.000Z","size":1323,"stargazers_count":599,"open_issues_count":44,"forks_count":89,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-04-13T00:49:09.794Z","etag":null,"topics":["alphafold2","cuda","evoformer","gpu","habana-gaudi","parallelism","protein-folding","protein-structure","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hpcaitech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-27T07:11:44.000Z","updated_at":"2025-04-04T04:13:35.000Z","dependencies_parsed_at":"2024-01-14T16:11:49.542Z","dependency_job_id":"1f4e6efd-b421-491a-a892-8fbacd672ef3","html_url":"https://github.com/hpcaitech/FastFold","commit_stats":{"total_commits":95,"total_committers":12,"mean_commits":7.916666666666667,"dds":0.6105263157894737,"last_synced_commit":"eba496808a91bbcd9661cf832349a418b197015f"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcaitech%2FFastFold","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcaitech%2FFastFold/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcaitech%2FFastFold/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcaitech%2FFastFold/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hpcaitech","download_url":"https://codeload.github.com/hpcaitech/FastFold/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251808405,"owners_count":21647285,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alphafold2","cuda","evoformer","gpu","habana-gaudi","parallelism","protein-folding","protein-structure","pytorch"],"created_at":"2024-08-02T14:00:45.640Z","updated_at":"2025-05-01T01:31:01.725Z","avatar_url":"https://github.com/hpcaitech.png","language":"Python","funding_links":[],"categories":["Structure prediction"],"sub_categories":[],"readme":"![](/assets/fold.jpg)\n\n# FastFold\n\n[![](https://img.shields.io/badge/Paper-PDF-green?style=flat\u0026logo=arXiv\u0026logoColor=green)](https://arxiv.org/abs/2203.00854)\n![](https://img.shields.io/badge/Made%20with-ColossalAI-blueviolet?style=flat)\n![](https://img.shields.io/badge/Habana-support-blue?style=flat\u0026logo=intel\u0026logoColor=blue)\n![](https://img.shields.io/github/v/release/hpcaitech/FastFold)\n[![GitHub license](https://img.shields.io/github/license/hpcaitech/FastFold)](https://github.com/hpcaitech/FastFold/blob/main/LICENSE)\n\n## News :triangular_flag_on_post:\n- [2023/01] Compatible with AlphaFold v2.3\n- [2023/01] Added support for inference and training of AlphaFold on [Intel Habana](https://habana.ai/) platform. For usage instructions, see [here](#Inference-or-Training-on-Intel-Habana).\n\n\u003cbr\u003e\n\nOptimizing Protein Structure Prediction Model Training and Inference on Heterogeneous Clusters\n\nFastFold provides a **high-performance implementation of Evoformer** with the following characteristics.\n\n1. Excellent kernel performance on GPU platform\n2. Supporting Dynamic Axial Parallelism(DAP)\n    * Break the memory limit of single GPU and reduce the overall training time\n    * DAP can significantly speed up inference and make ultra-long sequence inference possible\n3. Ease of use\n    * Huge performance gains with a few lines changes\n    * You don't need to care about how the parallel part is implemented\n4. Faster data processing, about 3x times faster on monomer, about 3Nx times faster on multimer with N sequence.\n5. Great Reduction on GPU memory, able to inference sequence containing more than **10000** residues.\n\n## Installation\n\nTo install FastFold, you will need:\n+ Python 3.8 or 3.9.\n+ [NVIDIA CUDA](https://developer.nvidia.com/cuda-downloads) 11.3 or above\n+ PyTorch 1.12 or above \n\nFor now, You can install FastFold:\n### Using Conda (Recommended)\n\nWe highly recommend installing an Anaconda or Miniconda environment and install PyTorch with conda.\nLines below would create a new conda environment called \"fastfold\":\n\n```shell\ngit clone https://github.com/hpcaitech/FastFold\ncd FastFold\nconda env create --name=fastfold -f environment.yml\nconda activate fastfold\npython setup.py install\n```\n\n#### Advanced\n\nTo leverage the power of FastFold, we recommend you to install [Triton](https://github.com/openai/triton).\n\n**NOTE: Triron needs CUDA 11.4 to run.**\n\n```bash\npip install -U --pre triton\n```\n\n\n## Use Docker\n\n### Build On Your Own\nRun the following command to build a docker image from Dockerfile provided.\n\n\u003e Building FastFold from scratch requires GPU support, you need to use Nvidia Docker Runtime as the default when doing `docker build`. More details can be found [here](https://stackoverflow.com/questions/59691207/docker-build-with-nvidia-runtime).\n\n```shell\ncd FastFold\ndocker build -t fastfold ./docker\n```\n\nRun the following command to start the docker container in interactive mode.\n```shell\ndocker run -ti --gpus all --rm --ipc=host fastfold bash\n```\n\n## Usage\n\nYou can use `Evoformer` as `nn.Module` in your project after `from fastfold.model.fastnn import Evoformer`:\n\n```python\nfrom fastfold.model.fastnn import Evoformer\nevoformer_layer = Evoformer()\n```\n\nIf you want to use Dynamic Axial Parallelism, add a line of initialize with `fastfold.distributed.init_dap`.\n\n```python\nfrom fastfold.distributed import init_dap\n\ninit_dap(args.dap_size)\n```\n\n### Download the dataset\nYou can down the dataset used to train FastFold  by the script `download_all_data.sh`:\n\n    ./scripts/download_all_data.sh data/\n\n### Inference\n\nYou can use FastFold with `inject_fastnn`. This will replace the evoformer from OpenFold with the high performance evoformer from FastFold.\n\n```python\nfrom fastfold.utils import inject_fastnn\n\nmodel = AlphaFold(config)\nimport_jax_weights_(model, args.param_path, version=args.model_name)\n\nmodel = inject_fastnn(model)\n```\n\nFor Dynamic Axial Parallelism, you can refer to `./inference.py`. Here is an example of 2 GPUs parallel inference:\n\n```shell\npython inference.py target.fasta data/pdb_mmcif/mmcif_files/ \\\n    --output_dir .outputs/ \\\n    --gpus 2 \\\n    --uniref90_database_path data/uniref90/uniref90.fasta \\\n    --mgnify_database_path data/mgnify/mgy_clusters_2022_05.fa \\\n    --pdb70_database_path data/pdb70/pdb70 \\\n    --uniref30_database_path data/uniref30/UniRef30_2021_03 \\\n    --bfd_database_path data/bfd/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt \\\n    --jackhmmer_binary_path `which jackhmmer` \\\n    --hhblits_binary_path `which hhblits` \\\n    --hhsearch_binary_path `which hhsearch` \\\n    --kalign_binary_path `which kalign` \\\n    --enable_workflow \\\n    --inplace\n```\nor run the script `./inference.sh`, you can change the parameter in the script, especisally those data path.\n```shell\n./inference.sh\n```\n\nAlphafold's data pre-processing takes a lot of time, so we speed up the data pre-process by [ray](https://docs.ray.io/en/latest/workflows/concepts.html) workflow, which achieves a 3x times faster speed. To run the inference with ray workflow, we add parameter `--enable_workflow` by default.\nTo reduce memory usage of embedding presentations, we also add parameter `--inplace` to share memory by defaul.\n\n#### inference with lower memory usage\nAlphafold's embedding presentations take up a lot of memory as the sequence length increases. To reduce memory usage, \nyou should add parameter `--chunk_size [N]` to cmdline or shell script `./inference.sh`. \nThe smaller you set N, the less memory will be used, but it will affect the speed. We can inference \na sequence of length 10000 in bf16 with 61GB memory on a Nvidia A100(80GB). For fp32, the max length is 8000.\n\u003e You need to set `PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:15000` to inference such an extreme long sequence.\n\n```shell\npython inference.py target.fasta data/pdb_mmcif/mmcif_files/ \\\n    --output_dir .outputs/ \\\n    --gpus 2 \\\n    --uniref90_database_path data/uniref90/uniref90.fasta \\\n    --mgnify_database_path data/mgnify/mgy_clusters_2022_05.fa \\\n    --pdb70_database_path data/pdb70/pdb70 \\\n    --uniref30_database_path data/uniref30/UniRef30_2021_03 \\\n    --bfd_database_path data/bfd/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt \\\n    --jackhmmer_binary_path `which jackhmmer` \\\n    --hhblits_binary_path `which hhblits` \\\n    --hhsearch_binary_path `which hhsearch` \\\n    --kalign_binary_path `which kalign`  \\\n    --enable_workflow \\\n    --inplace\n    --chunk_size N \\\n```\n\n#### inference multimer sequence\nAlphafold Multimer is supported. You can the following cmd or shell script `./inference_multimer.sh`.\nWorkflow and memory parameters mentioned above can also be used.\n```shell\npython inference.py target.fasta data/pdb_mmcif/mmcif_files/ \\\n    --output_dir ./ \\\n    --gpus 2 \\\n    --model_preset multimer \\\n    --uniref90_database_path data/uniref90/uniref90.fasta \\\n    --mgnify_database_path data/mgnify/mgy_clusters_2022_05.fa \\\n    --pdb70_database_path data/pdb70/pdb70 \\\n    --uniref30_database_path data/uniref30/UniRef30_2021_03 \\\n    --bfd_database_path data/bfd/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt \\\n    --uniprot_database_path data/uniprot/uniprot.fasta \\\n    --pdb_seqres_database_path data/pdb_seqres/pdb_seqres.txt  \\\n    --param_path data/params/params_model_1_multimer.npz \\\n    --model_name model_1_multimer \\\n    --jackhmmer_binary_path `which jackhmmer` \\\n    --hhblits_binary_path `which hhblits` \\\n    --hhsearch_binary_path `which hhsearch` \\\n    --kalign_binary_path `which kalign`\n```\n\n### Inference or Training on Intel Habana\n\nTo run AlphaFold inference or training on Intel Habana, you can follow the instructions in the [Installation Guide](https://docs.habana.ai/en/latest/Installation_Guide/) to set up your environment on Amazon EC2 DL1 instances or on-premise environments, and please use SynapseAI R1.7.1 to test as it was verified internally.\n\nOnce you have prepared your dataset and installed fastfold, you can use the following scripts:\n\n```shell\ncd fastfold/habana/fastnn/custom_op/; python setup.py build (this is for Gaudi, for Gaudi2 please use setup2.py) ; cd -\nbash habana/inference.sh\nbash habana/train.sh\n```\n\n## Performance Benchmark\n\nWe have included a performance benchmark script in `./benchmark`. You can benchmark the performance of Evoformer using different settings.\n\n```shell\ncd ./benchmark\ntorchrun --nproc_per_node=1 perf.py --msa-length 128 --res-length 256\n```\n\nBenchmark Dynamic Axial Parallelism with 2 GPUs:\n\n```shell\ncd ./benchmark\ntorchrun --nproc_per_node=2 perf.py --msa-length 128 --res-length 256 --dap-size 2\n```\n\nIf you want to benchmark with [OpenFold](https://github.com/aqlaboratory/openfold), you need to install OpenFold first and benchmark with option `--openfold`:\n\n```shell\ntorchrun --nproc_per_node=1 perf.py --msa-length 128 --res-length 256 --openfold\n```\n\n## Cite us\n\nCite this paper, if you use FastFold in your research publication.\n\n```\n@misc{cheng2022fastfold,\n      title={FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours}, \n      author={Shenggan Cheng and Ruidong Wu and Zhongming Yu and Binrui Li and Xiwen Zhang and Jian Peng and Yang You},\n      year={2022},\n      eprint={2203.00854},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n```\n\n## Acknowledgments\n\nWe would like to extend our special thanks to the Intel Habana team for their support in providing us with technology and resources on the Habana platform.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhpcaitech%2FFastFold","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhpcaitech%2FFastFold","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhpcaitech%2FFastFold/lists"}