{"id":22645329,"url":"https://github.com/sbl-sdsc/mmtf-spark","last_synced_at":"2025-04-12T00:33:43.418Z","repository":{"id":75069594,"uuid":"89403947","full_name":"sbl-sdsc/mmtf-spark","owner":"sbl-sdsc","description":"Methods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.","archived":false,"fork":false,"pushed_at":"2019-02-01T02:54:34.000Z","size":1650,"stargazers_count":21,"open_issues_count":9,"forks_count":32,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-25T20:15:37.085Z","etag":null,"topics":["apache-spark","big-data","machine-learning","protein-data-bank","protein-ligand-interactions","protein-protein-interaction","protein-sequences","protein-structure","scientific-computing"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sbl-sdsc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-04-25T20:30:43.000Z","updated_at":"2024-07-25T15:11:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"30085c2a-617b-4791-bff0-c071f3fdf33f","html_url":"https://github.com/sbl-sdsc/mmtf-spark","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbl-sdsc%2Fmmtf-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbl-sdsc%2Fmmtf-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbl-sdsc%2Fmmtf-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbl-sdsc%2Fmmtf-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sbl-sdsc","download_url":"https://codeload.github.com/sbl-sdsc/mmtf-spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248501674,"owners_count":21114676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","big-data","machine-learning","protein-data-bank","protein-ligand-interactions","protein-protein-interaction","protein-sequences","protein-structure","scientific-computing"],"created_at":"2024-12-09T06:05:02.796Z","updated_at":"2025-04-12T00:33:43.388Z","avatar_url":"https://github.com/sbl-sdsc.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mmtf-spark\n[![Build Status](https://www.travis-ci.org/sbl-sdsc/mmtf-spark.svg?branch=master)](https://www.travis-ci.org/sbl-sdsc/mmtf-spark)\n[![GitHub License](https://img.shields.io/github/license/sbl-sdsc/mmtf-spark.svg)](https://github.com/sbl-sdsc/mmtf-spark/blob/master/LICENSE)\n[![Version](http://img.shields.io/badge/version-0.2.0-yellowgreen.svg?style=flat)](https://github.com/sbl-sdsc/mmtf-spark)\n[![Download MMTF](http://img.shields.io/badge/download-MMTF_full-yellow.svg?style=flat)](https://mmtf.rcsb.org/v1.0/hadoopfiles/full.tar)\n[![Download MMTF Reduced](http://img.shields.io/badge/download-MMTF_reduced-orange.svg?style=flat)](https://mmtf.rcsb.org/v1.0/hadoopfiles/reduced.tar)\n[![Twitter](https://img.shields.io/twitter/url/https/twitter.com/mmtf_spec.svg?style=social)](https://twitter.com/mmtf_spec)\n\nMMTF-Spark is a Java open source project that provides APIs and sample applications for the scalable mining of 3D biomacromolecular structures, such as the Protein Data Bank (PDB) archive. MMTF-Spark uses Big Data technologies to enable high-performance parallel processing of macromolecular structures. MMTF-Spark use the following technology stack:\n* [Apache Spark](https://spark.apache.org/) a fast and general engine for large-scale distributed data processing.\n* [MMTF](https://mmtf.rcsb.org/) the Macromolecular Transmission Format for compact data storage, transmission and high-performance parsing\n* [Hadoop Sequence File](https://wiki.apache.org/hadoop/SequenceFile) a Big Data file format for parallel I/O\n* [Apache Parquet](https://parquet.apache.org/) a columnar data format to store dataframes\n* [BioJava](http://biojava.org/) a framework for processing biological data\n\n# Tutorials\nThe companion project [mmtf-workshop-2017](https://github.com/sbl-sdsc/mmtf-workshop-2017) offers an introduction to Apache Spark and in-depth tutorials and sample code how to use MMTF-Spark.\n\nIn addition, a Python version [MMTF-PySpark](https://github.com/sbl-sdsc/mmtf-pyspark) is under development. MMTF-PySpark offers demos as Jupyter notebooks as well as an experimental zero-install [Binder 2.0](https://elifesciences.org/labs/8653a61d/introducing-binder-2-0-share-your-interactive-research-environment) deployment of MMTF-PySpark.\n\n# Installation\n\n[MacOS and LINUX](docs/MacOsInstallation.md)\n\n[Windows](docs/WindowsInstallation.md)\n\n## PDB archive as MMTF-Hadoop Sequence Files\nFor high-performance, parallel processing, *mmtf-spark* can read the PDB archive in the [MMTF file format](https://doi.org/10.1371/journal.pcbi.1005575) from [Hadoop Sequence Files](https://wiki.apache.org/hadoop/SequenceFile). See [mmtf.rcsb.org](https://mmtf.rcsb.org/download.html) for more details. The installation instructions cover the download of MMTF-Hadoop Sequence files.\n\n## Running a Demo Application using spark-submit\n\nExample of running a simple structural query\n[(see PolyPeptideChainStatistics.java)](src/main/java/edu/sdsc/mmtf/spark/mappers/demos/PolyPeptideChainStatistics.java).\n\n```\nspark-submit --class edu.sdsc.mmtf.spark.mappers.demos.PolyPeptideChainStatistics  INSTALL_DIRECTORY/mmtf-spark/target/mmtf-spark-0.3.0-SNAPSHOT.jar\n```\nExample of running a structural alignment \n[(see DemoQueryVsAll.java)](src/main/java/edu/sdsc/mmtf/spark/alignments/demos/DemoQueryVsAll.java).\n\n```\nspark-submit --class edu.sdsc.mmtf.spark.alignments.demos.DemoQueryVsAll  INSTALL_DIRECTORY/mmtf-spark/target/mmtf-spark-0.3.0-SNAPSHOT.jar\n```\nExample of retrieving PDB metadata\n[(see PdbMetadataDemo.java)](src/main/java/edu/sdsc/mmtf/spark/datasets/demos/PdbMetadataDemo.java).\n\n```\nspark-submit --class edu.sdsc.mmtf.spark.datasets.demos.PdbMetadataDemo  INSTALL_DIRECTORY/mmtf-spark/target/mmtf-spark-0.3.0-SNAPSHOT.jar\n```\n\nExample of retrieving PDB annotations from the SIFTS project\n[(see SiftsDataDemo.java)](src/main/java/edu/sdsc/mmtf/spark/datasets/demos/SiftsDataDemo.java).\n\n```\nspark-submit --class edu.sdsc.mmtf.spark.datasets.demos.SiftsDataDemo INSTALL_DIRECTORY/mmtf-spark/target/mmtf-spark-0.3.0-SNAPSHOT.jar\n```\n\nExample with command line arguments. This example reads the PDB files\nin an input directory (recursively) and creates an MMTF-Hadoop Sequence file directory\n[(see PdbToMmtfFull.java)](src/main/java/edu/sdsc/mmtf/spark/io/demos/PdbToMmtfFull.java). \n\n```\nspark-submit --class edu.sdsc.mmtf.spark.io.demos.PdbToMmtfFull  INSTALL_DIRECTORY/mmtf-spark/target/mmtf-spark-0.2.0-SNAPSHOT.jar PDB_FILE_DIRECTORY MMTF_HADOOP_FILE_DIRECTORY\n```\n\n\n# How to Cite this Work\n\nBradley AR, Rose AS, Pavelka A, Valasatava Y, Duarte JM, Prlić A, Rose PW (2017) MMTF - an efficient file format for the transmission, visualization, and analysis of macromolecular structures. PLOS Computational Biology 13(6): e1005575. doi: [10.1371/journal.pcbi.1005575](https://doi.org/10.1371/journal.pcbi.1005575)\n\nValasatava Y, Bradley AR, Rose AS, Duarte JM, Prlić A, Rose PW (2017) Towards an efficient compression of 3D coordinates of macromolecular structures. PLOS ONE 12(3): e0174846. doi: [10.1371/journal.pone.01748464](https://doi.org/10.1371/journal.pone.0174846)\n\nRose AS, Bradley AR, Valasatava Y, Duarte JM, Prlić A, Rose PW (2018) NGL viewer: web-based molecular graphics for large complexes, Bioinformatics, bty419. doi: [10.1093/bioinformatics/bty419](https://doi.org/10.1093/bioinformatics/bty419)\n\nRose AS, Bradley AR, Valasatava Y, Duarte JM, Prlić A, Rose PW (2016) Web-based molecular graphics for large complexes. In Proceedings of the 21st International Conference on Web3D Technology (Web3D '16). ACM, New York, NY, USA, 185-186. doi: [10.1145/2945292.2945324](https://doi.org/10.1145/2945292.2945324)\n\n# Funding\nThis project is supported by the National Cancer Institute of the National Institutes of Health under Award Number U01CA198942. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsbl-sdsc%2Fmmtf-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsbl-sdsc%2Fmmtf-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsbl-sdsc%2Fmmtf-spark/lists"}