{"id":26674373,"url":"https://github.com/multiomics-analytics-group/retrieve_info_mgnifyapi","last_synced_at":"2025-07-28T16:07:48.969Z","repository":{"id":215521929,"uuid":"724072090","full_name":"Multiomics-Analytics-Group/Retrieve_info_MGnifyAPI","owner":"Multiomics-Analytics-Group","description":"Scripts to get information from MGnify programmatically using its API","archived":false,"fork":false,"pushed_at":"2025-03-02T18:23:36.000Z","size":2122,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-26T02:18:55.226Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Multiomics-Analytics-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-27T10:41:43.000Z","updated_at":"2024-02-13T18:01:44.000Z","dependencies_parsed_at":"2024-01-16T23:16:47.965Z","dependency_job_id":"d6d64282-99e8-451d-b4da-b581b307af25","html_url":"https://github.com/Multiomics-Analytics-Group/Retrieve_info_MGnifyAPI","commit_stats":null,"previous_names":["multiomics-analytics-group/retrieve_info_mgnifyapi"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Multiomics-Analytics-Group/Retrieve_info_MGnifyAPI","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Multiomics-Analytics-Group%2FRetrieve_info_MGnifyAPI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Multiomics-Analytics-Group%2FRetrieve_info_MGnifyAPI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Multiomics-Analytics-Group%2FRetrieve_info_MGnifyAPI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Multiomics-Analytics-Group%2FRetrieve_info_MGnifyAPI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Multiomics-Analytics-Group","download_url":"https://codeload.github.com/Multiomics-Analytics-Group/Retrieve_info_MGnifyAPI/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Multiomics-Analytics-Group%2FRetrieve_info_MGnifyAPI/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267543275,"owners_count":24104538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-28T02:00:09.689Z","response_time":68,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-26T02:19:00.173Z","updated_at":"2025-07-28T16:07:48.925Z","avatar_url":"https://github.com/Multiomics-Analytics-Group.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MGnify API Data Retrieval Scripts\nThe repository is designed to retrieve information and results from MGnify studies corresponding to a specific biome.\n\n---\n\n## Table of contents\n- [Description](#description) - Overview of the project's purpose and goals\n- [Getting started](#getting-started) - Instructions on how to begin with this project\n- [Repository structure](#repository_structure) - A layout of the repository's architecture, describing the purpose of each file or directory\n- [Next steps](#next-steps) - Next challanges and improvements for the repository \n- [Authors](#authors) - List of contributors to the project\n- [Acknowledgments](#acknowledgement) - Credits and thanks to those who helped with the project\n\n## Description \u003ca name = \"description\"\u003e\u003c/a\u003e\nThe repository is designed to gather information and results from MGnify.\nThis API allows for the exploration of studies originating from various biomes around the world.\nThe codes provided by this repository enable us to acquire the necessary identifiers to obtain metadata and FASTQ files required for multi-omic analyses.\nThe term \"multi-omic analysis\" refers to the [repository](https://github.com/biosustain/dsp_nf-metagenomics) developed in Nextflow, which carries out preprocessing and processing of bioinformatic data.\n\n\n## Getting started \u003ca name = \"getting-started\"\u003e\u003c/a\u003e\nProceeding with the steps to gather the data\n0. set up a Python virtual environment and install required libraries (specified in the `Pipfile` or `requirements.txt` file).\n1. `Functions_getInfo_MGnify_studies_analyses.py`: retrieves a summary of MGnify studies and analyses for a given biome and data type (amplicon, shotgun metagenomics, metatranscriptomic, or assembly). The attributes of the api requests can be modified in the script (example running \n`Scripts/example_main_get_summary_studies_and_analyses.py`).\n2. `Functions_get_results_from_MGnifystudy.py`: obtains abundance and functional tables, as well as other results for a MGnify study (example running `Scripts/example_main_get_results_from_MGnifystudy.py`).\n3. `Functions_get_samplesMetadata_from_MGnifystudy.py`: obtains metadata for the samples of a MGnify study (example running `Scripts/example_main_get_samplesMetadata_from_MGnifystudy.py`).\n4. `get_fastq_from_list_ids.py`: obtains FASTQ files from MGnify studies.\n\nModify the scripts to change the biome of interest, the data types to include, the desired study, and other attributes from the get requests to the MGnify API.\n\n\nConsider to use [Pipenv](https://pipenv.pypa.io/en/latest/) to create a Python virtual environment, which allows the management of python libraries and their dependencies.\nYou can find a detailed guide on how to use pipenv [here](https://realpython.com/pipenv-guide/).\nEach Pipenv virtual environment has a `Pipfile` with the names and versions of libraries installed in the virtual environment, and a `Pipfile.lock`, a JSON file that contains versions of libraries and their dependencies.\n\nTo create a Python virtual environment with libraries and dependencies required for this project, you should clone this GitHub repository, open a terminal, move to the folder containing this repository, and run the following commands:\n\n```bash\n# Install pipenv\n$ pip install pipenv\n\n# Create the Python virtual environment \n$ pipenv install\n\n# Activate the Python virtual environment \n$ pipenv shell\n```\n\nIf you are using an Azure VM, keep in mind the following considerations:\n1. Pipenv might not work as its installation path may not be included in the system's PATH.\n2. The Python version might not be up-to-date for installing a Pip package.\n\n```bash\n# check where it is located\n$ pip show pipenv | grep Location\n\n# open the shell configuration file\n$ nano /home/azureuser/.bashrc\n\n# add this line at the end of the file and save the changes\n$ export PATH=\"$PATH:/home/azureuser/.local/bin\"\n\n# reload the shell configuration file\n$ source /home/azureuser/.bashrc\n\n# verify the version which should now be recognized\n$ pipenv --version\n```\nIn case your python version is outdated, it should be updated to version 3.11\n\n```bash\n# add a PPA  \n$ sudo add-apt-repository ppa:deadsnakes/ppa\n$ sudo apt update\n\n# install the new version to be able to activate the pipenv environment\nsudo apt install python3.11\n```  \n\nAlternatively, you can create a conda virtual environment with the required libraries using the `requirements.txt` file.\nTo do this, you should clone this GitHub repository, open a terminal, move to the folder containing this repository, and run the following commands:\n\n```bash\n# Create the conda virtual environment\n$ conda create --name retrieve_info_MGnifyAPI python=3.11\n\n# Activate the conda virtual environment\n$ conda activate retrieve_info_MGnifyAPI\n\n# Install pip\n$ conda install pip\n\n# Install libraries and dependencies with pip \n$ pip install -r requirements.txt\n```\n\n## Obtain raw result files for a MGnify study\nThe `bulk_download` option of the `mg-toolkit` Python package provides a command line interface to download raw result files for a MGnify study.\nFor instance, to download the raw results files for the taxonomic analysis of the study [MGYS00001392](https://www.ebi.ac.uk/metagenomics/studies/MGYS00001392) obtained with the pipeline 5 or greater, you can run the following command:\n\n```bash\n$ mg-toolkit bulk_download -a MGYS00001392 --result_group taxonomic_analysis_unite -o Output/\n```\n\nYou can find more information about this package and additional options [here](https://pypi.org/project/mg-toolkit/). \n\n## Repository structure \u003ca name = \"repository_structure\"\u003e\u003c/a\u003e\nThe main files and directories of this repository are:\n\n|File|Description|\n|:-:|---|\n|[Scripts/](Scripts/)|Folder with Python scripts to to get information and results from MGnify studies for a given biome and study type|\n|[Output/](Results/)|Folder to save the resulting files|\n|[Pipfile](Pipfile)|File with names and versions of packages installed in the virtual environment|\n|[requeriments.txt](requeriments.txt)|File with names and versions of packages installed in the virtual environment|\n|[Pipfile.lock](Pipfile.lock)|Json file that contains versions of packages, and dependencies required for each package|\n\n\n## Next steps \u003ca name = \"next-steps\"\u003e\u003c/a\u003e\n1. rewrite the python scripts using if __name__=='__main__'  \n2. make the functions shorter \n3. create a new python script/function to upload FASTQ files in Azure container given a credentials file\n4. create a Nextflow pipeline able to parallelise the download of FASTQ files\n\n\n## Authors \u003ca name = \"authors\"\u003e\u003c/a\u003e\nContact me at [sayalaruano](https://github.com/sayalaruano) or [marcor@dtu.dk](https://github.com/marcoreverenna) for more detail or explanations.\n\n\n## **Further details**\nThe [MGnify documentation](https://docs.mgnify.org/src/docs/api.html) provides more information about the API. Also, you can browse the API endpoints interactively [here](https://www.ebi.ac.uk/metagenomics/api/latest/).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmultiomics-analytics-group%2Fretrieve_info_mgnifyapi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmultiomics-analytics-group%2Fretrieve_info_mgnifyapi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmultiomics-analytics-group%2Fretrieve_info_mgnifyapi/lists"}