{"id":27248237,"url":"https://github.com/rsanchezgarc/deepemhancer","last_synced_at":"2025-08-09T06:21:30.056Z","repository":{"id":65330406,"uuid":"274380955","full_name":"rsanchezgarc/deepEMhancer","owner":"rsanchezgarc","description":"Deep learning for cryo-EM maps post-processing","archived":false,"fork":false,"pushed_at":"2024-12-03T14:45:11.000Z","size":1932,"stargazers_count":53,"open_issues_count":6,"forks_count":8,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-10T23:08:08.551Z","etag":null,"topics":["cryo-em","deep-learning"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rsanchezgarc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-23T10:50:15.000Z","updated_at":"2025-04-08T00:57:27.000Z","dependencies_parsed_at":"2023-02-10T05:16:04.308Z","dependency_job_id":"f6742c7e-b606-4eab-8dba-2bcd96dcea49","html_url":"https://github.com/rsanchezgarc/deepEMhancer","commit_stats":{"total_commits":67,"total_committers":2,"mean_commits":33.5,"dds":0.02985074626865669,"last_synced_commit":"32dd3787b59807511ffcfdef10a357bd1e0e9d97"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rsanchezgarc%2FdeepEMhancer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rsanchezgarc%2FdeepEMhancer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rsanchezgarc%2FdeepEMhancer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rsanchezgarc%2FdeepEMhancer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rsanchezgarc","download_url":"https://codeload.github.com/rsanchezgarc/deepEMhancer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248312143,"owners_count":21082638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cryo-em","deep-learning"],"created_at":"2025-04-10T23:08:13.209Z","updated_at":"2025-04-10T23:08:13.692Z","avatar_url":"https://github.com/rsanchezgarc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deep cryo-EM Map Enhancer (DeepEMhancer)\n**DeepEMhancer** is a python package designed to perform post-processing of\ncryo-EM maps as described in \"\u003ca href=https://doi.org/10.1038/s42003-021-02399-1 \u003eDeepEMhancer:\na deep learning solution for cryo-EM volume post-processing\u003c/a\u003e\", by Sanchez-Garcia et al, 2021.\u003cbr\u003e\u003e\nDeepEMhancer is a deep learning model trained on pairs of experimental volumes and atomic model-corrected volumes that is \nable to obtain post-processed maps using as input raw volumes, preferably half maps. Please notice that post-translational \nmodifications and ligands were not included in the traning set and consequently, results for these features could be inaccurate.\u003cbr\u003e\n\nSimply speaking, DeepEMhancer performs a non-linear post-processing of cryo-EM maps that produces two main effects:\n1) Local sharpening-like post-processing.\n2) Automatic masking/denoising of cryo-EM maps.\n\n\n## Table of Contents  \n- [INSTALLATION](#installation)  \n- [USAGE GUIDE](#usage-guide)  \n- [EXAMPLES](#examples)  \n- [TROUBLESHOOTING](#Troubleshooting)  \n\u003cbr\u003e\u003cbr\u003e\nTo get a complete description of usage, execute\n`deepemhancer -h`\n## INSTALLATION:\n\n- [Requirements](#requirements)\n- [Install from source option](#install-from-source-option)\n- [Install from Anaconda cloud](#install-from-anaconda-cloud)\n- [Alternative installation for old versions](#alternative-installation-for-old-versions)\n- [No conda installation](#no-conda-installation)\n\n#### Requirements\nDeepEMhancer has been tested on Linux systems.\nCurrent version employs Tensorflow version 2.10 that requires CUDA 11. Our installation recipe will\nautomatically install, among other packages, Tensorflow=2.10 and CUDA 11.8. You can always install the older version\nthat used Tensorflow 1.14 and cuda 10.2. If your drivers are not compatible with any of the configurations\nand you cannot update them, you can try to compile tensorflow-gpu=2.10 using your library settings instead of \ninstalling it using conda.\n\n### Install from source option:\nThe best option to keep you updated. Currently, it uses Tensorflow version 2.10 and CUDA\u003e11.2. This option works\nwell in the new Nvidia GPUs (Ampere) but is should also work for the older versions. See other installation options \nif you want to install the original version of deepEMhancer that does NOT work in Nvidia Ampere GPUs \u003cbr\u003e\nRequires anaconda/miniconda, that can be obtained from \u003cref\u003ehttps://www.anaconda.com/products/individual\u003c/ref\u003e\n\u003cbr\u003e\u003cbr\u003eSteps:\n1) Clone this repository and cd inside\n```\ngit clone https://github.com/rsanchezgarc/deepEMhancer\ncd deepEMhancer\n```\n2) Create a conda environment with the required dependencies\n```\nconda env create -f deepEMhancer_env.yml  -n deepEMhancer_env\n```\n3) Activate the environment. You always need to activate the environment before executing deepEMhancer\n```\nconda activate deepEMhancer_env\n```\n4) Set CUDA LD_LIBRARY_PATH\n```\nconda env config vars set LD_LIBRARY_PATH=${CONDA_PREFIX}/lib/python3.9/site-packages/nvidia/cudnn/lib:${CONDA_PREFIX}/lib/:$LD_LIBRARY_PATH\nconda activate deepEMhancer_env #To make the changes effective\n```\n5) Install deepEMhancer\n```\npython -m pip install . --no-deps\n```\n6) Download our deep learning models\n```\ndeepemhancer --download\n```\n7) Ready! Do not forget to activate the environment for future usages. For a complete help use:\n```\ndeepemhancer -h\n```\n7) Optionally, you can remove the folder, since deepemhancer will be available anywhere once you activate the environment\n\n### Install from Anaconda cloud:\nRequires anaconda/miniconda, that can be obtained from \u003cref\u003ehhttps://www.anaconda.com/products/individual\u003c/ref\u003e.\n\n1) Create a fresh conda environment\n```\nconda create -n deepEMhancer_env python=3.10\n```\n2) Activate the environment. You always need to activate the environment before executing deepEMhancer\n```\nconda activate deepEMhancer_env\n```\n4) Install deepEMhancer\n```\nconda install deepEMhancer -c rsanchez1369 -c anaconda -c conda-forge\n```\n5) Download our deep learning models\n```\ndeepemhancer --download\n```\n6) Ready! Do not forget to activate the environment for future usages. For a complete help use:\n```\ndeepemhancer -h\n```\n\n### Alternative installation for old versions\nYou can install from the repository the older deepEMhancer version that works well in old GPUs. The process is identical\nto the option number 1, \"Install from source option\", except that you need to use the tag 0.14\n\u003cbr\u003e\u003cbr\u003eSteps:\n1) Clone the tag \"0.14\" of the repository repository and cd inside\n```\ngit clone --depth 1 --branch \"0.14\"  https://github.com/rsanchezgarc/deepEMhancer\ncd deepEMhancer\n```\nThe following steps are the same as in option 1.\u003cbr\u003e\n\n2) Create a conda environment with the required dependencies\n```\nconda env create -f deepEMhancer_env.yml  -n deepEMhancer_env\n```\n3) Activate the environment. You always need to activate the environment before executing deepEMhancer\n```\nconda activate deepEMhancer_env\n```\n4) Install deepEMhancer\n```\npython -m pip install . --no-deps\n```\n5) Download our deep learning models\n```\ndeepemhancer --download\n```\n6) Ready! Do not forget to activate the environment for future usages. For a complete help use:\n```\ndeepemhancer -h\n```\n7) Optionally, you can remove the folder, since deepemhancer will be available anywhere once you activate the environment\n\nIf your NVIDIA drivers are too old to work with CUDA \u003e 10.0, you can still install an even older version of Tensorflow. \nThis option is only recommended for people with old NVIDIA drivers that are still able to work with\nCUDA 10.0.\n\nThe steps for this option are exactly the same as above with the exception\nof step 2. Thus, instead of using  \"deepEMhancer_env.yml\" when creating the environment,\n```\nconda env create -f deepEMhancer_env.yml  -n deepEMhancer_env\n```\n\"alternative_installation/deepEMhancer_cud10.0.env.yml\" should be used.\n\n```\nconda env create -f alternative_installation/deepEMhancer_cud10.0.env.yml  -n deepEMhancer_env\n```\n \nIt has been reported that some problems with cudnn may occur when using this installation option. Please,\nsee [TROUBLESHOOTING](#Troubleshooting) section 2 for a proposed solution. \n\n\n### No conda installation\nOnly works for python\u003e3.7. Virtualenv is recommended to isolate packages.\n\n1) Clone this repository and cd inside\n```\ngit clone https://github.com/rsanchezgarc/deepEMhancer\ncd deepEMhancer\n```\n\n1.1. Optionally, create a virtual environment and activate it\n```\npip install virtualenv\nvirtualenv --system-site-packages -p python3 ./deepEMhancer_env\nsource ./deepEMhancer_env/bin/activate\n```\n2) Install deepEMhancer (using Tensorflow 1.14)\n- For CPU only use (expect running times ~ 24h)\n```\nDEEPEMHANCER_CPU_ONLY=1 python -m pip install .\n```\n- With GPU support\n  - Install CUDA 11.7 and cudnn 8.4. Make sure that they are in the LD_LIBRARY_PATH. If you are in a conda environment you probably want to add permanently the following `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/`\n  - install python packages\n```\npip install .\n```\n  - Check if GPUs are successfully detected.\n```\npython -c \"import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))\"\n```\n\nYou will see errors like ```Could not dlopen library 'libcudart.so.11.0'; dlerror``` if CUDA and/or cudnn \n(libcudnn.so.8) are not correctly installed or detected. On the contrary, if you see the message\n```I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device```,\nTensorflow has been able to recognize the GPUs.\n  \n5) Download our deep learning models\n```\ndeepemhancer --download\n```\n6) Ready! Do not forget to activate the environment, if used (step 1.1), for future usages. For a complete help use:\n```\ndeepemhancer -h\n```\n\n## Usage guide:\n##### About the input\n\nDeepEMhancer was trained using half-maps. Thus, as input, both half-maps are the preferred option (`deepemhancer -i half1.mrc -i2 half2.mrc`).\u003cbr\u003e \nFull maps obtained from refinement process (RELION auto-refine, cryoSPARC heterogenus refinement...) are equally valid.\u003cbr\u003e\nHowever, deepEMhancer will not work correctly if post-processed (masked, sharpened...) maps are provided as input \n(e.g. RELION postprocessing maps).\n\n##### About the deep learning models (-p option)\nWe provide 3 different deep learning models. The default one is the tightTarget model, that was trained using\ntightly masked volumes. This is the default option and all the statistics reported in the publication were obtained \nusing this model. Additionally, we provide a wideTarget model that was trained using less tightly masked maps. Finally,\nwe have also trained a model (highRes) using a subset of the maps with resolutions \u003c4 Å and fewer empty cubes.\u003cbr\u003e\nWe recommend our users to try the different options and choose the one that looks nicer to them. As a guidance, \nwe suggest to employ the highRes model for maps with overall resolution better than 4 Å and a moderate amount of bad\nresolution regions. HighRes solutions tend to be noisier than others, but also more enhanced. \nIf the overall resolution is worse, or the number of low resolution regions is high, the tightTarget\nmodel should do a good job. For cases in which both tightTarget and highRes produce too tightly masked solutions, possibly removing\nsome parts of the protein as if they were noise, we recommend to employ the wideTarget model.\n\n##### About the normaliztion\nOne of the key aspects to succesfully employ DeepEMhancer is the normalization of the input volumes.\nThe default normalization mode, mode 1, normalizes the data such that the statistics of the noise regions\nare forced to adopt a mean value of 0 and a standard deviation of 0.1.\u003cbr\u003e\nIf no flag is provided, deepEMhancer will try to automatically determine a spherical shell of noise from\nwhich the statistics of the noise are estimated. This automatic normalization tends to work well, although it\nmay fail in some cases. For example, hollow proteins or fiber proteins could cause problems.\u003cbr\u003e\nAlternatively, the user can manually determine the statistics of the noise and provide them to the program using\nthe flag `--noiseStats mean_noise std_noise`. One easy way to determine the noise statistics is to employ UCSF Chimera\nto crop a noise-only region of the map (`Volume Viewer\u003eFeatures\u003eRegion bounds`) and then compute the statistics (`Volume Viewer\u003eTools\u003eVolume Mean, SD, RSD`).\n\u003cbr\u003e\nFinally, as an alternative normalization, mode 2 normalizes the input using a binary mask (1 protein, 0 not protein).\nThis option was introduced to deal with masked maps (which are not suitable for default DeepEMhancer) and is not recommended\nwhen it is possible to employ normalization mode 1.\n\n##### About the batch size\nDeepEMhancer processes input maps by chunking them into smaller cubes that are sent to GPUs. Batch size parameter represent\nthe number of smaller cubes that are simultaneously processed by the GPUs. A typical value for an 8 GB GPU could be\u003cbr\u003e \n```--batch_size 6```. If OUT OF MEMORY error happens, try to lower batch_size, and if low GPU usage is observed (via nvidia-smi), try\nto increase it. Setting the environmental variable `TF_FORCE_GPU_ALLOW_GROWTH='true'` prior execution could also help to fix some GPU memory errors. When using multiple GPUs, for certain box sizes, there might happen a reported bug affecting the batch_size, please see [TROUBLESHOOTING](#Troubleshooting) error 3.\n\n\n\n## Examples\n- Download deep learning models\n```\ndeepemhancer --download\n```\n- Post-process input map path/to/inputVol.mrc and save it at path/to/outputVol.mrc using default  deep model (tightTarget)\n```\ndeepemhancer  -i path/to/inputVol.mrc -o  path/to/outputVol.mrc\n```\n- Post-process input map path/to/inputVol.mrc and save it at path/to/outputVol.mrc using softer deep model (wideTarget)\n```\ndeepemhancer -p wideTarget -i path/to/inputVol.mrc -o  path/to/outputVol.mrc\n```\n- Post-process input map path/to/inputVol.mrc and save it at path/to/outputVol.mrc using high resolution deep model\n```\ndeepemhancer -p highRes -i path/to/inputVol.mrc -o  path/to/outputVol.mrc\n```\n- Post-process input map path/to/inputVol.mrc and save it at path/to/outputVol.mrc using high resolution deep learning model located in path/to/deep/learningModel\n```\ndeepemhancer -p highRes  --deepLearningModelDir path/to/deep/learningModel -i path/to/inputVol.mrc -o  path/to/outputVol.mrc\n```\n- Post-process input map path/to/inputVol.mrc and save it at path/to/outputVol.mrc using high resolution  deep model and providing normalization information (mean\n    and standard deviation of the noise)\n```    \ndeepemhancer -p highRes -i path/to/inputVol.mrc -o  path/to/outputVol.mrc --noiseStats 0.12 0.03\n```\n\n\n## TROUBLESHOOTING\n\n1.  \n- Error: \n```\ntensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version\n```\n- Explanation: The drivers of your NVIDA GPU are too old.\u003cbr\u003e\n\n- Solution: Update your drivers to version \u003e= 418.39. Alternatively, for driver versions 410.48 to 418.39 you could\ntry the \"Alternative installation for Nvida-Driver 410\", the \"No conda installation\" or install yourself Tensorflow \nusing your CUDA setup. Although we have not tested it, deepEMhancer will probably also work with \nolder Tensorflow versions that require CUDA 9, so they could be also considered.\n\n2.\n- Error:\n```\n  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.\n\t [[{{node conv3d_1/convolution}}]]\n0 successful operations.\n0 derived errors ignored.\n```\nor\n\n```\nE tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR\n```\n- Explanation: This is a reported issue for Tensorflow, and many times occurs when getting out of GPU memory. \nIn other cases is related with an incompatibility between CUDA and cudnn versions.\u003cbr\u003e\n\n- Solution: \n  - If it is caused by memory constraints, set dynamic GPU allocation using the environment variable \nTF_FORCE_GPU_ALLOW_GROWTH='true'. E.g. \n```TF_FORCE_GPU_ALLOW_GROWTH='true' deepemhancer -i ~/tmp/useCase/EMD-0193.mrc -o ~/tmp/outVolDeepEMhancer/out.mrc\n```\n\n  - If it is caused by incompatibility between CUDA and cudnn, you should try to reinstall it ensuring that\n    CUDA and cudnn versions match and they are compatible with the Tensorflow version. We are using Tensorflow\n    version 14, but we think that older versions, compatible with CUDA 9 could also work.\n\n\n\n3. \n- Error:\n```\nF ./tensorflow/core/kernels/conv_2d_gpu.h:935] Non-OK-status: CudaLaunchKernel( SwapDimension1And2InTensor3UsingTiles\u003cT, kNumThreads, kTileSize, kTileSize, conjugate\u003e, total_tiles_count, kNumThreads, 0, d.stream(), input, input_dims, output) status: Internal: invalid configuration argument\n\nAborted (core dumped)\n```\n- Explanation: This is a reported issue for Tensorflow when using multiple GPUS and the number of subcubes or the batch size is not divisible by the number of GPUs\n- Solution: Use only one GPU (`-g 1`) and/or batch size 1 (`-b 1`)\n\n4. \n- Error:\nIt is not possible to download the models. \n- Explanation:\nSometimes our server may not be reachable due to network problems.\n- Solution:\nDownload them from https://zenodo.org/record/7432763\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frsanchezgarc%2Fdeepemhancer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frsanchezgarc%2Fdeepemhancer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frsanchezgarc%2Fdeepemhancer/lists"}