{"id":25895569,"url":"https://github.com/diamondlightsource/cryoem-services","last_synced_at":"2025-03-02T22:32:03.001Z","repository":{"id":212471591,"uuid":"728642671","full_name":"DiamondLightSource/cryoem-services","owner":"DiamondLightSource","description":"Services and configuration for cryo-EM pipelines","archived":false,"fork":false,"pushed_at":"2024-05-21T14:44:13.000Z","size":251,"stargazers_count":1,"open_issues_count":7,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-22T14:34:33.994Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DiamondLightSource.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-07T11:34:01.000Z","updated_at":"2024-08-12T10:00:20.044Z","dependencies_parsed_at":"2023-12-19T15:09:02.071Z","dependency_job_id":"97b77c55-43ea-4517-adbe-87edde33f85b","html_url":"https://github.com/DiamondLightSource/cryoem-services","commit_stats":null,"previous_names":["diamondlightsource/cryoem-services"],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fcryoem-services","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fcryoem-services/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fcryoem-services/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fcryoem-services/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DiamondLightSource","download_url":"https://codeload.github.com/DiamondLightSource/cryoem-services/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241582515,"owners_count":19985845,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-02T22:32:02.190Z","updated_at":"2025-03-02T22:32:02.991Z","avatar_url":"https://github.com/DiamondLightSource.png","language":"Python","readme":"# cryoem-services\n\n[![PyPI release](https://img.shields.io/pypi/v/cryoemservices.svg)](https://pypi.python.org/pypi/cryoemservices)\n[![Supported Python versions](https://img.shields.io/pypi/pyversions/cryoemservices.svg)](https://pypi.python.org/pypi/cryoemservices)\n[![Test coverage](https://codecov.io/gh/DiamondLightSource/cryoem-services/branch/main/graph/badge.svg)](https://codecov.io/gh/DiamondLightSource/cryoem-services)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)\n\nServices and configuration for cryo-EM pipelines.\n\nThis package consists of a number of services to process cryo-EM micrographs,\nboth for single particle analysis and tomography,\nusing a range of commonly used cryo-EM processing software.\nThese services can be run independently to process data,\nor as part of a wider structure for performing live analysis during microscope collection.\nFor live analysis, this package integrates with a package\nfor transferring and monitoring collected data,\n[Murfey](https://github.com/DiamondLightSource/python-murfey),\nand a database for storing processing outcomes,\n[ISPyB](https://github.com/DiamondLightSource/ispyb-database).\n\nTo run these services the software executables being called must be installed.\nThese do not come with this package.\n\n# Tomography processing\n\nThe tomography processing pipeline consists of:\n\n- Motion correction\n- CTF estimation\n- Tomogram alignment\n- Tomogram denoising using [Topaz](http://topaz-em.readthedocs.io)\n- Segmentation using [membrain-seg](https://github.com/teamtomo/membrain-seg)\n\nThe results of this processing can be opened and continued using\n[Relion 5.0](https://relion.readthedocs.io).\n\n# Single particle analysis\n\nThe single particle analysis pipeline produces a project\nthat can be opened and continued using\n[CCP-EM doppio](https://www.ccpem.ac.uk/docs/doppio/user_guide.html)\nor [Relion](https://relion.readthedocs.io).\n\nThe processing pipeline consists of:\n\n- Motion correction\n- CTF estimation\n- Particle picking\n- (Optionally) Ice thickness estimation\n- Particle extraction and rebatching\n- 2D classification using Relion\n- Automated 2D class selection using Relion\n- 3D classification using Relion\n- 3D Refinement and post-processing\n- BFactor estimation by refinement with varying particle count\n\n# Services currently available\n\nThe following services are provided for running the pipelines:\n\n- Utility services:\n  - **ClusterSubmission**: Submits wrapper scripts to an HPC cluster\n  - **EMISPyB**: Inserts results into an ISPyB database\n  - **Images**: Creates thumbnail images for viewing processing outcomes\n  - **NodeCreator**: Creates Relion project files for the services run\n  - **ProcessRecipe**: Converts recipes into messages suitable for processing services\n- Processing services:\n  - **BFactor**: Performs the setup for 3D refinement with varying particle count\n  - **CrYOLO**: Particle picking on micrographs using [crYOLO](https://cryolo.readthedocs.io)\n  - **CTFFind**: CTF estimation on micrographs using [CTFFIND4](https://grigoriefflab.umassmed.edu/ctffind4)\n  - **DenoiseSlurm**: Tomogram denoising, submitted to a slurm HPC cluster, using [Topaz](http://topaz-em.readthedocs.io)\n  - **Extract**: Extracts picked particles from micrographs\n  - **ExtractClass**: Extracts particles from a given 3D class\n  - **IceBreaker**: Ice thickness estimation with [IceBreaker](https://github.com/DiamondLightSource/python-icebreaker)\n  - **MembrainSeg**: Tomogram segmentation, submitted to a slurm HPC cluster, using [membrain-seg](https://github.com/teamtomo/membrain-seg)\n  - **MotionCorr**: Motion correction of micrographs using [MotionCor2](http://emcore.ucsf.edu/ucsf-software) or [Relion](https://relion.readthedocs.io), optionally submitted to a slurm HPC cluster\n  - **PostProcess**: Post-processing of 3D refinements using [Relion](https://relion.readthedocs.io)\n  - **SelectClasses**: Runs automated 2D class selection using [Relion](https://relion.readthedocs.io) and re-batches the particles from these classes\n  - **SelectParticles**: Creates files listing batches of extracted particles\n  - **TomoAlign**: Tomogram reconstruction from a list of micrographs using [imod](https://bio3d.colorado.edu/imod) and [AreTomo2](https://github.com/czimaginginstitute/AreTomo2)\n  - **TomoAlignSlurm**: Tomogram alignment processing submitted to a slurm HPC cluster\n\nThere are also three wrapper scripts that can be run on an HPC cluster using the ClusterSubmission service.\nThese perform 2D classification, 3D classification and 3D refinement\nusing [Relion](https://relion.readthedocs.io).\n\n# Running services\n\nThe services in this package are run using\n[zocalo](https://github.com/DiamondLightSource/python-zocalo)\nand [python-workflows](https://github.com/DiamondLightSource/python-workflows).\nThey consume messages off a [RabbitMQ](https://www.rabbitmq.com/)\ninstance and processing happens in sequences defined by the recipes in the `recipes` folder.\nTo start a service run the `cryoemservices.service` command and specify the service name.\nFor example, to start a motion correction service:\n\n```bash\n$ cryoemservices.service -s MotionCorr -c config_file.yaml\n```\n\nThe configuration file should contain the following:\n\n```yaml\nrabbitmq_credentials: \u003cfile with connection credentials to rabbitmq\u003e\nrecipe_directory: \u003cdirectory containing the recipes to run\u003e\nispyb_credentials: \u003c(Optionally) file with credentials for an ispyb database\u003e\nslurm_credentials:\n  default: \u003c(Optionally) credentials for a slurm RestAPI\u003e\ngraylog_host: \u003c(Optionally) the name of a graylog instance\u003e\ngraylog_port: \u003c(Optionally) the port used by graylog\u003e\n```\n\nOnce started, these services will initialise and then wait for messages to be sent to them.\nMessages are sent through a message broker,\ncurrently [RabbitMQ](http://www.rabbitmq.com) is supported using pika transport in `python-workflows`.\nIndividual processing stages can be run by sending a dictionary of the parameters,\nbut the processing pipelines are designed to run through recipes.\n\nA recipe is a specication of a series of steps to carry out,\nand how these steps interact with each other.\nRecipes for the current processing pipelines are provided in the `recipes` folder.\n\nTo run a recipe in python a dictionary needs to be provided consisting of\nthe recipe name and the parameters expected by the recipe.\nThe following snippet shows an example of the setup needed.\nThis will send a message to a running **ProcessRecipe** service which\nprepares the recipe for the processing services.\n\n```python\nimport workflows.transport.pika_transport as pt\n\nexample_message = {\n    \"recipes\": [\"em-tomo-align\"],\n    \"parameters\": {\n        \"path_pattern\": \"micrograph_*.mrc\",\n        \"pix_size\": \"1\",\n        ...\n    },\n}\n\ntransport = pt.PikaTransport()\ntransport.connect()\ntransport.send(\"processing_recipe\", example_message)\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiamondlightsource%2Fcryoem-services","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdiamondlightsource%2Fcryoem-services","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiamondlightsource%2Fcryoem-services/lists"}