{"id":17204259,"url":"https://github.com/andrew-s-rosen/covalent-hpc-plugin","last_synced_at":"2025-04-13T21:21:10.045Z","repository":{"id":185930342,"uuid":"674084724","full_name":"Andrew-S-Rosen/covalent-hpc-plugin","owner":"Andrew-S-Rosen","description":"Covalent plugin for HPC batch job schedulers (e.g. Slurm, PBS, LSF, Flux, Cobalt) built around PSI/J","archived":false,"fork":false,"pushed_at":"2024-07-08T22:11:56.000Z","size":257,"stargazers_count":9,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2024-07-09T03:07:24.529Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Andrew-S-Rosen.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-03T05:33:17.000Z","updated_at":"2024-07-09T03:07:36.579Z","dependencies_parsed_at":"2023-10-16T10:34:09.462Z","dependency_job_id":"0e2dafaa-7cf7-48c6-b1c9-43bbf26edc58","html_url":"https://github.com/Andrew-S-Rosen/covalent-hpc-plugin","commit_stats":null,"previous_names":["arosen93/covalent-hpc-plugin","quantum-accelerators/covalent-hpc-plugin","andrew-s-rosen/covalent-hpc-plugin"],"tags_count":8,"template":false,"template_full_name":"AgnostiqHQ/covalent-executor-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Andrew-S-Rosen%2Fcovalent-hpc-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Andrew-S-Rosen%2Fcovalent-hpc-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Andrew-S-Rosen%2Fcovalent-hpc-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Andrew-S-Rosen%2Fcovalent-hpc-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Andrew-S-Rosen","download_url":"https://codeload.github.com/Andrew-S-Rosen/covalent-hpc-plugin/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248783271,"owners_count":21160899,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T02:21:13.310Z","updated_at":"2025-04-13T21:21:10.010Z","avatar_url":"https://github.com/Andrew-S-Rosen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u0026nbsp;\n\n\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/AgnostiqHQ/covalent/master/doc/source/_static/covalent_readme_banner.svg\" width=150%\u003e\n\n\u003c/div\u003e\n\n## Covalent HPC Plugin\n\n[Covalent](https://www.covalent.xyz) is a Pythonic workflow tool used to execute tasks on advanced computing hardware. This executor plugin uses [PSI/J](https://exaworks.org/psij-python/index.html) to allow Covalent to seamlessly interface with a variety of common high-performance computing job schedulers and pilot systems (e.g. [Slurm](https://slurm.schedmd.com/), [PBS](https://www.openpbs.org/), [LSF](https://www.ibm.com/products/hpc-workload-management), [Flux](https://flux-framework.readthedocs.io/en/latest/), [Cobalt](https://git.cels.anl.gov/aig-public/cobalt), [RADICAL-Pilot](https://radicalpilot.readthedocs.io/en/stable/)). For workflows to be deployable, users must have SSH access to the login node, access to the job scheduler, and write access to the remote filesystem.\n\n## Installation\n\n### Server Environment\n\nTo use this plugin with Covalent, simply install it using `pip` in whatever Python environment you use to run the Covalent server (your local machine by default):\n\n```\npip install covalent-hpc-plugin\n```\n\nRun the following in Python to have Covalent automatically register the plugin:\n\n```python\nimport covalent\n```\n\n### HPC Environment\n\nAdditionally, on the remote machine(s) where you plan to execute Covalent workflows with this plugin, ensure that the remote Python environment has Covalent and PSI/J installed:\n\n```\npip install covalent psij-python\n```\n\nNote that the Python major and minor version numbers on both the local and remote machines must match to ensure reliable (un)pickling of the various objects.\n\n## Usage\n\n### Default Configuration Parameters\n\nBy default, when you install the `covalent-hpc-plugin` and run `import covalent` for the first time, your Covalent [configuration file](https://docs.covalent.xyz/docs/user-documentation/how-to/customization/) (found at `~/.config/covalent/covalent.conf` by default) will automatically be updated to include the following sections. These are not all of the available parameters but are simply the default values.\n\n```\n[executors.hpc]\naddress = \"\"\nusername = \"\"\nssh_key_file = \"~/.ssh/id_rsa\"\ninstance = \"slurm\"\nlauncher = \"single\"\ninherit_environment = true\npre_launch_cmds = []\npost_launch_cmds = []\nshebang = \"#!/bin/bash\"\nremote_python_exe = \"python\"\nremote_workdir = \"~/covalent-workdir\"\ncreate_unique_workdir = false\ncache_dir = \"~/.cache/covalent\"\npoll_freq = 60\n\n[executors.hpc.environment]\n\n[executors.hpc.resource_spec_kwargs]\nnode_count = 1\nprocesses_per_node = 1\ngpu_cores_per_process = 0\n\n[executors.hpc.job_attributes_kwargs]\nduration = 10\n```\n\nYou can modify various parameters in the Covalent config file as-needed to better suit your needs, such as the `address` of the remote machine, the `username` to use when logging in, the `ssh_key_file` to use for authentication, the type of job scheduler (`instance`), and much more. Note that PSI/J is a common interface to many common job schedulers, so you only need to toggle the `instance` to switch between job schedulers.\n\nA full description of the various input parameters are described in the docstrings of the `HPCExecutor` class, reproduced below:\n\nhttps://github.com/Quantum-Accelerators/covalent-hpc-plugin/blob/25785d0c546851c4b11e5c227f2e7aebb12aba8c/covalent_hpc_plugin/hpc.py#L115-L159\n\n### Defining Resource Specifications and Job Attributes\n\nTwo of the most important sets of parameters are `resource_spec_kwargs` and `job_attributes_kwargs`, which specify the resources required for the job (e.g. number of nodes, number of processes per node, etc.) and the job attributes (e.g. duration, queue name, etc.), respectively.\n\n1. `resource_spec_kwargs` is a dictionary of keyword arguments passed to PSI/J's [`ResourceSpecV1`](https://exaworks.org/psij-python/docs/v/0.9.0/.generated/psij.html#psij.resource_spec.ResourceSpecV1) class\n2. `job_attributes_kwargs` is a dictionary of keyword arguments passed to PSI/J's [`JobAttributes`](https://exaworks.org/psij-python/docs/v/0.9.0/.generated/psij.html#psij.JobAttributes) class.\n\nThe allowed types are listed [here](https://github.com/Quantum-Accelerators/covalent-hpc-plugin/blob/48611a9d7c5c51403ac69d4b446f35e7748bd479/covalent_hpc_plugin/hpc.py#L85-L111).\n\n### Using the Plugin in a Workflow: Approach 1\n\nWith the configuration file appropriately set up, one can run a workflow on the HPC machine as follows:\n\n```python\nimport covalent as ct\n\n@ct.electron(executor=\"HPCExecutor\")\ndef add(a, b):\n    return a + b\n\n@ct.lattice\ndef workflow(a, b):\n    return add(a, b)\n\n\ndispatch_id = ct.dispatch(workflow)(1, 2)\nresult = ct.get_result(dispatch_id)\n```\n\n### Using the Plugin in a Workflow: Approach 2\n\nIf you wish to modify the various parameters within your Python script rather than solely relying on the the Covalent configuration file, it is possible to do that as well by instantiating a custom instance of the `HPCExecutor` class. An example with some commonly used parameters is shown below. By default, any parameters not specified in the `HPCExecutor` will be inherited from the configuration file.\n\n```python\nimport covalent as ct\n\nexecutor = ct.executor.HPCExecutor(\n    address=\"coolmachine.university.edu\",\n    username=\"UserName\",\n    ssh_key_file=\"~/.ssh/id_rsa\",\n    instance=\"slurm\",\n    remote_conda_env=\"myenv\",\n    environment={\"HELLO\": \"WORLD\"},\n    resource_spec_kwargs={\n        \"node_count\": 2,\n        \"processes_per_node\": 24\n    },\n    job_attributes_kwargs={\n        \"duration\": 30, # minutes\n        \"queue_name\": \"debug\",\n        \"project_name\": \"AccountName\",\n    },\n    launcher=\"single\",\n    remote_workdir=\"~/covalent-workdir\",\n)\n\n@ct.electron(executor=executor)\ndef add(a, b):\n    return a + b\n\n@ct.lattice\ndef workflow(a, b):\n    return add(a, b)\n\n\ndispatch_id = ct.dispatch(workflow)(1, 2)\nresult = ct.get_result(dispatch_id)\n```\n\n### Working Example: Perlmutter\n\nThe following is a minimal working example to submit a Covalent job on NERSC's [Perlmutter](https://docs.nersc.gov/systems/perlmutter/) machine. It assumes that you have used the [sshproxy](https://docs.nersc.gov/connect/mfa/#sshproxy) utility to generate a certificate file in order to circumvent the need for multi-factor authentication for each login.\n\n```python\nimport covalent as ct\n\nexecutor = ct.executor.HPCExecutor(\n    address=\"perlmutter-p1.nersc.gov\",\n    username=\"UserName\",\n    ssh_key_file=\"~/.ssh/nersc\",\n    cert_file=\"~/.ssh/nersc-cert.pub\",\n    remote_conda_env=\"myenv\",\n    job_attributes_kwargs={\n        \"project_name\": \"ProjectName\",\n        \"custom_attributes\": {\"slurm.constraint\": \"cpu\", \"slurm.qos\": \"debug\"},\n    },\n)\n\n@ct.electron(executor=executor)\ndef add(a, b):\n    return a + b\n\n@ct.lattice\ndef workflow(a, b):\n    return add(a, b)\n\n\ndispatch_id = ct.dispatch(workflow)(1, 2)\nresult = ct.get_result(dispatch_id)\n```\n\n## Troubleshooting\n\nThe most common cause of issues is related to the job scheduler details (i.e. the `resource_spec_kwargs` and the `job_attributes_kwargs`). If your job fails on the remote machine, set `cleanup=False` and then check the files left behind in the working directory as well as the `~/.psij` directory for a history and various log files associated with your attempted job submissions.\n\n## Release Notes\n\nRelease notes are available in the [Changelog](CHANGELOG.md).\n\n## Credit\n\nThis plugin was developed by [Andrew S. Rosen](https://github.com/Andrew-S-Rosen), building off of prior work by the Agnostiq team on the [covalent-slurm-plugin](https://github.com/AgnostiqHQ/covalent-slurm-plugin).\n\nIf you use this plugin, be sure to cite Covalent as follows:\n\n\u003e W. J. Cunningham, S. K. Radha, F. Hasan, J. Kanem, S. W. Neagle, and S. Sanand.\n\u003e _Covalent._ Zenodo, 2022. https://doi.org/10.5281/zenodo.5903364\n\n## License\n\nCovalent is licensed under the Apache 2.0 License. Covalent may be distributed under other licenses upon request. See the [LICENSE](https://github.com/Quantum-Accelerators/covalent-hpc-plugin/blob/develop/LICENSE) file or contact the [support team](mailto:support@agnostiq.ai) for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrew-s-rosen%2Fcovalent-hpc-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandrew-s-rosen%2Fcovalent-hpc-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrew-s-rosen%2Fcovalent-hpc-plugin/lists"}