{"id":26055857,"url":"https://github.com/ncar/ncar-jobqueue","last_synced_at":"2025-04-11T03:21:49.188Z","repository":{"id":34290917,"uuid":"174445591","full_name":"NCAR/ncar-jobqueue","owner":"NCAR","description":"Utilities for configuring dask-jobqueue with appropriate settings for NCAR clusters","archived":false,"fork":false,"pushed_at":"2025-04-07T19:11:15.000Z","size":146,"stargazers_count":14,"open_issues_count":7,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-07T20:24:33.035Z","etag":null,"topics":["dask","dask-jobqueue"],"latest_commit_sha":null,"homepage":"https://jobqueue.dask.org/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NCAR.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-08T01:10:19.000Z","updated_at":"2025-03-20T20:22:57.000Z","dependencies_parsed_at":"2023-02-16T13:15:25.461Z","dependency_job_id":"5083f34f-5811-457d-bd5b-471f1607dc49","html_url":"https://github.com/NCAR/ncar-jobqueue","commit_stats":{"total_commits":109,"total_committers":6,"mean_commits":"18.166666666666668","dds":0.4862385321100917,"last_synced_commit":"23d7dbfc38369b928a6e1a44e1da98765576f189"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCAR%2Fncar-jobqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCAR%2Fncar-jobqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCAR%2Fncar-jobqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCAR%2Fncar-jobqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NCAR","download_url":"https://codeload.github.com/NCAR/ncar-jobqueue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248334290,"owners_count":21086367,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dask","dask-jobqueue"],"created_at":"2025-03-08T10:23:45.220Z","updated_at":"2025-04-11T03:21:49.166Z","avatar_url":"https://github.com/NCAR.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"| CI          | [![GitHub Workflow Status][github-ci-badge]][github-ci-link] [![Code Coverage Status][codecov-badge]][codecov-link] [![pre-commit.ci status][pre-commit.ci-badge]][pre-commit.ci-link] |\n| :---------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |\n| **Docs**    |                                                                     [![Documentation Status][rtd-badge]][rtd-link]                                                                     |\n| **Package** |                                                          [![Conda][conda-badge]][conda-link] [![PyPI][pypi-badge]][pypi-link]                                                          |\n| **License** |                                                                         [![License][license-badge]][repo-link]                                                                         |\n\n# ncar-jobqueue\n\n- [ncar-jobqueue](#ncar-jobqueue)\n  - [Installation](#installation)\n  - [Configuration](#configuration)\n  - [Usage](#usage)\n    - [Casper](#casper)\n    - [Derecho](#derecho)\n    - [Hobart](#hobart)\n    - [Izumi](#izumi)\n    - [Non-NCAR machines](#non-ncar-machines)\n\n`ncar-jobqueue` provides utilities for configuring [dask-jobqueue](https://dask-jobqueue.readthedocs.io/en/latest/) with appropriate default settings for NCAR's clusters.\n\nThe following compute servers are supported:\n\n- Casper (casper.hpc.ucar.edu)\n- Derecho (derecho.hpc.ucar.edu)\n- Hobart (hobart.cgd.ucar.edu)\n- Izumi (izumi.unified.ucar.edu)\n\n**CISL discourages the use of Derecho for Dask. Please use Casper instead unless you are sure you can properly utilize a significant portion of the CPU cores on a Derecho node (e.g., via [dask-mpi](https://mpi.dask.org/en/latest/)).**\n\n## Installation\n\nNCAR-jobqueue can be installed from PyPI with pip:\n\n```bash\npython -m pip install ncar-jobqueue\n```\n\nNCAR-jobqueue is also available from conda-forge for conda installations:\n\n```bash\nconda install -c conda-forge ncar-jobqueue\n```\n\n## Configuration\n\n`ncar-jobqueue` provides a custom configuration file with appropriate default settings for different clusters. This configuration file resides in `~/.config/dask/ncar-jobqueue.yaml`:\n\n\u003cdetails\u003e\n\u003csummary\u003encar-jobqueue.yaml\u003c/summary\u003e\n\n```yaml\ncasper:\n  pbs:\n    #project: XXXXXXXX\n    name: dask-worker-casper\n    cores: 1 # Total number of cores per job\n    memory: '4GiB' # Total amount of memory per job\n    processes: 1 # Number of Python processes per job\n    interface: ext # Network interface to use (high-speed ethernet)\n    walltime: '01:00:00'\n    resource-spec: select=1:ncpus=1:mem=4GB\n    queue: casper\n    log-directory: '/glade/derecho/scratch/${USER}/dask/casper/logs'\n    local-directory: '/glade/derecho/scratch/${USER}/dask/casper/local-dir'\n    job-extra: ['-r n']\n    env-extra: []\n    death-timeout: 60\n\nderecho:\n  pbs:\n    #project: XXXXXXXX\n    name: dask-worker-derecho\n    cores: 1 # Total number of cores per job\n    memory: '4GiB' # Total amount of memory per job\n    processes: 1 # Number of Python processes per job\n    interface: hsn0 # Network interface to use (Slingshot)\n    queue: develop\n    walltime: '01:00:00'\n    resource-spec: select=1:ncpus=128:mem=235GB\n    log-directory: '/glade/derecho/scratch/${USER}/dask/derecho/logs'\n    local-directory: '/glade/derecho/scratch/${USER}/dask/derecho/local-dir'\n    job-extra: ['-l job_priority=economy', '-r -n']\n    env-extra: []\n    death-timeout: 60\n\nhobart:\n  pbs:\n    name: dask-worker-hobart\n    cores: 10 # Total number of cores per job\n    memory: '96GB' # Total amount of memory per job\n    processes: 10 # Number of Python processes per job\n    # interface: null              # ib0 doesn't seem to be working on Hobart\n    queue: medium\n    walltime: '08:00:00'\n    resource-spec: nodes=1:ppn=48\n    log-directory: '/scratch/cluster/${USER}/dask/hobart/logs'\n    local-directory: '/scratch/cluster/${USER}/dask/hobart/local-dir'\n    job-extra: ['-r n']\n    env-extra: []\n    death-timeout: 60\n\nizumi:\n  pbs:\n    name: dask-worker-izumi\n    cores: 10 # Total number of cores per job\n    memory: '96GB' # Total amount of memory per job\n    processes: 10 # Number of Python processes per job\n    # interface: null              # ib0 doesn't seem to be working on Hobart\n    queue: medium\n    walltime: '08:00:00'\n    resource-spec: nodes=1:ppn=48\n    log-directory: '/scratch/cluster/${USER}/dask/izumi/logs'\n    local-directory: '/scratch/cluster/${USER}/dask/izumi/local-dir'\n    job-extra: ['-r n']\n    env-extra: []\n    death-timeout: 60\n```\n\n\u003c/details\u003e\n\n**Note:**\n\n- To configure a default project account that is used by `dask-jobqueue` when submitting batch jobs, uncomment the `project` key/line in `~/.config/dask/ncar-jobqueue.yaml` and set it to an appropriate value.\n\n## Usage\n\n**Note:**\n\n⚠️ Online documentation for `dask-jobqueue` is available [here][rtd-link]. ⚠️\n\n### Casper\n\n```python\n\u003e\u003e\u003e from ncar_jobqueue import NCARCluster\n\u003e\u003e\u003e from dask.distributed import Client\n\u003e\u003e\u003e cluster = NCARCluster(project='XXXXXXXX')\n\u003e\u003e\u003e cluster\nPBSCluster(0f23b4bf, 'tcp://xx.xxx.x.x:xxxx', workers=0, threads=0, memory=0 B)\n\u003e\u003e\u003e cluster.scale(jobs=2)\n\u003e\u003e\u003e client = Client(cluster)\n```\n\n### Derecho\n\n```python\n\u003e\u003e\u003e from ncar_jobqueue import NCARCluster\n\u003e\u003e\u003e from dask.distributed import Client\n\u003e\u003e\u003e cluster = NCARCluster(project='XXXXXXXX')\n\u003e\u003e\u003e cluster\nPBSCluster(0f23b4bf, 'tcp://xx.xxx.x.x:xxxx', workers=0, threads=0, memory=0 B)\n\u003e\u003e\u003e cluster.scale(jobs=2)\n\u003e\u003e\u003e client = Client(cluster)\n```\n\n### Hobart\n\n```python\n\u003e\u003e\u003e from ncar_jobqueue import NCARCluster\n\u003e\u003e\u003e from dask.distributed import Client\n\u003e\u003e\u003e cluster = NCARCluster()\n\u003e\u003e\u003e cluster\nPBSCluster(0f23b4bf, 'tcp://xx.xxx.x.x:xxxx', workers=0, threads=0, memory=0 B)\n\u003e\u003e\u003e cluster.scale(jobs=2)\n\u003e\u003e\u003e client = Client(cluster)\n```\n\n### Izumi\n\n```python\n\u003e\u003e\u003e from ncar_jobqueue import NCARCluster\n\u003e\u003e\u003e from dask.distributed import Client\n\u003e\u003e\u003e cluster = NCARCluster()\n\u003e\u003e\u003e cluster\nPBSCluster(0f23b4bf, 'tcp://xx.xxx.x.x:xxxx', workers=0, threads=0, memory=0 B)\n\u003e\u003e\u003e cluster.scale(jobs=2)\n\u003e\u003e\u003e client = Client(cluster)\n```\n\n### Non-NCAR machines\n\nOn non-NCAR machines, `ncar-jobqueue` will warn the user, and it will use `distributed.LocalCluster`:\n\n```python\n\u003e\u003e\u003e from ncar_jobqueue import NCARCluster\n.../ncar_jobqueue/cluster.py:17: UserWarning: Unable to determine which NCAR cluster you are running on... Returning a `distributed.LocalCluster` class.\nwarn(message)\n\u003e\u003e\u003e from dask.distributed import Client\n\u003e\u003e\u003e cluster = NCARCluster()\n\u003e\u003e\u003e cluster\nLocalCluster(3a7dd0f6, 'tcp://127.0.0.1:64184', workers=4, threads=8, memory=17.18 GB)\n```\n\n[github-ci-badge]: https://img.shields.io/github/actions/workflow/status/NCAR/ncar-jobqueue/ci.yaml\n[github-ci-link]: https://github.com/NCAR/ncar-jobqueue/actions?query=workflow%3ACI\n[codecov-badge]: https://img.shields.io/codecov/c/github/NCAR/ncar-jobqueue.svg?logo=codecov\n[codecov-link]: https://codecov.io/gh/NCAR/ncar-jobqueue\n[rtd-badge]: https://img.shields.io/readthedocs/dask-jobqueue/latest.svg\n[rtd-link]: https://jobqueue.dask.org/en/latest/\n[pypi-badge]: https://img.shields.io/pypi/v/ncar-jobqueue?logo=pypi\n[pypi-link]: https://pypi.org/project/ncar-jobqueue\n[conda-badge]: https://img.shields.io/conda/vn/conda-forge/ncar-jobqueue?logo=anaconda\n[conda-link]: https://anaconda.org/conda-forge/ncar-jobqueue\n[license-badge]: https://img.shields.io/github/license/NCAR/ncar-jobqueue\n[repo-link]: https://github.com/NCAR/ncar-jobqueue\n[pre-commit.ci-badge]: https://results.pre-commit.ci/badge/github/NCAR/ncar-jobqueue/main.svg\n[pre-commit.ci-link]: https://results.pre-commit.ci/latest/github/NCAR/ncar-jobqueue/main\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fncar%2Fncar-jobqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fncar%2Fncar-jobqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fncar%2Fncar-jobqueue/lists"}