https://github.com/cedadev/jasmin-daskgateway
https://github.com/cedadev/jasmin-daskgateway
Last synced: 7 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/cedadev/jasmin-daskgateway
- Owner: cedadev
- License: bsd-3-clause
- Created: 2022-06-21T14:33:16.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2023-04-17T08:44:43.000Z (about 3 years ago)
- Last Synced: 2025-01-06T02:09:23.146Z (over 1 year ago)
- Size: 22.5 KB
- Stars: 7
- Watchers: 6
- Forks: 0
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Using Dask in Lotus from the JASMIN Notebooks Service.
Using dask in Lotus requires connecting to a proxy service, called dask-gateway, which will create Lotus jobs on your behalf and proxy communication between your dask client and the dask clusters running in Lotus.
Before using dask gateway on JASMIN, you **first need to apply for the "dask" service on the JASMIN accounts portal**, which you can do so here: https://accounts.jasmin.ac.uk/services/additional_services/dask/
## Creating a dask cluster.
To create and connect to a dask cluster, you can use the following python snippet. Note that this will only work on the JASMIN Notebook service.
```python
import dask_gateway
# Create a connection to dask-gateway.
gw = dask_gateway.Gateway("https://dask-gateway.jasmin.ac.uk", auth="jupyterhub")
# Inspect and change the options if required before creating your cluster.
options = gw.cluster_options()
options.worker_cores = 2
# Create a dask cluster, or, if one already exists, connect to it.
# This stage creates the scheduler job in SLURM, so may take some time.
# While your job queues.
clusters = gw.list_clusters()
if not clusters:
cluster = gw.new_cluster(options, shutdown_on_close=False)
else:
cluster = gw.connect(clusters[0].name)
# Create at least one worker, and allow your cluster to scale to three.
cluster.adapt(minimum=1, maximum=3)
# Get a dask client.
client = cluster.get_client()
```
When you are done, and whish to release your cluster:
```python
cluster.shutdown()
```
## Other information and FAQs.
### Accessing the dask dashboard.
Currently the dask dashboard is not accessible from a browser outside the JASMIN firewall. If you're browser fails to load the dashboard link returned, please use our [interactive login service](https://help.jasmin.ac.uk/article/4810-graphical-linux-desktop-access-using-nx) to run a browser inside the firewall to view your dashboard.