https://github.com/dmcg/dask-multicloud-poc
https://github.com/dmcg/dask-multicloud-poc
Last synced: 5 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/dmcg/dask-multicloud-poc
- Owner: dmcg
- License: other
- Created: 2022-06-20T15:31:32.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2022-07-11T11:35:27.000Z (about 4 years ago)
- Last Synced: 2025-10-08T19:35:02.321Z (10 months ago)
- Language: Shell
- Size: 2.42 MB
- Stars: 6
- Watchers: 6
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Dask Multicloud Proof of Concept
We have devised a technique for creating a Dask cluster where worker nodes are
hosted in different data centres, connected by a mesh VPN that allows the
scheduler and workers to communicate and exchange results.
A novel (ab)use of Dask resources allows us to run data processing tasks on the
workers in the cluster closest to the source data, so that communication between
data centres is minimised. If combined with zarr to give access to huge
hyper-cube datasets in object storage, we believe that the technique could
realise the potential of data-proximate distributed computing in the Cloud.
This repository documents a running proof-of-concept that addresses these problems.
It contains
## ./demo/
We show the working of the system in a Jupyter notebook
[dask-multi-cloud.ipynb](./demo/dask-multi-cloud.ipynb)
and more details in
[dask-multi-cloud-details.ipynb](./demo/dask-multi-cloud-details.ipynb).
## ./build.sh
Builds the Docker image used to host the scheduler and workers.
## ./docker/
Resources for the Docker image.
## ./ansible/
An example Ansible playbook showing how to commission worker machines.
## ./start-cluster
A script to start a Dask scheduler and a distributed cluster.