https://github.com/jamesramm/qflow

Distributed ANUGA & Tuflow Modelling
https://github.com/jamesramm/qflow

cloud distributed gis hydrology python task-queue tuflow

Last synced: 15 days ago
JSON representation

Distributed ANUGA & Tuflow Modelling

• Host: GitHub
• URL: https://github.com/jamesramm/qflow
• Owner: JamesRamm
• License: agpl-3.0
• Created: 2017-07-17T11:16:39.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2022-12-26T20:35:40.000Z (almost 2 years ago)
• Last Synced: 2024-10-12T07:34:04.634Z (about 1 month ago)
• Topics: cloud, distributed, gis, hydrology, python, task-queue, tuflow
• Language: Python
• Homepage:
• Size: 479 KB
• Stars: 3
• Watchers: 2
• Forks: 1
• Open Issues: 19
• Metadata Files:
  • Readme: README.rst
  • Changelog: HISTORY.rst
  • Contributing: CONTRIBUTING.rst
  • License: LICENSE

Awesome Lists containing this project

README

        
==========

QFlow

==========

.. image:: https://codecov.io/gh/JamesRamm/qflow/branch/master/graph/badge.svg

        :target: https://codecov.io/gh/JamesRamm/qflow

.. image:: https://img.shields.io/travis/openfloodmap/qflow.svg

        :target: https://travis-ci.org/openfloodmap/qflow

.. image:: https://pyup.io/repos/github/openfloodmap/qflow/shield.svg

     :target: https://pyup.io/repos/github/openfloodmap/qflow/

     :alt: Updates

QFlow runs Anuga or Tuflow models across a distributed computing cluster.

QFlow manages the distribution of model runs across the cluster automatically, ensuring maximum usage

of your HPC resources and keeping your local computer resources free for your day to day tasks.

QFlow uses Celery & Redis to manage the task queue and runs on Python 3.5+

.. figure:: docs/qflow_arch.png

    :scale: 100 %

    :align: center

    :alt: QFlow conceptual diagram

Quick start

-----------

Follow this to get started using QFlow from the source.

- First install redis and setup conda environments for ANUGA. This can be done by running ``install.sh`` in the ``tools`` directory

- Launch 1 or more celery workers: ``tools/run.sh``

In order to execute an ANUGA script:

.. code-block:: python

        from qflow import tasks

        result = tasks.run_anuga.delay('/path/to/my/script.py')

In the above snippet, ``result`` is a celery ``AsyncResult``. You can check the status of the result by accessing ``result.state``.

If the task has finished, use ``result.result`` to access the task results.

You can easily queue up multiple ANUGA or Tuflow tasks:

.. code-block:: python

        from qflow import tasks

        scripts = ['~/run1.py', '~/run2.py', '~/run3.py']

        results = [tasks.run_anuga.delay(script) for script in scripts]

This script will return immeadiately after adding the tasks to the queue. Your worker(s) will then process each

task until no more are available. You can add another worker at any time and it will start picking up tasks from the queue.

Run with Docker

-----------------

A docker image is provided for QFlow (openfloodmap/qflow).

You can run with ``docker run qflow``.

Note that this image does not run redis - use a tool such as docker-compose to ensure Redis can be accessed from the qflow container.