Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/GusEllerm/GIS_IndexGen

Prototype implementation of LivePaper with GIS indices as the subject.
https://github.com/GusEllerm/GIS_IndexGen

cwl cwl-workflow gis livedata livepaper remote-sensing

Last synced: 3 months ago
JSON representation

Prototype implementation of LivePaper with GIS indices as the subject.

Awesome Lists containing this project

README 

        
# Project topology



For this example CWL works as the computational engine, running the workflow and generating the tiff outputs. 

Nodejs is used to publish these results as a webpage. 

CWL_output contains workflow outputs, CWL contains the cwl code to execute workflows and the python scripts that actually do the computation. 



To run the webserver install the npm dependencies (`npm install`) and run npm start from top directory. Current webserver is listening on port 3000.



# Running examples



To run examples you will need to install the dependencies in requirements.txt in your local env. 

testCL.txt contains example executions of the index workflows. Use these, or your own data to produce outputs. 



Using pipenv is encouraged as it is easy to install all required dependencies from the Pipfile. 



Graphviz will need to be installed. 



# TODO



Currently, the webserver is not ready for deployment however the CWL is reasonably complete. 



# Example index matrix calculations



Assuming you are using pipenv, you can enter the venv using `pipenv shell`

Currently, only four indexes are implemented:

- NDVI

- RECI

- NDRE

- GNDVI



## Executing python scripts without CWL wrapper

### index_def.py script -- generates both the pickle files from the data source (band matrix's) and the index pickle file (index matrix)



`python3 CWL/Workflows/Modules/Scripts/index_def.py -i [name of index e.g. NDVI] -b [list of input bands space seperated]`



e.g. 



`python3 CWL/Workflows/Modules/Scripts/index_def.py -i NDVI -b ~/Projects/LivePaper-prototype/CWL/Workflow_inputs/Data/S2A_MSIL2A_20220207T222541_N0400_R029_T59GMK_20220208T001202.SAFE/GRANULE/L2A_T59GMK_A034632_20220207T222543/IMG_DATA/R20m/T59GMK_20220207T222541_B04_20m.jp2 ~/Projects/LivePaper-prototype/CWL/Workflow_inputs/Data/S2A_MSIL2A_20220207T222541_N0400_R029_T59GMK_20220208T001202.SAFE/GRANULE/L2A_T59GMK_A034632_20220207T222543/IMG_DATA/R20m/T59GMK_20220207T222541_B8A_20m.jp2`



### tiff_gen.py script -- generates a tiff image from an index matrix



`python3 CWL/Workflows/Modules/Scripts/tiff_gen.py -i [index matrix] -c [color map]`



e.g.



`python3 CWL/WOrkflows/Modules/Scripts/tiff_gen.py -i ~/Projects/LivePaper-prototype/T59GMK_20220207T222541_NDVI_20m.pickle -c RdYlGn`



A list of available colors can be found [here] (https://matplotlib.org/stable/tutorials/colors/colormaps.html)



## Executing workflows with CWL wrappers



Assuming you have a reference cwl-runner installed (included within dependencies) you should be able to access cwl-runner from your venv. 



### Running single Modules



The CWL is structured such that every workflow is built from two modules:

- index_def.cwl

- tiff_gen.cwl



These modules can be run independently, or as a workflow. 



To run the entire workflow:



`cwl-runner CWL/Workflows/multi_workflow.cwl CWL/Workflow_inputs/multi_workflow.yaml`



This will produce a tiff image for all four index's, as well as versions for each resolution possible (r10m, r20m)



# Notes



Assuming the issue of identity is solved by the researcher knowing what artefacts are generated by their workflow. 

For this example workflow seven artefacts are assumed to be generated on workflow completion. 

- GNDVI 10m tiff

- GNDVI 20m tiff

- NDRE 20m tiff

- NDVI 10m tiff

- NDVI 20m tiff

- RECI 10m tiff

- RECI 20m tiff



Supplimentary data is provided on piecemeal processing of each tiff.