{"id":13296998,"url":"https://github.com/xenon-middleware/xenonflow","last_synced_at":"2025-03-10T09:33:04.293Z","repository":{"id":39547778,"uuid":"63334137","full_name":"xenon-middleware/xenonflow","owner":"xenon-middleware","description":"Run CWL workflows using Xenon through a REST api","archived":false,"fork":false,"pushed_at":"2023-01-07T04:09:20.000Z","size":2799,"stargazers_count":4,"open_issues_count":21,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-06-11T17:29:06.546Z","etag":null,"topics":["cwl","java","workflow","xenon"],"latest_commit_sha":null,"homepage":"https://xenon-middleware.github.io/xenonflow/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xenon-middleware.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null}},"created_at":"2016-07-14T12:19:13.000Z","updated_at":"2023-03-15T13:30:41.000Z","dependencies_parsed_at":"2023-02-06T11:30:44.935Z","dependency_job_id":null,"html_url":"https://github.com/xenon-middleware/xenonflow","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xenon-middleware%2Fxenonflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xenon-middleware%2Fxenonflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xenon-middleware%2Fxenonflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xenon-middleware%2Fxenonflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xenon-middleware","download_url":"https://codeload.github.com/xenon-middleware/xenonflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221164334,"owners_count":16767311,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cwl","java","workflow","xenon"],"created_at":"2024-07-29T17:21:20.891Z","updated_at":"2024-10-23T06:30:57.323Z","avatar_url":"https://github.com/xenon-middleware.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n[![Build and Test Xenonflow](https://github.com/xenon-middleware/xenon-flow/actions/workflows/build.yml/badge.svg?branch=master)](https://github.com/xenon-middleware/xenon-flow/actions/workflows/build.yml)\n[![DOI](https://zenodo.org/badge/63334137.svg)](https://zenodo.org/badge/latestdoi/63334137)\n\n##### CWL Compliance v1.0\n\n![Required](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/required.json?icon=commonwl)\n![Workflow](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/workflow.json?icon=commonwl)\n![CommanLine](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/command_line_tool.json?icon=commonwl)\n![Docker](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/docker.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/env_var.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/expression_tool.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/initial_work_dir.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/inline_javascript.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/multiple_input.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/resource.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/scatter.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/schema_def.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/shell_command.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/step_input.json?icon=commonwl)\n![Badge](https://badgen.net/https/raw.githubusercontent.com/xenon-middleware/xenon-flow/gh-pages/badges/subworkflow.json?icon=commonwl)\n\n\n# Xenonflow\nRun CWL workflows using Xenon through a REST api.\n\n# Usage:\nThe following diagram shows a rough overview of the interaction when using xenonflow.\nThe overview shows 3 file systems, all of which can be configured for Xenonflow (See the quick-start guide below).\n1. The input filesystem, this should contain all the input files needed for running the cwl workflow\n2. The cwl filesystem, this filesystem should contain the cwl workflows you want to run with Xenonflow.\n3. The output filesystem, this is where xenonflow will put the output of the workflows.\n\nOn the right you can see a compute resource: Xenonflow can be configured to run on a number of computing backends, including the local machine, to actually execute the cwl workflow.\n\nBefore making a call to the Xenonflow REST API make sure the data is available on the input filesystem and the workflow is available on the cwl filesystems.\nThe rest call will return a JSON object which contains some information on the job you just submitted. Such as its current state, what input was provided, a uri to the job (for instance to poll the server for new states) and a uri to the log of the job.\n\nAfter the workflow is completed the results will be available in the target filesystem\n\n![Xenonflow Usage Pattern](https://user-images.githubusercontent.com/16776108/116083884-718bdd80-a69d-11eb-982a-7351b1e586f3.png)\n\n\n# Quick-start\n## 1. Install the dependencies:\n - Java 11\n - a cwl runner\n\nFor the cwl runner you can use the reference implementation called cwltool.\nIt can be installed by\n```bash\npip install cwltool\n```\nYou may need to use pip3 on some systems.\nFor a full list of available cwl runners check https://www.commonwl.org/#Implementations\n\nAfter installing the cwl runner it is a good idea to double check that your workflow can be run\nusing the runner on the command line\n\n## 2. Download Xenon-flow\n```\nwget https://github.com/xenon-middleware/xenon-flow/releases/V1.0\nunzip xenonflow-1.0.zip\n```\n\n## 3. Configure the server\nConfiguration of the server is done by editing the `XENONFLOW_HOME/config/config.yml` file.\nAs well as the `XENONFLOW_HOME/config/application.properties`.\n\nBy default it is set the use the local file system as the source and the local\ncomputer to run workflows.\n\nFor information on which filesystems and schedulers can be used refer to the xenon documentation: https://xenon-middleware.github.io/xenon/versions/3.1.0/javadoc/.\n\n### config.yml\nXenon-flow configuration consists of\n1. `sourceFileSystem`: Any filesystem supported by Xenon can be used here\n2. `targetFileSystem`: Any filesystem supported by Xenon can be used here\n3. `cwlFileSystem`: Any filesystem supported by Xenon can be used here\n4. `ComputeResources`: A map of compute resource descriptions for Xenon\n5. Each resource has the following settings:\n    1. `cwlCommand`: A script to run the cwl runner, allowing for python environments to be started first.\n    \t* Default:\n    \t\t```![xenonflow (3)](https://user-images.githubusercontent.com/16776108/116083865-6d5fc000-a69d-11eb-81ca-5e82fa2c6727.png)\n\n    \t\t#!/usr/bin/env bash\n\n    \t\tcwltool $@\n    \t\t```\n    2. `scheduler`: A Xenon scheduler\n    3. `filesystem` A Xenon filesystem\n    4. Both the scheduler and filesystem have to following format:\n        1. `adaptor`: The name of the Xenon adaptor (for instance slurm for scheduler or sftp for filesystem)\n        2. `location`: The URI for the resource\n        3. `credential`: Optional credentials (if not supplied the current user and ~/.ssh/id_rsa is used)\n        \t1. `user`: Username\n        \t2. `password`: Password in base64 encoded\n        4. `properties`: Optional properties (usually not needed)\n\n#### Environment Variables\nThere are two environment variables that can be set in your environement which can then be\nused in the config.yml file: `XENONFLOW_FILES` and `XENONFLOW_HOME`.\n\n \n\n### application.properties\nThe application.properties needs configuration for the following things:\n1. api-key\n\t1. `xenonflow.http.auth-token-header-name` controls the name of the header that holds the api key\n\t2. `xenonflow.http.auth-token` the value of the api key. IMPORTANT you should really change this one\n2. The Database Configuration.\n\t* These settings should be changed!\n    \t1. `spring.datasource.username` The database username\n    \t2. `spring.datasource.password` The database password3.\n\t* The following settings can be left as is.\n    \t1. `server.port` The port for the server to run on.\n    \t2. `local.server.address=localhost` The servername.\n    \t3. `server.http.interface` Set up the server to be publicaly available by setting this to 0.0.0.0\n\n\n## 4. Start the server\nThe following command will run the server.\n```\n./bin/xenonflow\n```\n\n## 5. Run a workflow\nPut the workflow and any input files and directories to into the location as defined by the `sourceFileSystem` in the config. For instance when using a webdav server, upload the files there.\n\nSend a POST http request with a job description to the server.\n\n### Example:\n\nGiven the echo command-line-tool (in yaml notation):\n\n```yaml\ncwlVersion: v1.0\nclass: CommandLineTool\ninputs:\n  - id: inp\n    type: string\n    inputBinding: {}\n\noutputs:\n  - id: out\n    type: string\n    outputBinding:\n      glob: out.txt\n      loadContents: true\n      outputEval: $(self[0].contents)\n\nbaseCommand: echo\nstdout: out.txt\n```\n\nThe job description looks something like the following.\n\nNote that the input map contains a key `inp` which refers to the corresponding input of the echo command-line-tool.\n\n```json\n{\n    \"name\": \"My First Workflow\",\n    \"workflow\": \"cwl/echo.cwl\",\n    \"input\": {\n        \"inp\": \"Hello CWL Server!\"\n    }\n}\n```\n\n```bash\ncurl -X POST -H \"Content-Type: application/json\" -H \"api-key: \u003cinsert api key here\u003e\" -d '{\"name\": \"My First Workflow\",\"workflow\": \"cwl/echo.cwl\",\"input\": {\"inp\": \"Hello CWL Server!\"}}' http://localhost:8080/jobs\n```\n\n### Using the jobid or jobname in a workflow\nIf you need access to the jobid generated by xenonflow, or the jobname that was used to submit the workflow\nthen you can add them as inputs to your cwl file as parameters with the ids `xenonflow_jobid` and `xenonflow_jobname` respectively.\nXenonflow will then automatically inject the values into the job-order.json as input to the cwl file.\n\nFor example the following cwl file will echo the xenonflow_jobid and xenonflow_jobname:\n```yaml\ncwlVersion: v1.0\nclass: CommandLineTool\ninputs:\n  - id: xenonflow_jobid\n    type: string\n    inputBinding:\n      position: 1\n  - id: xenonflow_jobname\n    type: string\n    inputBinding:\n      position: 2\n\noutputs:\n  - id: out\n    type: string\n    outputBinding:\n      glob: out.txt\n      loadContents: true\n      outputEval: $(self[0].contents)\n\nbaseCommand: echo\nstdout: out.txt\n```\n\n\n\n### Running Xenonflow behind a proxy server\nWe recommend running xenonflow behind a proxy server. Both nginx and apache httpd are good candidates for this. In addition both nginx and apache httpd can act as webdav servers which xenonflow can use as a sourceFileSystem.\n\nDoing this requires no changes to the configuration of xenonflow as long as the correct X-forwarded-* headers are set in the proxy server.\n\nTo ensure that xenonflow returns the correct uri's for the jobs you should set the following headers:\n* X-Forwarded-Host\n* X-Forwarded-Server\n* X-Forwarded-Proto\n* X-Forwarded-Port\n* X-Forwarded-Prefix\n\nBelow is an example location from a nginx config that correctly proxies a xenonflow instance running at localhost:8080\n```nginx\n...\nlocation /api/ {\n\n    include cors;\n    proxy_pass http://localhost:8080/;\n    proxy_redirect off;\n    proxy_set_header Host $host;\n    proxy_set_header X-Forwarded-Host $host;\n    proxy_set_header X-Forwarded-Server $host;\n    proxy_set_header X-Forwarded-Proto http;\n    proxy_set_header X-Forwarded-Port $server_port;\n    proxy_set_header X-Forwarded-Prefix /api/;\n}\n...\n```\n\n### Running Xenonflow in SSL mode\nTo run xenonflow in ssl (https) mode you can follow the following steps:\n1. Please read https://dzone.com/articles/spring-boot-secured-by-lets-encrypt for setup using Letsencrypt.\n2. You should now have a certificate with a private key store\n3. You should now set the following properties in the application.properties file:\n   1. `server.ssl.enabled=true` Enable ssl encryption in the server\n   2. `server.ssl.key-store-type` The keystore type (spring boot only supports PKCS12).\n   3. `server.ssl.key-store` The store for the certificate files.\n   4. `server.ssl.key-store-password` The password to the key store.\n   5. `server.ssl.key-alias` The alias as given to the keystore.\n\n### Cleaning Input Data\n*Warning:* This will delete the input data on the source directory. It is recommended to set\nthe input filesystem to a different location than the cwl and output filesystems so files are\nnot lost by accident.\n\nYou can have xenonflow clean up the input files after a task has run by setting the `clearOnJobDone` parameter\nto true in the sourceFileSystem.\n\ni.e.\n\n```yaml\nsourceFileSystem:\n   adaptor: file\n   location: ${XENONFLOW_FILES}/input\n   clearOnJobDone: true\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxenon-middleware%2Fxenonflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxenon-middleware%2Fxenonflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxenon-middleware%2Fxenonflow/lists"}