{"id":24641186,"url":"https://github.com/elixir-cloud-aai/demo-tes-hybrid-cloud","last_synced_at":"2026-05-11T16:33:23.181Z","repository":{"id":198189551,"uuid":"610684010","full_name":"elixir-cloud-aai/demo-tes-hybrid-cloud","owner":"elixir-cloud-aai","description":"Demonstrating GA4GH TES usage in hybrid cloud use cases","archived":false,"fork":false,"pushed_at":"2024-01-22T15:01:19.000Z","size":546,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":26,"default_branch":"main","last_synced_at":"2025-03-20T11:49:05.471Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elixir-cloud-aai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-03-07T09:18:19.000Z","updated_at":"2023-03-10T10:54:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"32a1f3db-e8f4-4c0a-a287-28ea48e422ad","html_url":"https://github.com/elixir-cloud-aai/demo-tes-hybrid-cloud","commit_stats":null,"previous_names":["elixir-cloud-aai/demo-tes-hybrid-cloud"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/elixir-cloud-aai/demo-tes-hybrid-cloud","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-cloud-aai%2Fdemo-tes-hybrid-cloud","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-cloud-aai%2Fdemo-tes-hybrid-cloud/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-cloud-aai%2Fdemo-tes-hybrid-cloud/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-cloud-aai%2Fdemo-tes-hybrid-cloud/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elixir-cloud-aai","download_url":"https://codeload.github.com/elixir-cloud-aai/demo-tes-hybrid-cloud/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-cloud-aai%2Fdemo-tes-hybrid-cloud/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32903370,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-10T13:40:02.631Z","status":"online","status_checked_at":"2026-05-11T02:00:05.975Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-25T12:12:34.368Z","updated_at":"2026-05-11T16:33:23.164Z","avatar_url":"https://github.com/elixir-cloud-aai.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Demonstrator: Hybrid Cloud via GA4GH TES\n\n## Objectives\n\nDemonstrate the federated execution of a Snakemake workflow across a network of compute\ncenters operationalized through the GA4GH TES API.\n\n\u003e **Note:** This demonstrator was adapted from an [earlier demonstrator][demo-cwl]\n\u003e focusing on the execution of a [CWL][lang-cwl] workflow.\n\n## Service requirements\n\nThe following services were already deployed by ELIXIR for the demo:\n\n| Service | Service Type | Version / Image | Configuration | Comment |\n| --- | --- | --- | --- | --- |\n| [TESK][soft-tesk] | TES | `cerit.io/tes-wes/tesk-api:0.1` | Authorization checks disabled | Deployments for Kubernetes and OpenShift |\n| [Funnel][soft-funnel] | TES | `0.10.1` (OpenPBS) / commit `#52ef90f` (Slurm) | Basic authentication | Deployments for OpenPBS and Slurm; others possible |\n| [proTES][soft-protes] | TES gateway | `elixircloud/protes:20230218` | Authorization checks disabled; all TESK and Funnel instances need to be listed in `tes.service_list` in the app configuration prior to executing the demo | \u0026mdash; |\n| [MinIO][soft-minio] | Storage (S3) | `RELEASE.2022-05-08T23-50-31Z` | Standard | Uses filesystem mode ([unsupported in newer versions][soft-minio-filesystem-mode]) |\n\n## Client requirements\n\nYou can install the demo requirements with Conda or Mamba:\n\n```bash\nconda env install -n demo-tes-hybrid-cloud -f environment.yml\n# or\nmamba env install -n demo-tes-hybrid-cloud -f environment.yml\n```\n\nReinstall `py-tes` and check out a specific commit:\n\n```bash\npip uninstall -y py-tes\npip install git+https://github.com/ohsu-comp-bio/py-tes.git@5379b2a08bc911f7af55bd835dbd85aad0124c6a\n```\n\nVerify the successful installation:\n\n```bash\ncurl --version\njq --version\njupyter --version\njupyter-lab --version\nsnakemake --version\n```\n\n## Create configuration files\n\nThe demonstrator is set up in a way that it can be run on, in principle, on any\nFunnel/TESK TES network and on any input files. Therefore, we first need to create a\ncouple of configuration files that represent _your_ specific setup.\n\n### TES instances\n\nFirst, you need to create a listing of the available TES instances in a comma-separated\nfile `.tes_instances`. Two fields/columns are required, a description of the TES\nTES instance, and the URL pointing to it.\n\n\u003e You can use the following command to create such a file, but make sure to\n\u003e replace the example contents and do not use commas in the name/description\n\u003e field:\n\n```bash\ncat \u003c\u003c \"EOF\" \u003e .tes_instances\nFunnel/Slurm @ YourNode,https://tes.your-node.org/\nFunnel/PBS @ YetAnotherNode,https://tes.yet-another-node.org/\nTESK/Kubernetes @ OtherNode,https://tes.other-node.org/\nEOF\n```\n\n### Workflow inputs\n\nYou will also need to create a comma-separated file `.inputs`, containing the locations\nof input files and the URLs pointing to them, in the first and second field,\nrespectively. Additionally, the file needs to include one row, in which the first field\nis `workflow` (case-sensitive!). This file will be used as an input to the example CWL\nworkflow in the last part. You can use the following command to create such a file, but\nmake sure to replace the example contents.  Also make sure not to include commas in the\nfile location field.\n\n```bash\ncat \u003c\u003c \"EOF\" \u003e .inputs\nlocation_x,https://link.to.some/file.txt\nlocation_y,https://link.to.some/other_file.tab\nlocation_z,https://link.to.some/third_file.png\nworkflow,https://link.to.some/workflow_input\n```\n\n### Credentials\n\nFinally, you will need to create a secrets file `.env` with the following command. You\ncan either set the environment variables in your shell or set the actual values in the\ncommand below. Either way, you need to create the file, it is not sufficient to just the\nenvironment variables in your shell!\n\n```bash\ncat \u003c\u003c EOF \u003e .env\nFUNNEL_SERVER_USER=$FUNNEL_SERVER_USER\nFUNNEL_SERVER_PASSWORD=$FUNNEL_SERVER_PASSWORD\nTES_GATEWAY=$TES_GATEWAY\nEOF\n```\n\n## Start notebook server\n\nStart the Jupiter notebook server with the following command:\n\n```bash\njupyter-lab demo.ipynb\n```\n\nAlternatively, run the demo via Jupyter Notebook or via your shell terminal.\n\n## Contact\n\nFor questions and inquiries, please contact \u003ccloud-service@elixir-europe.org\u003e. You can\nalso reach out to us if you would like to execute the demonstrator on the ELIXIR Cloud\ninfrastructure.\n\n[demo-cwl]: \u003chttps://github.com/elixir-cloud-aai/elixir-cloud-demos/tree/main/demos/2023-ecp-f2f\u003e\n[docs-jupyter-lab]: \u003chttps://jupyterlab.readthedocs.io/\u003e\n[lang-cwl]: \u003chttps://www.commonwl.org/\u003e\n[lang-smk]: \u003chttps://snakemake.readthedocs.io/\u003e\n[soft-conda]: \u003chttps://conda.io/\u003e\n[soft-curl]: \u003chttps://curl.se/\u003e\n[soft-funnel]: \u003chttps://ohsu-comp-bio.github.io/funnel\u003e\n[soft-jupyter]: \u003chttps://jupyter.org/\u003e\n[soft-kube]: \u003chttps://kubernetes.io/\u003e\n[soft-mamba]: \u003chttps://mamba.readthedocs.io/\u003e\n[soft-minio]: \u003chttps://min.io/\u003e\n[soft-minio-filesystem-mode]: \u003chttps://min.io/docs/minio/linux/operations/install-deploy-manage/migrate-fs-gateway.html\u003e\n[soft-protes]: \u003chttps://github.com/elixir-cloud-aai/proTES\u003e\n[soft-py-tes]: \u003chttps://github.com/ohsu-comp-bio/py-tes\u003e\n[soft-tesk]: \u003chttps://github.com/elixir-cloud-aai/tesk\u003e\n[specs-tes]: \u003chttps://github.com/ga4gh/task-execution-schemas/\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felixir-cloud-aai%2Fdemo-tes-hybrid-cloud","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felixir-cloud-aai%2Fdemo-tes-hybrid-cloud","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felixir-cloud-aai%2Fdemo-tes-hybrid-cloud/lists"}