{"id":15336422,"url":"https://github.com/kinow/auto-mhm-test-domains","last_synced_at":"2025-04-15T03:12:29.373Z","repository":{"id":161024087,"uuid":"624628558","full_name":"kinow/auto-mhm-test-domains","owner":"kinow","description":"Repository with an example workflow to run mHM with its test data, using containers","archived":false,"fork":false,"pushed_at":"2025-03-14T07:36:15.000Z","size":38031,"stargazers_count":4,"open_issues_count":5,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-15T03:12:09.996Z","etag":null,"topics":["autosubmit","climate","docker","hydrology","model","workflow"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kinow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-04-06T22:35:27.000Z","updated_at":"2025-03-14T07:36:18.000Z","dependencies_parsed_at":"2023-06-26T01:51:11.159Z","dependency_job_id":"ddc9e38e-2c16-4460-8941-140d191a9f1a","html_url":"https://github.com/kinow/auto-mhm-test-domains","commit_stats":{"total_commits":79,"total_committers":2,"mean_commits":39.5,"dds":"0.025316455696202556","last_synced_commit":"c2672568fb14706c6e66481bc7ceac467db8261e"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinow%2Fauto-mhm-test-domains","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinow%2Fauto-mhm-test-domains/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinow%2Fauto-mhm-test-domains/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinow%2Fauto-mhm-test-domains/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kinow","download_url":"https://codeload.github.com/kinow/auto-mhm-test-domains/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248997079,"owners_count":21195799,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autosubmit","climate","docker","hydrology","model","workflow"],"created_at":"2024-10-01T10:15:48.349Z","updated_at":"2025-04-15T03:12:29.366Z","avatar_url":"https://github.com/kinow.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n--\u003e\n\n[![DOI](https://zenodo.org/badge/624628558.svg)](https://zenodo.org/badge/latestdoi/624628558)\n\n# Autosubmit mHM workflow with mHM test domains data\n\nThis repository contains the configuration for an [Autosubmit][autosubmit]\nexperiment that runs [mHM][mhm], the mesoscale hydrological model, using\nthe mHM test domain data in a workflow. A Docker container is used to install\nthe dependencies and run mHM.\n\nThe two images below were created using the workflow in this\nrepository, and running the [`plot.py`][plotpy] script (it uses\nXarray and Matplotlib to parse the NetCDF test output).\n\n\u003cdiv\u003e\n  \u003cimg src=\"./docs/plot.gif\" style=\"max-width: 400px;\" alt=\"mHM test domain plot\" /\u003e\n\u003c/div\u003e\n\n## Prerequisites\n\n\u003c!--\nNOTE: With CWL you can list the software requirements for a computational\n      workflow. Unfortunately we do not have the same for Autosubmit. But\n      maybe we could find a way to give a specification of requirements?\n      CWL can also declare that a workflow or tool needs Internet, and\n      even Docker. So everything in this section can be part of the CWL\n      Workflow definition. Would be nice to have something we could use\n      in Autosubmit (like a standard way of defining it?).\n--\u003e\n\nIn order to run the workflow you will need the following\nprerequisites:\n\n1. Internet connection\n2. Linux\n3. Autosubmit 4.0.98 or greater (e.g. `pip install autosubmit==4.0.98`)\n4. Docker \u0026 Singularity 3.11.x or greater for running the container\n5. ImageMagick's `convert` to generate the GIF animation using the PNG files produced by `plot.py`\n\nFor a list of software used, besides `mHM`, see the [`Dockerfile`][dockerfile].\n\n## Build the containers\n\nTo build the Docker container image first, use this command.\n\n```bash\n$ docker build --no-cache=true --tag \"auto-mhm-test-domains/mhm:v5.12.1.dev228\" --load .\n```\n\nNow, to build the Singularity container, using the Docker container image, use this command.\n\n```bash\n$ singularity build --force mhm.sif docker-daemon://auto-mhm-test-domains/mhm:v5.12.1.dev228\n```\n\nNow copy the created container `mhm.sif` into the location specified in `conf/mhm.yml`.\n\n**NOTE**: The workflow expects the `mhm.sif` Singularity file to exist\non each platform.\n\n## Workflow\n\nAn Autosubmit experiment created with\n`autosubmit expid -H \u003cYOUR_PLATFORM\u003e -d \"mHM\" -min -repo https://github.com/kinow/auto-mhm-test-domains.git -git_branch master -conf conf/bootstrap`\nwhen run will clone this Git repository, and prepare the Docker\n\u0026 Singularity containers, transfer all the required data over to\nthe chosen platform (it can be local for testing) and execute\nthe mHM model for each start date (as the mHM simulation period date).\nThe last tasks in the workflow will plot the data before cleaning the\nsimulation logs and files.\n\n\u003cimg src=\"./docs/mhm-workflow-graph.png\" style=\"max-width: 400px; max-height: 400px;\"  alt=\"mHM workflow graph\"/\u003e\n\nThe plots are copied back to the local workflow folder, and\neverything can be used to package an RO-Crate.\n\n## Running\n\nYou will need an Autosubmit experiment first, so that you\ncan import the configuration files from this Git repository.\nRun the following command to create a new Autosubmit experiment.\n\n```bash\n$ autosubmit expid \\\n    --description \"Autosubmit mHM test domains\" \\\n    --minimal_configuration \\\n    --git_as_conf conf/bootstrap \\\n    --git_repo https://github.com/kinow/auto-mhm-test-domains.git \\\n    --git_branch master\n```\n\nThis will create a new experiment, using SSH connections to the\n“local” platform. On the first execution of `autosubmit create`,\nit will clone the Git repository specified and load the\nconfiguration from the subdirectory specified in the `git_as_conf`\nparameter.\n\n\u003e NOTE: The output of `autosubmit expid` contains the ID of an\n\u003e experiment. Replace `\u003cEXPID\u003e` by that value in the next\n\u003e commands.\n\nThe `platforms.yml` file in this repository contains placeholders\nlike `\u003cUSER\u003e`, `\u003cREMOTE_HOST\u003e`, etc., that must be filled in before\nthe workflow can be used. Alternatively, you can create the\nfollowing file with your platform configuration (replace the\nvalues by your user account information).\n\n```yaml\n# File: ~/.config/autosubmit/platforms.yml\n\nPLATFORMS:\n  # You can use a different name, just adjust your `autosubmit expid`\n  # command and `conf/platforms.yml`, and this other platforms.yml\n  # files.\n  REMOTE:\n    TYPE: ps\n    HOST: localhost\n    ADD_PROJECT_TO_HOST: false\n    SCRATCH_DIR: /tmp/scratch/\n    PROJECT: mhm-project\n    USER: kinow\n```\n\n\u003e NOTE: The `SCRATCH_DIR` and `PROJECT` directories must already\n\u003e       exist or be created before following the next steps.\n\nThe next command is to prepare the experiment workflow (i.e.\nparse and validate its configuration and produce a workflow graph,\nprepare jobs and scripts, etc.):\n\n```bash\n$ autosubmit create \u003cEXPID\u003e\n```\n\nIf everything goes well you should see the workflow graph plot\nappear on your screen (if you have an X server running). Close\nthe PDF and now run the workflow with the following command:\n\n```bash\n$ autosubmit run \u003cEXPID\u003e\n```\n\nThat will execute the complete workflow of your Autosubmit\nexperiment. If you used `nohup`, or if you have another\ncommand-line terminal, you can monitor the execution of\nthe workflow with the following commands:\n\n```bash\n# plot a new PDF with the progress of your workflow\n$ autosubmit monitor \u003cEXPID\u003e\n# print the output logs of your workflow\n$ autosubmit cat-log --file o \u003cEXPID\u003e\n# print the error logs of your workflow, in `tail -f` mode\n$ autosubmit cat-log --file o --mode e \u003cEXPID\u003e\n```\n\n\u003e NOTE: Autosubmit commands produce log files on disk\n\u003e for traceability. You can increase the log levels\n\u003e with the `-lf` (log file) and `lc` (log console)\n\u003e parameter flags.\n\nAnd if you are using a version of Autosubmit that supports\nRO-Crate, you can create an archive with the provenance\nmetadata file, the workflow configuration, plots, logs, and\nother traces included.\n\n```bash\n$ autosubmit archive --rocrate \u003cEXPID\u003e\n```\n\n\u003e NOTE: If you use RO-Crate, you will have to provide a\n\u003e JSON-LD file with extra metadata about your\n\u003e experiment and workflow to have a more complete\n\u003e RO-Crate file (using the Workflow Run Crate Profile).\n\n## RO-Crate\n\nThis is an example RO-Crate configuration file for this Autosubmit workflow.\nYou can add this setting anywhere, like `\u003cEXPID\u003e/conf/rocrate.yml`.\n\n```yaml\nROCRATE:\n  INPUTS:\n    # Add the extra configuration keys to be exported.\n    # This one is from the mhm.yml file, we export everything\n    # as inputs.\n    - \"MHM\"\n  OUTPUTS:\n    - \"*/*.gif\"\n  PATCH: |\n    {\n      \"@graph\": [\n        {\n          \"@id\": \"./\",\n          \"license\": \"Apache-2.0\",\n          \"creator\": {\n            \"@id\": \"https://orcid.org/0000-0001-8250-4074\"\n          },\n          \"publisher\": {\n            \"@id\": \"https://ror.org/05sd8tv96\"\n          }\n        },\n        {\n          \"@id\": \"#create-action\",\n          \"@type\": \"CreateAction\",\n          \"name\": \"Run mHM\",\n          \"instrument\": { \"@id\": \"workflow.yml\" },\n          \"agent\": { \"@id\": \"https://orcid.org/0000-0001-8250-4074\" }\n        },\n        {\n          \"@id\": \"ro-crate-metadata.json\",\n          \"author\": [\n            {\n              \"@id\": \"https://orcid.org/0000-0001-8250-4074\"\n            }\n          ]\n        },\n        {\n          \"@id\": \"https://orcid.org/0000-0001-8250-4074\",\n          \"@type\": \"Person\",\n          \"affiliation\": {\n              \"@id\": \"https://ror.org/05sd8tv96\"\n          },\n          \"contactPoint\": {\n              \"@id\": \"mailto: bruno.depaulakinoshita@bsc.es\"\n          },\n          \"name\": \"Bruno P. Kinoshita\"\n        },\n        {\n            \"@id\": \"mailto: bruno.depaulakinoshita@bsc.es\",\n            \"@type\": \"ContactPoint\",\n            \"contactType\": \"Author\",\n            \"email\": \"bruno.depaulakinoshita@bsc.es\",\n            \"identifier\": \"bruno.depaulakinoshita@bsc.es\",\n            \"url\": \"https://orcid.org/0000-0001-8250-4074\"\n        },\n        {\n            \"@id\": \"https://ror.org/05sd8tv96\",\n            \"@type\": \"Organization\",\n            \"name\": \"Barcelona Supercomputing Center\"\n        }\n      ]\n    }\n```\n\n## Troubleshooting\n\nIf you have issues running the workflow that are related to the containers,\nyou can try re-building without the cache by using the `docker build --no-cache`\nflag.\n\nAnother option to troubleshoot issues with the containers is to use the final\ncontainer image, `mhm.sif`, directly in the command-line, without the workflow.\n\nFor example:\n\n```bash\n$ singularity run mhm.sif mhm-download -b develop -d 1 -p /tmp/data\n```\n\n## License\n\nThis workflow is licensed under the GNU General Public License V3. You can\nfind the complete license at the file [`LICENSE.txt`][license].\n\nThe license applies only to the workflow configuration and code,\nand not to the rest of the code and tools used.\n\nPlease, note that:\n\n- Autosubmit is licensed under the GNU General Public License v3.\n- mHM is licensed under the GNU General Public License v3.\n\n[autosubmit]: https://autosubmit.readthedocs.io/\n\n[mhm]: https://mhm.pages.ufz.de/mhm/stable/\n\n[dockerfile]: ./Dockerfile\n\n[license]: ./LICENSE.txt\n\n[plotpy]: ./plot.py\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkinow%2Fauto-mhm-test-domains","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkinow%2Fauto-mhm-test-domains","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkinow%2Fauto-mhm-test-domains/lists"}