{"id":20749722,"url":"https://github.com/tilde-lab/yascheduler","last_synced_at":"2025-04-28T12:53:02.814Z","repository":{"id":37594887,"uuid":"222936146","full_name":"tilde-lab/yascheduler","owner":"tilde-lab","description":"Yet another cloud computing scheduler for the high-throughput cloud scientific simulations","archived":false,"fork":false,"pushed_at":"2024-07-25T17:22:40.000Z","size":535,"stargazers_count":5,"open_issues_count":21,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-05T16:19:27.490Z","etag":null,"topics":["ab-initio","azure","azure-api-management","azure-cloud-services","hetzner","hetzner-api","hetzner-cloud","materials-informatics","materials-science","python","queues","scheduler","upscale"],"latest_commit_sha":null,"homepage":"https://mpds.io/search/ab%20initio%20calculations","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tilde-lab.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-20T12:48:49.000Z","updated_at":"2024-07-25T17:22:44.000Z","dependencies_parsed_at":"2023-10-12T06:43:02.983Z","dependency_job_id":"83c12337-c196-47f2-b46f-0cc8e7db1e9d","html_url":"https://github.com/tilde-lab/yascheduler","commit_stats":{"total_commits":267,"total_committers":6,"mean_commits":44.5,"dds":0.5205992509363295,"last_synced_commit":"aa971baa209184de3bc4e61fc71df60303855899"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tilde-lab%2Fyascheduler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tilde-lab%2Fyascheduler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tilde-lab%2Fyascheduler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tilde-lab%2Fyascheduler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tilde-lab","download_url":"https://codeload.github.com/tilde-lab/yascheduler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225015804,"owners_count":17407476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-initio","azure","azure-api-management","azure-cloud-services","hetzner","hetzner-api","hetzner-cloud","materials-informatics","materials-science","python","queues","scheduler","upscale"],"created_at":"2024-11-17T08:24:18.037Z","updated_at":"2024-11-17T08:24:18.524Z","avatar_url":"https://github.com/tilde-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Yet another computing scheduler \u0026 cloud orchestration engine\n\n[![DOI](https://zenodo.org/badge/222936146.svg)](https://doi.org/10.5281/zenodo.7693555)\n[![PyPI](https://img.shields.io/pypi/v/yascheduler.svg?style=flat)](https://pypi.org/project/yascheduler)\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Ftilde-lab%2Fyascheduler.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Ftilde-lab%2Fyascheduler?ref=badge_shield)\n\n**Yascheduler** is a simple job scheduler designed for submitting scientific\ncalculations and copying back the results from the computing clouds.\n\nCurrently it supports several scientific simulation codes in chemistry\nand solid state physics.\nAny other scientific simulation code can be supported via the declarative\ncontrol template system (see `yascheduler.conf` settings file).\nThere is an [example dummy C++ code][dummy-engine] with its configuration template.\n\n## Installation\n\nUse `pip` and PyPI: `pip install yascheduler`.\n\nThe last updates and bugfixes can be obtained cloning the repository:\n\n```sh\ngit clone https://github.com/tilde-lab/yascheduler.git\npip install yascheduler/\n```\n\nThe installation procedure creates the configuration file located at\n`/etc/yascheduler/yascheduler.conf`.\nThe file contains credentials for Postgres database access, used directories,\ncloud providers and scientific simulation codes (called _engines_).\nPlease check and amend this file with the correct credentials. The database\nand the system service should then be initialized with `yainit` script.\n\n## Usage\n\n```python\nfrom yascheduler import Yascheduler\n\nyac = Yascheduler()\nlabel = \"test assignment\"\nengine = \"pcrystal\"\nstruct_input = str(...)  # simulation control file: crystal structure\nsetup_input = str(...)  # simulation control file: main setup, can include struct_input\nresult = yac.queue_submit_task(\n    label, {\"fort.34\": struct_input, \"INPUT\": setup_input}, engine\n)\nprint(result)\n```\n\nOr run directly in console with `yascheduler` (use a key `-l DEBUG` to change the log level).\n\n_Supervisor_ config reads e.g.:\n\n```\n[program:scheduler]\ncommand=/usr/local/bin/yascheduler\nuser=root\nautostart=true\nautorestart=true\nstderr_logfile=/data/yascheduler.log\nstdout_logfile=/data/yascheduler.log\n```\n\nFile paths can be set using the environment variables:\n\n- `YASCHEDULER_CONF_PATH`\n\n  Configuration file.\n\n  _Default_: `/etc/yascheduler/yascheduler.conf`\n\n- `YASCHEDULER_LOG_PATH`\n\n  Log file path.\n\n  _Default_: `/var/log/yascheduler.log`\n\n- `YASCHEDULER_PID_PATH`\n\n  PID file.\n\n  _Default_: `/var/run/yascheduler.pid`\n\n## Configuration File Reference\n\n### Database Configuration `[db]`\n\nConnection to a PostgreSQL database.\n\n- `user`\n\n  The username to connect to the PostgreSQL server with.\n\n- `password`\n\n  The user password to connect to the server with. This parameter is optional\n\n- `host`\n\n  The hostname of the PostgreSQL server to connect with.\n\n- `port`\n\n  The TCP/IP port of the PostgreSQL server instance.\n\n  _Default_: `5432`\n\n- `database`\n\n  The name of the database instance to connect with.\n\n  _Default_: Same as `user`\n\n### Local Settings `[local]`\n\n- `data_dir`\n\n  Path to root directory of local data files.\n  Can be relative to the current working directory.\n\n  _Default_: `./data` (but it's always a good idea to set up explicitly!)\n\n  _Example_: `/srv/yadata`\n\n- `tasks_dir`\n\n  Path to directory with tasks results.\n\n  _Default_: `tasks` under `data_dir`\n\n  _Example_: `%(data_dir)s/tasks`\n\n- `keys_dir`\n\n  Path to directory with SSH keys. Make sure it only contains the private keys.\n\n  _Default_: `keys` under `data_dir`\n\n  _Example_: `%(data_dir)s/keys`\n\n- `engines_dir`\n\n  Path to directory with engines repository.\n\n  _Default_: `engines` under `data_dir`\n\n  _Example_: `%(data_dir)s/engines`\n\n- `webhook_reqs_limit`\n\n  Maximum number of in-flight webhook http requests.\n\n  _Default_: 5\n\n- `conn_machine_limit`\n\n  Maximum number of concurrent SSH connection's `connect` requests.\n\n  _Default_: 10\n\n- `conn_machine_pending`\n\n  Maximum number of pending SSH connection's `connect` requests.\n\n  _Default_: 10\n\n- `allocate_limit`\n\n  Maximum number of concurrent task or node allocation requests.\n\n  _Default_: 20\n\n- `allocate_pending`\n\n  Maximum number of pending task or node allocation requests.\n\n  _Default_: 1\n\n- `consume_limit`\n\n  Maximum number of concurrent task's results downloads.\n\n  _Default_: 20\n\n- `consume_pending`\n\n  Maximum number of pending task's results downloads.\n\n  _Default_: 1\n\n- `deallocate_limit`\n\n  Maximum number of concurrent node deallocation requests.\n\n  _Default_: 5\n\n- `deallocate_pending`\n\n  Maximum number of pending node deallocation requests.\n\n  _Default_: 1\n\n### Remote Settings `[remote]`\n\n- `data_dir`\n\n  Path to root directory of data files on remote node.\n  Can be relative to the remote current working directory (usually `$HOME`).\n\n  _Default_: `./data`\n\n  _Example_: `/src/yadata`\n\n- `tasks_dir`\n\n  Path to directory with tasks results on remote node.\n\n  _Default_: `tasks` under `data_dir`\n\n  _Example_: `%(data_dir)s/tasks`\n\n- `engines_dir`\n\n  Path to directory with engines on remote node.\n\n  _Default_: `engines` under `data_dir`\n\n  _Example_: `%(data_dir)s/engines`\n\n- `user`\n\n  Default ssh username.\n\n  _Default_: `root`\n\n- `jump_user`\n\n  Username of default SSH _jump host_ (if used).\n\n- `jump_host`\n\n  Host of default SSH _jump host_ (if used).\n\n### Providers `[clouds]`\n\nAll cloud providers settings are set in the `[cloud]` group.\nEach provider has its own settings prefix.\n\nThese settings are common to all the providers:\n\n- `*_max_nodes`\n\n  The maximum number of nodes for a given provider.\n  The provider is not used if the value is less than 1.\n\n- `*_user`\n\n  Per provider override of `remote.user`.\n\n- `*_priority`\n\n  Per provider priority of node allocation.\n  Sorted in descending order, so the cloud with the highest value is the first.\n\n- `*_idle_tolerance`\n\n  Per provider idle tolerance (in seconds) for deallocation of nodes.\n\n  _Default_: different for providers, starting from 120 seconds.\n\n- `*_jump_user`\n\n  Username of this cloud SSH jump host (if used).\n\n- `*_jump_host`\n\n  Host of this cloud SSH jump host (if used).\n\n#### Hetzner\n\nSettings prefix is `hetzner`.\n\n- `hetzner_token`\n\n  API token with Read \u0026 Write permissions for the project.\n\n- `hetzner_server_type`\n\n  Server type (size).\n\n  _Default_: `cx52`\n\n- `hetzner_image_name`\n\n  Image name for new nodes.\n\n  _Default_: `debian-11`\n\n#### Azure\n\nAzure Cloud should be pre-configured for `yascheduler`. See [Cloud Providers](CLOUD.md).\n\nSettings prefix is `az`.\n\n- `az_tenant_id`\n\n  Tenant ID of Azure Active Directory.\n\n- `az_client_id`\n\n  Application ID.\n\n- `az_client_secret`\n\n  Client Secret value from the **Application Registration**.\n\n- `az_subscription_id`\n\n  Subscription ID\n\n- `az_resource_group`\n\n  Resource Group name.\n\n  _Default_: `yascheduler-rg`\n\n- `az_user`\n\n  SSH username. `root` is not supported.\n\n- `az_location`\n\n  Default location for resources.\n\n  _Default_: `westeurope`\n\n- `az_vnet`\n\n  Virtual network name.\n\n  _Default_: `yascheduler-vnet`\n\n- `az_subnet`\n\n  Subnet name.\n\n  _Default_: `yascheduler-subnet`\n\n- `az_nsg`\n\n  Network security group name.\n\n  _Default_: `yascheduler-nsg`\n\n- `az_vm_image`\n\n  OS image name.\n\n  _Default_: `Debian`\n\n- `az_vm_size`\n\n  Machine size.\n\n  _Default_: `Standard_B1s`\n\n#### UpCloud\n\nSettings prefix is `upcloud`.\n\n- `upcloud_login`\n\n  Username.\n\n- `upcloud_password`\n\n  Password.\n\n### Engines `[engine.*]`\n\nSupported engines should be defined in the section(s) `[engine.name]`.\nThe name is alphanumeric string to represent the real engine name.\nOnce set, it cannot be changed later.\n\n- `platforms`\n\n  List of supported platform, separated by space or newline.\n\n  _Default_: `debian-10`\n  _Example_: `mY-cOoL-OS another-cool-os`\n\n- `platform_packages`\n\n  A list of required packages, separated by space or newline, which\n  will be installed by the system package manager.\n\n  _Default_: []\n  _Example_: `openmpi-bin wget`\n\n- `deploy_local_files`\n\n  A list of filenames, separated by space or newline, which will be copied\n  from local `%(engines_dir)s/%(engine_name)s` to remote\n  `%(engines_dir)s/%(engine_name)s`.\n  Conflicts with `deploy_local_archive` and `deploy_remote_archive`.\n\n  _Example_: `dummyengine`\n\n- `deploy_local_archive`\n\n  A name of the local archive (`.tar.gz`) which will be copied\n  from local `%(engines_dir)s/%(engine_name)s` to the remote machine and\n  then unarchived to the `%(engines_dir)s/%(engine_name)s`.\n  Conflicts with `deploy_local_archive` and `deploy_remote_archive`.\n\n  _Example_: `dummyengine.tar.gz`\n\n- `deploy_remote_archive`\n\n  The url to the engine arhive (`.tar.gz`) which will be downloaded\n  to the remote machine and then unarchived to the\n  `%(engines_dir)s/%(engine_name)s`.\n  Conflicts with `deploy_local_archive` and `deploy_remote_archive`.\n\n  _Example_: `https://example.org/dummyengine.tar.gz`\n\n- `spawn`\n\n  This command is used by the scheduler to initiate calculations.\n\n  ```sh\n  cp {task_path}/INPUT OUTPUT \u0026\u0026 mpirun -np {ncpus} --allow-run-as-root \\\n    -wd {task_path} {engine_path}/Pcrystal \u003e\u003e OUTPUT 2\u003e\u00261\n\n  ```\n\n  _Example_: `{engine_path}/gulp \u003c INPUT \u003e OUTPUT`\n\n- `check_pname`\n\n  Process name used to check that the task is still running.\n  Conflicts with `check_cmd`.\n\n  _Example_: `dummyengine`\n\n- `check_cmd`\n\n  Command used to check that the task is still running.\n  Conflicts with `check_pname`. See also `check_cmd_code`.\n\n  _Example_: `ps ax -ocomm= | grep -q dummyengine`\n\n- `check_cmd_code`\n\n  Expected exit code of command from `check_cmd`.\n  If code matches than task is running.\n\n  _Default_: `0`\n\n- `sleep_interval`\n\n  Interval in seconds between the task checks.\n  Set to a higher value if you are expecting long running jobs.\n\n  _Default_: `10`\n\n- `input_files`\n\n  A list of task input file names, separated by a space or new line,\n  that will be copied to the remote directory of the task before it is started.\n  The first input is considered as the **main** input.\n\n  _Example_: `INPUT sibling.file`\n\n- `output_files`\n\n  A list of task output file names, separated by a space or new line,\n  that will be copied from the remote directory of the task after it is finished.\n\n  _Example_: `INPUT OUTPUT`\n\n## Aiida Integration\n\nSee the detailed instructions for the [MPDS-AiiDA-CRYSTAL workflows][mpds-aiida]\nas well as the [ansible-mpds][ansible-aiida] repository. In essence:\n\n```sh\nssh aiidauser@localhost # important\nreentry scan\nverdi computer setup\nverdi computer test $COMPUTER\nverdi code setup\n```\n\n[ansible-aiida]: https://github.com/mpds-io/ansible-mpds\n[mpds-aiida]: https://github.com/mpds-io/mpds-aiida\n[dummy-engine]: https://github.com/tilde-lab/dummy-engine\n\n## License\n\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Ftilde-lab%2Fyascheduler.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Ftilde-lab%2Fyascheduler?ref=badge_large)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftilde-lab%2Fyascheduler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftilde-lab%2Fyascheduler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftilde-lab%2Fyascheduler/lists"}