{"id":18431364,"url":"https://github.com/mpds-io/mpds-aiida","last_synced_at":"2025-04-07T18:32:45.376Z","repository":{"id":50751981,"uuid":"183018648","full_name":"mpds-io/mpds-aiida","owner":"mpds-io","description":"Automated computational workflows based on the MPDS data platform using the CRYSTAL first-principles engine","archived":false,"fork":false,"pushed_at":"2024-06-05T12:57:33.000Z","size":1044,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-06-05T14:43:41.674Z","etag":null,"topics":["ab-initio","materials-informatics","materials-science","mpds-platform"],"latest_commit_sha":null,"homepage":"https://mpds.io/search/ab%20initio%20calculations","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mpds-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-23T13:14:55.000Z","updated_at":"2024-06-05T12:57:37.000Z","dependencies_parsed_at":"2024-06-05T14:41:51.909Z","dependency_job_id":"1243d767-aa32-48c6-bc79-841d7d5ccec8","html_url":"https://github.com/mpds-io/mpds-aiida","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpds-io%2Fmpds-aiida","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpds-io%2Fmpds-aiida/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpds-io%2Fmpds-aiida/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpds-io%2Fmpds-aiida/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mpds-io","download_url":"https://codeload.github.com/mpds-io/mpds-aiida/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223288523,"owners_count":17120406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-initio","materials-informatics","materials-science","mpds-platform"],"created_at":"2024-11-06T05:24:35.741Z","updated_at":"2024-11-06T05:24:36.170Z","avatar_url":"https://github.com/mpds-io.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Cloud factory for the accurate materials data\n==========\n\n[![DOI](https://zenodo.org/badge/183018648.svg)](https://doi.org/10.5281/zenodo.7693214)\n\nusing the MPDS data platform, AiiDA workflows, and CRYSTAL simulation engine.\n\n![MPDS](https://raw.githubusercontent.com/mpds-io/mpds-aiida/master/mpds.png \"MPDS + AiiDA + CRYSTAL\") ![AiiDA](https://raw.githubusercontent.com/mpds-io/mpds-aiida/master/aiida.png \"AiiDA + MPDS + CRYSTAL\") ![CRYSTAL](https://raw.githubusercontent.com/mpds-io/mpds-aiida/master/crystal.jpg \"CRYSTAL + MPDS + AiiDA\")\n\n\n## Rationale\n\n- get accurate encyclopedic, reference, and benchmarking scientific data\n- get vast systematic training data for machine learning\n- use the cheap commodity cloud environment (_not_ necessarily the HPC cluster)\n- ensure provenance tracking and reproducibility of simulations with AiiDA\n\n\n## Installation\n\nThe code in this repo requires the [aiida-crystal-dft](https://github.com/tilde-lab/aiida-crystal-dft), [yascheduler](https://github.com/tilde-lab/yascheduler), and [mpds-ml-labs](https://github.com/mpds-io/mpds-ml-labs) Python packages installed. In their turn, they depend on the [aiida](https://github.com/aiidateam/aiida-core), [mpds_client](https://github.com/mpds-io/mpds_client), and other Python packages.\n\nThus, installation is as follows (replace `pip` with `pip3` if needed and mind virtual env):\n\n```shell\npip install git+https://github.com/tilde-lab/aiida-crystal-dft\npip install git+https://github.com/tilde-lab/yascheduler\npip install git+https://github.com/mpds-io/mpds-ml-labs\ngit clone https://github.com/mpds-io/mpds-aiida\npip install mpds-aiida/\n```\n\nHere some reader's AiiDA experience is assumed. Note, since the AiiDA does _not_ support cloud environments, the custom cloud scheduler engine [yascheduler](https://github.com/tilde-lab/yascheduler) should be employed. This scheduler manages the [CRYSTAL](http://www.crystal.unito.it) simulation engine at the cloud VPS instances and encapsulates all the details, concerning the remote *computer* task submission, queue, and results retrieval, as well as the VPS management. This scheduler runs its own daemon and lives together with the AiiDA at the same machine. However, AiiDA considers it as a remote service, accessible via the `ssh` transport, so the command `ssh $USER@localhost` should pass. To achieve that, the reader might run _e.g._:\n\n```shell\nssh-keygen -t rsa\ncat ~/.ssh/id_rsa.pub \u003e\u003e ~/.ssh/authorized_keys\nssh $USER@localhost\n```\n(Note, that the AiiDA should be aware of the `~/.ssh/id_rsa.pub` key file while SSH setup!)\n\nFor simplicity the `yascheduler` can share the database with AiiDA. Setting up the `yascheduler` looks like:\n\n```shell\nvi /etc/yascheduler/yascheduler.conf\nyainit\nservice yascheduler start\n```\n\nThe AiiDA should be set up normally, and the stub remote *computer* (_e.g._ `cluster: yascheduler`), as well as the stub CRYSTAL *code* (_e.g._ `codes: Pcrystal`) should be added:\n\n```shell\nreentry scan\nverdi setup\nverdi computer setup\nverdi computer configure ssh $COMPUTER\nverdi computer test $COMPUTER --print-traceback\nverdi code setup\n```\n\nWhy stub? Because the *computer* and *code* management is delegated to the `yascheduler`, taking care of the on-demand cloud resources management.\n\nThe Gaussian basis sets used by CRYSTAL engine should be added to the AiiDA database. We download the entire basis set library from the [CRYSTAL website](http://www.crystal.unito.it/basis-sets.php) and save some selected basis sets as `*.basis` files using the script `scripts/bs_unito_download.py`. Then, in a subfolder with the `*.basis` files, one runs:\n\n```shell\nverdi data crystal_dft uploadfamily --name=$BASIS_FAMILY\n```\n\nor, to add the internal basis sets predefined in CRYSTAL:\n\n```shell\nverdi data crystal_dft createpredefined\n```\n\nThen the desired name ($BASIS_FAMILY) should be used in the calculation settings inside `mpds_aiida/calc_templates` (see below).\n\n\n## Usage\n\nThe [MPDS platform](https://mpds.io) is the main data source for generating the simulation inputs and checking the simulation results. An access to the binary compounds data subset is free, one should [login at the MPDS](https://mpds.io/#modal/login) and get the MPDS API key:\n\n```shell\nexport MPDS_KEY=...\n```\n(Please do not forget to withdraw _i.e._ invalidate the API key after finishing the work.)\n\nA template system is used to control the calculation parameters, see the `mpds_aiida/calc_templates` subfolder. Note, that the `options: resources` template directive makes no sense with our custom cloud scheduler. The `cluster`, `codes`, and `basis_family` template directives have to be specified exactly as defined above.\n\nThe following on-demand cloud providers are currently supported (resp. `yascheduler` directives given in brackets):\n\n- [Hetzner](https://www.hetzner.com/cloud) (`hetzner_token`, `hetzner_max_nodes`), API token must be issued for a project\n- [Upcloud](https://upcloud.com) (`upcloud_login`, `upcloud_pass`, `upcloud_max_nodes`), API permissions are set in account settings\n\nAt the moment of writing, the chosen default Hetzner configuration (CX51) runs a test task for **2-2.5 hours** on average and costs **EUR 35.88** per month, the chosen default Upcloud configuration (8 cores, 4Gb memory) runs a test task for **1.5 hours** on average and costs **$89** per month.\n\nMore examples are given in the `scripts` subfolder.\n\nAn operation principle is briefly illustrated below.\n\n![General workflow](https://raw.githubusercontent.com/mpds-io/mpds-aiida/master/workflow.png \"General workflow\")\n\nNote: this repo is subject to change and presents an ongoing work in progress.\n\n\n## Licensing\n\n- This code: [MIT](https://en.wikipedia.org/wiki/MIT_License)\n- CRYSTAL engine: [commercial](https://www.crystal.unito.it)\n\nThe resulting data are available at the [MPDS platform](https://mpds.io/search/ab%20initio%20calculations), according to the CC BY 4.0 license.\n\n\n## Issues and troubleshooting\n\nPlease, report any issues in the respective repositories: [aiida-crystal-dft](https://github.com/tilde-lab/aiida-crystal-dft), [yascheduler](https://github.com/tilde-lab/yascheduler), [mpds-ml-labs](https://github.com/mpds-io/mpds-ml-labs), [aiida](https://github.com/aiidateam/aiida-core), [mpds_client](https://github.com/mpds-io/mpds_client), _etc._\n\nThe Google Cloud machines need first to be prepared via the web-browser SSH console (note `sudo -i`). The file `/etc/ssh/sshd_config` should be changed to allow `root` user to log in.\n\nThe Amazon EC2 machines need first to be accessed with the `admin` user (note `sudo -i`). Then the file `/root/.ssh/authorized_keys` needs to be cleaned to allow `root` user to log in.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpds-io%2Fmpds-aiida","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmpds-io%2Fmpds-aiida","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpds-io%2Fmpds-aiida/lists"}