{"id":25019468,"url":"https://github.com/aiidateam/aiida-firecrest","last_synced_at":"2025-04-13T04:04:52.635Z","repository":{"id":45590473,"uuid":"434543193","full_name":"aiidateam/aiida-firecrest","owner":"aiidateam","description":"AiiDA Transport/Scheduler plugins for interfacing with FirecREST (https://products.cscs.ch/firecrest/)","archived":false,"fork":false,"pushed_at":"2025-04-07T21:08:40.000Z","size":220,"stargazers_count":3,"open_issues_count":22,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-13T04:04:19.460Z","etag":null,"topics":["aiida-plugin","firecrest"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aiidateam.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-03T09:45:29.000Z","updated_at":"2025-01-22T10:43:24.000Z","dependencies_parsed_at":"2024-12-03T01:19:18.680Z","dependency_job_id":"c28c09a8-535d-4d09-b7ae-d87a8d5625c8","html_url":"https://github.com/aiidateam/aiida-firecrest","commit_stats":{"total_commits":16,"total_committers":1,"mean_commits":16.0,"dds":0.0,"last_synced_commit":"64c1584fdbb42c8561387932c7e23ab4bb657182"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aiidateam%2Faiida-firecrest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aiidateam%2Faiida-firecrest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aiidateam%2Faiida-firecrest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aiidateam%2Faiida-firecrest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aiidateam","download_url":"https://codeload.github.com/aiidateam/aiida-firecrest/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248661707,"owners_count":21141450,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiida-plugin","firecrest"],"created_at":"2025-02-05T11:50:41.535Z","updated_at":"2025-04-13T04:04:52.585Z","avatar_url":"https://github.com/aiidateam.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aiida-firecrest [IN-DEVELOPMENT]\n\n[![Coverage Status][codecov-badge]][codecov-link]\n[![Code style: black][black-badge]][black-link]\n\n\u003cbr\u003e\n\u003cdiv align=\"left\"\u003e\n  \u003cimg src=\"assets/logo.png\" alt=\"Repository Logo\" width=\"200\" style=\"margin-top: 20px; margin-bottom: 20px;\"/\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\nAiiDA Transport/Scheduler plugins for interfacing with [FirecREST](https://products.cscs.ch/firecrest/), via [pyfirecrest](https://github.com/eth-cscs/pyfirecrest).\n\nIt is currently tested against [FirecREST v2.6.0](https://github.com/eth-cscs/pyfirecrest/tree/v2.6.0).\n\n\n## Installation\n\nInstall via GitHub or PyPI:\n\n```bash\npip install git+https://github.com/aiidateam/aiida-firecrest.git\n# pip not yet available\npip install aiida-firecrest\n```\n\nYou should then be able to see the `firecrest` transport and scheduler plugins loaded in AiiDA:\n\n```console\n$ verdi plugin list aiida.transports firecrest\nA plugin to connect to a FirecREST server.\nIt must be used together with the 'firecrest' scheduler plugin.\nAuthentication parameters:\n  url: URL to the FirecREST server\n  token_uri: URI for retrieving FirecREST authentication tokens\n  client_id: FirecREST client ID\n  client_secret: FirecREST client secret\n  client_machine: FirecREST machine secret\n  small_file_size_mb: Below this size, file bytes will be sent in a single API call.\n\n$ verdi plugin list aiida.schedulers firecrest\nA plugin to connect to a FirecREST server.\nIt must be used together with the 'firecrest' transport plugin.\n```\n\nYou can then create a `Computer` in AiiDA:\n\n```console\n$ verdi computer setup\nReport: enter ? for help.\nReport: enter ! to ignore the default and set no value.\nComputer label: firecrest-client                 # your choice\nHostname: unused                                 # your choice, irrelevant\nDescription []: My FirecREST client plugin       # your choice\nTransport plugin: firecrest\nScheduler plugin: firecrest\nShebang line (first line of each script, starting with #!) [#!/bin/bash]:\nWork directory on the computer [/scratch/{username}/aiida/]:\nMpirun command [mpirun -np {tot_num_mpiprocs}]:\nDefault number of CPUs per machine:  2           # depending on your compute resource\nDefault amount of memory per machine (kB).: 100  # depending on your compute resource\nEscape CLI arguments in double quotes [y/N]:\nSuccess: Computer\u003c3\u003e firecrest-client created\nReport: Note: before the computer can be used, it has to be configured with the command:\nReport:   verdi -p MYPROFILE computer configure firecrest firecrest-client\n```\n\n```console\n$ verdi -p MYPROFILE computer configure firecrest firecrest-client\nReport: enter ? for help.\nReport: enter ! to ignore the default and set no value.\nServer URL: https://firecrest.cscs.ch          # this for CSCS\nToken URI: https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token\nClient ID: username-client\nClient Secret: xyz\nCompute resource (Machine): daint\nTemp directory on server: /scratch/something/ # \"A temp directory on user's space on the server for creating temporary files (compression, extraction, etc.)\"\nFirecREST api version [Enter 0 to get this info from server] [0]: 0\nMaximum file size for direct transfer (MB) [Enter 0 to get this info from server] [0]: 0\nReport: Configuring computer firecrest-client for user chrisj_sewell@hotmail.com.\nSuccess: firecrest-client successfully configured for chrisj_sewell@hotmail.com\n```\n\nYou can always check your config with\n```console\n$ verdi computer show firecrest-client\n```\n\nSee also the [pyfirecrest CLI](https://github.com/eth-cscs/pyfirecrest), for directly interacting with a FirecREST server.\n\n\nAfter this, everything should function normally through AiiDA with no problems.\nSee [tests/test_calculation.py](tests/test_calculation.py) for a working example of how to use the plugin, via the AiiDA API.\n\nIf you encounter any problems/bug, please don't hesitate to open an issue on this repository.\n\n### Current Issues\n\nCalculations are now running successfully, however, there are still issues regarding efficiency, Could be improved:\n\n1. Monitoring / management of API request rates could to be improved. Currently this is left up to pyfirecrest.\n2. Each transfer request includes 2 seconds of `sleep` time, imposed by pyfirecrest. One can takes use of their `async` client, but with current design of `aiida-core`, the gain will be minimum. (see the [closing comment of issue#94 on pyfirecrest](https://github.com/eth-cscs/pyfirecrest/issues/94) and [PR#6079 on aiida-core ](https://github.com/aiidateam/aiida-core/pull/6079))\n\n## For developers\n\n```bash\ngit clone\ncd aiida-firecrest\npip install -e .[dev]\n```\n\n### Code Style\n\nTo format the code and lint it, run [pre-commit](https://pre-commit.com/):\n\n```bash\npre-commit run --all-files\n```\n\n### Testing\n\nIt is recommended to run the tests via [tox](https://tox.readthedocs.io/en/latest/).\n\n```bash\ntox\n```\n\nBy default, the tests are run using a monkey patched pyfirecrest.\nThis allows for quick testing and debugging of the plugin, without needing to connect to a real server, but is obviously not guaranteed to be fully representative of the real behaviour.\n\nTo have a guaranteed proof, you may also provide connections details to a real FirecREST server:\n\n```bash\ntox -- --firecrest-config=\".firecrest-demo-config.json\"\n```\n\n\nIf a config file is provided, tox sets up a client environment with the information\nin the config file and uses pyfirecrest to communicate with the server.\n```plaintext\n┌─────────────────┐───►┌─────────────┐───►┌──────────────────┐\n│ aiida_firecrest │    │ pyfirecrest │    │ FirecREST server │\n└─────────────────┘◄───└─────────────┘◄───└──────────────────┘\n```\n\nif a config file is not provided, it monkeypatches pyfirecrest so we never actually communicate with a server.\n```plaintext\n┌─────────────────┐───►┌─────────────────────────────┐\n│ aiida_firecrest │    │ pyfirecrest (monkeypatched) │\n└─────────────────┘◄───└─────────────────────────────┘\n```\n\nThe format of the `.firecrest-demo-config.json` file, for example is like:\n\n\n```json\n {\n    \"url\": \"https://firecrest-tds.cscs.ch\",\n    \"token_uri\": \"https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token\",\n    \"client_id\": \"username-client\",\n    \"client_secret\": \"path-to-secret-file\",\n    \"compute_resource\": \"daint\",\n    \"temp_directory\": \"/scratch/snx3000/username/\",\n    \"small_file_size_mb\": 5.0,\n    \"workdir\": \"/scratch/snx3000/username/\",\n    \"api_version\": \"1.16.0\"\n}\n```\n\nIn this mode, if you want to inspect the generated files, after a failure, you can use:\n\n```bash\ntox -- --firecrest-config=\".firecrest-demo-config.json\" --firecrest-no-clean\n```\n\n**These tests were successful against [FirecREST v1.16.0](https://github.com/eth-cscs/firecrest/releases/tag/v1.16.0), except those who require to list directories in a symlink directory, which fail due to a bug in FirecREST. [An issue](https://github.com/eth-cscs/firecrest/issues/205) is open on FirecREST repo about this.**\n\nInstead of a real server (which requires an account and credential), tests can also run against a docker image provided by FirecREST. See [firecrest_demo.py](firecrest_demo.py) for how to start up a demo server, and also [server-tests.yml](.github/workflows/server-tests.yml) for how the tests are run against the demo server on GitHub Actions.\n\n\u003c!-- If you want to analyse statistics of the API requests made by each test,\nyou can use the `--firecrest-requests` option:\n\n```bash\ntox -- --firecrest-requests\n``` --\u003e\n\n#### Notes on using the demo server on MacOS\n\nA few issues have been noted when using the demo server on MacOS (non-Mx):\n\n`docker-compose up` can fail, with an error that port 7000 is already in use.\nRunning `lsof -i :7000` you may see it is used by `afs3-fileserver`,\nwhich can be fixed by turning off the Airplay receiver\n(see \u003chttps://github.com/cookiecutter/cookiecutter-django/issues/3499\u003e)\n\nLarge file uploads can fail, because the server provides a URL with ``192.168.220.19`` that actually needs to be ``localhost``.\nTo fix this, ensure that you set `FIRECREST_LOCAL_TESTING = true` in your environment\n(set by default if running `tox`).\n\nLarge file downloads has the same problem, but even with this fix, it will still fail with a 403 HTTP error, due to a signature mismatch.\nNo automatic workaround has been found for this yet,\nalthough it is of note that you can find these files directly where you your `firecrest` Github repo is cloned, `/path/to/firecrest/deploy/demo/minio/` plus the path of the URL.\n\n[codecov-badge]: https://codecov.io/gh/aiidateam/aiida-firecrest/branch/main/graph/badge.svg\n[codecov-link]: https://codecov.io/gh/aiidateam/aiida-firecrest\n[black-badge]: https://img.shields.io/badge/code%20style-black-000000.svg\n[black-link]: https://github.com/ambv/black\n\n\n### :bug: Fishing Bugs :bug:\n\nFirst, start with running tests locally with no `config` file given, that would monkeypatch pyfirecrest. These set of test do not guarantee that the whole firecrest protocol is working, but it's very useful to quickly check if `aiida-firecrest` is behaving as it's expected to do. To run just simply use `pytest` or `tox`.\n\nIf these tests pass and the bug persists, consider providing a `config` file to run the tests on a docker image or directly on a real server. Be aware of versioning, pyfirecrest doesn't check which version of api it's interacting with.  (see https://github.com/eth-cscs/pyfirecrest/issues/116)\n\nIf the bug persists and test still passes, then most certainly it's a problem of `aiida-firecrest`.\nIf not, probably the issue is from FirecREST, you might open an issue to [pyfirecrest](https://github.com/eth-cscs/pyfirecrest) or [`FirecREST`](https://github.com/eth-cscs/firecrest).\n\n### Acknowledgment:\nThis project is supported by SwissTwins fundings.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faiidateam%2Faiida-firecrest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faiidateam%2Faiida-firecrest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faiidateam%2Faiida-firecrest/lists"}