{"id":24562571,"url":"https://github.com/humanbrainproject/pyunicore","last_synced_at":"2025-04-19T17:42:48.464Z","repository":{"id":19308827,"uuid":"79567732","full_name":"HumanBrainProject/pyunicore","owner":"HumanBrainProject","description":"UNICORE REST bindings for python","archived":false,"fork":false,"pushed_at":"2025-03-18T13:17:11.000Z","size":489,"stargazers_count":9,"open_issues_count":1,"forks_count":10,"subscribers_count":11,"default_branch":"dev","last_synced_at":"2025-03-29T11:01:30.643Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HumanBrainProject.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-20T14:53:23.000Z","updated_at":"2025-03-18T13:17:15.000Z","dependencies_parsed_at":"2023-02-15T10:00:48.675Z","dependency_job_id":"2cbca6a2-a40c-4c5f-bda0-738cde12df06","html_url":"https://github.com/HumanBrainProject/pyunicore","commit_stats":{"total_commits":186,"total_committers":11,"mean_commits":16.90909090909091,"dds":0.5,"last_synced_commit":"858515276fb3582f64066059e33a2d29a99ac450"},"previous_names":[],"tags_count":65,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanBrainProject%2Fpyunicore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanBrainProject%2Fpyunicore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanBrainProject%2Fpyunicore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanBrainProject%2Fpyunicore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HumanBrainProject","download_url":"https://codeload.github.com/HumanBrainProject/pyunicore/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249751606,"owners_count":21320338,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-23T09:16:23.344Z","updated_at":"2025-04-19T17:42:48.446Z","avatar_url":"https://github.com/HumanBrainProject.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PyUNICORE, a Python library for using UNICORE and UFTP\n\nThis library covers the UNICORE REST API, making common tasks like\nfile access, job submission and management, workflow submission and\nmanagement more convenient, and integrating UNICORE features better\nwith typical Python usage.\n\nThe full, up-to-date documentation of the REST API can be found\n[here](https://unicore-docs.readthedocs.io/en/latest/user-docs/rest-api)\n\nIn addition, this library contains code for using UFTP (UNICORE FTP)\nfor filesystem mounts with FUSE, a UFTP driver for\n[PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2)\nand a UNICORE implementation of a\n[Dask Cluster](https://distributed.dask.org/en/stable/)\n\nThis project has received funding from the European Union’s\nHorizon 2020 Framework Programme for Research and Innovation under the\nSpecific Grant Agreement Nos. 720270, 785907 and 945539\n(Human Brain Project SGA 1, 2 and 3)\n\nSee LICENSE file for licensing information\n\n## Documentation\n\nThe complete documentation of PyUNICORE can be viewed\n[here](https://pyunicore.readthedocs.io/en/latest/)\n\n## Installation\n\nInstall from PyPI with\n\n    pip install -U pyunicore\n\nAdditional extra packages may be required for your use case:\n\n * Using the UFTP fuse driver requires \"fusepy\"\n * Using UFTP with pyfilesystem requires \"fs\"\n * Creating JWT tokens signed with keys requires the\n  \"cryptography\" package\n\nYou can install (one or more) extras with pip:\n\n    pip install -U pyunicore[crypto,fs,fuse]\n\n## Basic usage\n\n### Creating a client for a UNICORE site\n\n```Python\nimport pyunicore.client as uc_client\nimport pyunicore.credentials as uc_credentials\nimport json\n\nbase_url = \"https://localhost:8080/DEMO-SITE/rest/core\"\n\n# authenticate with username/password\ncredential = uc_credentials.UsernamePassword(\"demouser\", \"test123\")\n\nclient = uc_client.Client(credential, base_url)\nprint(json.dumps(client.properties, indent = 2))\n```\n\nPyUNICORE supports a variety of\n[authentication options](https://pyunicore.readthedocs.io/en/latest/authentication.html).\n\n### Run a job and read result files\n\n```Python\nmy_job = {'Executable': 'date'}\n\njob = client.new_job(job_description=my_job, inputs=[])\nprint(json.dumps(job.properties, indent = 2))\n\njob.poll() # wait for job to finish\n\nwork_dir = job.working_dir\nprint(json.dumps(work_dir.properties, indent = 2))\n\nstdout = work_dir.stat(\"/stdout\")\nprint(json.dumps(stdout.properties, indent = 2))\ncontent = stdout.raw().read()\nprint(content)\n```\n\n### Connect to a Registry and list all registered services\n\n```Python\nregistry_url = \"https://localhost:8080/REGISTRY/rest/registries/default_registry\"\n\n# authenticate with username/password\ncredential = uc_credentials.UsernamePassword(\"demouser\", \"test123\")\n\nr = uc_client.Registry(credential, registry_url)\nprint(r.site_urls)\n```\n\n### Further reading\n\nMore examples for using PyUNICORE can be found in the \"integration-tests\"\nfolder in the source code repository.\n\n## UFTP examples\n\n### Using UFTP for PyFilesystem\n\nYou can create a [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) `FS`\nobject either directly in code, or implicitely via a URL.\n\nThe convenient way is via URL:\n\n```Python\nfrom fs import open_fs\nfs_url = \"uftp://demouser:test123@localhost:9000/rest/auth/TEST:/data\"\nuftp_fs = open_fs(fs_url)\n```\n\n[More...](https://pyunicore.readthedocs.io/en/latest/uftp.html#using-uftp-for-pyfilesystem)\n\n### Mounting remote filesystems via UFTP\n\nPyUNICORE contains a FUSE driver based on [fusepy](https://pypi.org/project/fusepy),\nallowing you to mount a remote filesystem via UFTP. Mounting is a two step process,\n\n  * authenticate to an Auth server, giving you the UFTPD host/port and one-time password\n  * run the FUSE driver\n\n[More...](https://pyunicore.readthedocs.io/en/latest/uftp.html#mounting-remote-filesystems-via-uftp)\n\n## Tunneling / port forwarding\n\nOpens a local server socket for clients to connect to, where traffic\ngets forwarded to a service on a HPC cluster login (or compute) node.\nThis feature requires UNICORE 9.1.0 or later on the server side.\n\nYou can use this feature in two ways\n\n * in your own applications via the `pyunicore.client.Job` class.\n * you can also open a tunnel from the command line using the\n   'pyunicore.forwarder' module\n\n[More...](https://pyunicore.readthedocs.io/en/latest/port_forwarding.html)\n\n## Dask cluster implementation (experimental)\n\nPyUNICORE provides an implementation of a Dask Cluster, allowing to\nrun the Dask client on your local host (or in a Jupyter notebook in\nthe Cloud), and have the Dask scheduler and workers running remotely\non the HPC site.\n\n[More...](https://pyunicore.readthedocs.io/en/latest/dask.html)\n\n\n### Convert a CWL job to UNICORE\n\nPyUNICORE provides a tool to convert a CWL CommanLineTool and input into a\nUNICORE job file. Given the following YAML files that describe a\nCommandLineTool wrapper for the echo command and an input file:\n\n```yaml\n# echo.cwl\n\ncwlVersion: v1.2\n\nclass: CommandLineTool\nbaseCommand: echo\n\ninputs:\n  message:\n    type: string\n    inputBinding:\n      position: 1\n\noutputs: []\n```\n\n```yaml\n# hello_world.yml\n\nmessage: \"Hello World\"\n```\n\nA UNICORE job file can be generated using the following command:\n\n```bash\nunicore-cwl-runner echo.cwl hello_world.yml \u003e hello_world.u\n```\n\n## Helpers\n\nThe `pyunicore.helpers` module provides helper code for:\n\n* Connecting to\n  * a Registry (`pyunicore.helpers.connect_to_registry`).\n  * a site via a Registry URL (`pyunicore.helpers.connect_to_site_from_registry`).\n  * a site via its core URL (`pyunicore.helpers.connect_to_site`).\n* Defining descriptions as a dataclass and easily converting to a `dict` as required by `pyunicore.client.Client.new_job` via a `to_dict()` method:\n  * `pyunicore.helpers.jobs.Description` for `pyunicore.client.Client.new_job()`\n  * `pyunicore.helpers.workflows.Description` for `pyunicore.client.WorkflowService.new_workflow()`\n* Defining a workflow description\n\n### Connecting to a Registry\n\n```Python\nimport json\nimport pyunicore.credentials as uc_credentials\nimport pyunicore.helpers as helpers\n\nregistry_url = \"https://localhost:8080/REGISTRY/rest/registries/default_registry\"\n\ncredentials = uc_credentials.UsernamePassword(\"demouser\", \"test123\")\n\nclient = helpers.connection.connect_to_registry(\n    registry_url=registry_url,\n    credentials=credentials,\n)\nprint(json.dumps(client.properties, indent=2))\n```\n\n### Connecting to a site via a Registry\n\n```Python\nimport json\nimport pyunicore.credentials as uc_credentials\nimport pyunicore.helpers as helpers\n\nregistry_url = \"https://localhost:8080/REGISTRY/rest/registries/default_registry\"\nsite = \"DEMO-SITE\"\n\ncredentials = uc_credentials.UsernamePassword(\"demouser\", \"test123\")\n\nclient = helpers.connection.connect_to_site_from_registry(\n    registry_url=registry_url,\n    site_name=site,\n    credentials=credentials,\n)\nprint(json.dumps(client.properties, indent=2))\n```\n\n### Connecting to a site directly\n\n```Python\nimport json\nimport pyunicore.credentials as uc_credentials\nimport pyunicore.helpers as helpers\n\nsite_url = \"https://localhost:8080/DEMO-SITE/rest/core\"\n\ncredentials = uc_credentials.UsernamePassword(\"demouser\", \"test123\")\n\nclient = helpers.connection.connect_to_site(\n    site_api_url=site_url ,\n    credentials=credentials,\n)\nprint(json.dumps(client.properties, indent=2))\n```\n\n### Defining a job or workflow\n\n```Python\nfrom pyunicore import helpers\n\nclient = ...\n\nresources = helpers.jobs.Resources(nodes=4)\njob = helpers.jobs.Description(\n    executable=\"ls\",\n    project=\"demoproject\",\n    resources=resources\n)\n\nclient.new_job(job.to_dict())\n```\n\nThis works analogously for `pyunicore.helpers.workflows`.\n\n## Contributing\n\n1. Fork the repository\n2. Install the development dependencies\n\n   ```bash\n   pip install -r requirements-dev.txt\n   ```\n\n3. Install pre-commit hooks\n\n   ```bash\n   pre-commit install\n   ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumanbrainproject%2Fpyunicore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhumanbrainproject%2Fpyunicore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumanbrainproject%2Fpyunicore/lists"}