{"id":28591449,"url":"https://github.com/thehyve/fairspace","last_synced_at":"2026-03-10T05:31:51.649Z","repository":{"id":37803142,"uuid":"136937918","full_name":"thehyve/fairspace","owner":"thehyve","description":"Fairspace provides a secure environment for managing research data.","archived":false,"fork":false,"pushed_at":"2025-08-13T10:04:37.000Z","size":74820,"stargazers_count":14,"open_issues_count":4,"forks_count":6,"subscribers_count":10,"default_branch":"dev","last_synced_at":"2025-08-13T11:34:29.972Z","etag":null,"topics":["fair","fairspace","metadata","rdf","research-data","webdav"],"latest_commit_sha":null,"homepage":"https://fairspace.nl","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thehyve.png","metadata":{"files":{"readme":"README.adoc","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-06-11T14:28:43.000Z","updated_at":"2025-07-12T09:39:52.000Z","dependencies_parsed_at":"2024-01-02T14:27:36.421Z","dependency_job_id":"1b9ab549-b977-4573-9f65-921d2fbaa33c","html_url":"https://github.com/thehyve/fairspace","commit_stats":null,"previous_names":[],"tags_count":588,"template":false,"template_full_name":null,"purl":"pkg:github/thehyve/fairspace","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thehyve%2Ffairspace","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thehyve%2Ffairspace/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thehyve%2Ffairspace/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thehyve%2Ffairspace/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thehyve","download_url":"https://codeload.github.com/thehyve/fairspace/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thehyve%2Ffairspace/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30326071,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T05:25:20.737Z","status":"ssl_error","status_checked_at":"2026-03-10T05:25:17.430Z","response_time":106,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fair","fairspace","metadata","rdf","research-data","webdav"],"created_at":"2025-06-11T09:14:51.160Z","updated_at":"2026-03-10T05:31:51.613Z","avatar_url":"https://github.com/thehyve.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"= Fairspace\nThe Hyve\nVERSION, {docdate}\n:description: Fairspace documentation.\n:author: The Hyve\n:doctype: book\n:showtitle!:\n:url-repo: https://github.com/thehyve/fairspace\n:source-highlighter: rouge\n:icons: font\n:toc: macro\n:toclevels: 3\n:toc-title: Fairspace\n\n:Jena: https://jena.apache.org/\n:RDF: https://en.wikipedia.org/wiki/Resource_Description_Framework\n:SPARQL: https://www.w3.org/TR/sparql11-query/\n:SHACL: https://www.w3.org/TR/shacl/\n:Keycloak: https://www.keycloak.org/\n:Keycloak_server_administration: https://www.keycloak.org/docs/latest/server_admin/\n:JupyterHub: https://jupyterhub.readthedocs.io/\n:FAIR: https://www.go-fair.org/fair-principles/\n:WebDAV: https://en.wikipedia.org/wiki/WebDAV\n:json-ld: https://json-ld.org/\n:jsonld-schema: https://raw.githubusercontent.com/json-ld/json-ld.org/master/schemas/jsonld-schema.json\n:turtle: https://www.w3.org/TR/turtle/\n:n-triples: https://www.w3.org/TR/n-triples/\n\nimage:docs/images/Fairspace.png[Fairspace]\n\nFairspace is a secure place for managing research data.\nResearch teams have their own workspaces in which they\ncan manage research data collections.\nResearchers can upload directories and files to data collections.\nData access is organised on data collection level.\nCollections can be shared with other teams or individual researchers.\nAlso, collections can be published for all researchers in the organisation. +\n\nCollections and files can be annotated with descriptive metadata.\nThe metadata is stored using the {RDF}[Resource Description Framework (RDF)] in\nan {Jena}[Apache Jena] database.\nFor the metadata, a data model can be configured that suits\nthe data management needs of the organisation.\nThe data model is specified using the {SHACL}[Shapes Constraint Language (SHACL)],\nsee the section on \u003c\u003cData model and view configuration\u003e\u003e.\nDescriptive metadata entities (e.g., subjects, projects, samples) should be added to the database by a\ncareful process, ensuring that duplicates and inconsistencies are avoided and\nthat all entities have proper unique identifiers.\nThe application provides overviews of the available metadata entities.\nIn the collection browser, researchers can link their collections and files to these entities\nor add textual descriptions and key words.\n\n\n.*Key features*\n****\n\n* Fairspace is a data repository that enables researchers to securely *store* and *organise* their research data sets,\nand *share* the data with collaborators.\n* Fairspace lets researchers annotate their data collections with relevant metadata properties\nand link the data to associated metadata entities (subjects, samples, projects, etc.).\nThis helps researchers find their own data and make it *findable* for others,\ncontributing to implementation of the {FAIR}[FAIR principles].\n* Fairspace ensures that all metadata entities have a unique identifier and checks\nmetadata consistency and validity upon data entry.\n* Fairspace allows organisations to *customise* the configured data model,\nby specifying custom entity types and constraints.\nThis enables the adoption of community standards for metadata relevant for the research domain,\nwhich contributes to the *reusability* of the data.\n* Fairspace uses the {RDF}[Resource Description Framework (RDF)] and {WebDAV}[WebDAV] standards for data exchange,\nand stimulates the use of standard vocabularies,\ncontributing to *interoperability* of data.\n****\n\ntoc::[]\n\n\n\n== Usage\n\n=== User interface\n\n==== Login\n\nUsers are authenticated using {Keycloak}[Keycloak], an open-source identity provider\nthat provides secure authentication methods and can be configured to integrate\nwith institutional identity providers using user federation or identity brokering,\nsee the {Keycloak_server_administration}[Keycloak server administration] pages.\n\nThe user either logs in directly using Keycloak or is forwarded to a configured\nexternal login:\n\nimage:docs/images/screenshots/Keycloak login.png[Keycloak login]\n\n==== Workspaces\n\nUsers enter Fairspace on the workspaces page that lists all workspaces.\nA workspace represents a team in the organisation that collaborates on research data collections.\n\nimage:docs/images/screenshots/Workspace list.png[Workspace list]\n\nWorkspace administrators can edit the workspace overview page and\nmanage workspace membership. All workspace members can add collections to the workspace.\n\nimage:docs/images/screenshots/Workspace overview.png[Workspace overview]\n\n==== Collections\n\nThe contents of collections can be navigated in the collections browser.\nIt behaves like a regular file browser. Click to select a directory or file\nand see its metadata, double click to navigate into directories or\nopen a file.\n\nAccess is managed on collection level.\nUsers with at least _write_ access to a collection can upload files or directories,\nrename or delete files, restore old file versions,\nand edit the associated metadata.\n\nUsers with _manage_ access can share collections with other users or workspaces,\nand change the default access mode for workspace members.\nCollection managers can also change the status of the collection\n(_Active_, _Read-only_ or _Archived_), change the view mode\n(_Restricted_, _Metadata published_ or _Data published_ -- only available in the _Read-only_ status),\ndelete the collection,\nor transfer ownership of the collection to another workspace.\n\nimage:docs/images/screenshots/Collection browser.png[Collection browser]\n\nDue to the data loss prevention, data in Fairspace is not removed from the system on deletion.\nDeleted collections and files can still be viewed in the application using \"Show deleted\" switch.\nThe goal is to prevent deleted data from being overwritten by users (not to create collections or files with the paths\nthat already existed in the system) and to allow administrators to perform special actions (to be performed only in exceptional special cases),\nlike undeletion or permanent removal, to revert accidental removal or creation of a collection or a file.\n\n===== Metadata forms\n\nUsers with write access to the collection can annotate collections,\ndirectories and files using _metadata forms_.\nFree text fields, like description and key words, can be entered freely,\nlinks to shared entities, like subjects, samples and projects, or\nvalues from a controlled vocabulary, like taxonomy or analysis type,\ncan be selected from a list:\n\nimage:docs/images/screenshots/Metadata form.png[Metadata form]\n\nThe shared metadata entities and controlled vocabularies cannot\nbe added via the user interface.\nThe \u003c\u003cRDF metadata\u003e\u003e API should be used for that instead.\n\n===== Metadata upload\n\nAnother way to annotate directories and files is by uploading a comma-separated values (CSV) file with metadata.\nThis section describes the CSV-based format used for bulk metadata uploads.\n\nThe file should be a valid CSV-file:\n\n* Records are separated with a ``,``-character.\n* Values may be enclosed in double quotes: ``\"value\"``.\n* In values that contain a double, the double quotes need to be escaped by replacing them with double double quotes:\n``Example \"quoted\" text`` becomes ``\"Example \"\"quoted\"\" text\"``.\n\nIn the metadata upload, lines starting with ``#`` are ignored. These lines are considered to be comments.\n\nThe file should have a header row containing the names of the columns.\nThe mandatory ``Path`` column is used for the file path. For the property columns, the name should match exactly the name of the property in the database.\n\nThe format of the values is as follows:\n\n* _Path_: the relative path to a file or a directory (relative to the collection or directory where the file is uploaded).\nUse ``./`` for the current directory or collection.\n* _Entity types_ can be referenced by ID or unique label.\n* Multiple values must be separated by the pipe symbol ``|``, e.g.,\nuse ``test|lab`` to enter the values ``test`` and ``lab``.\n\nThe file can be uploaded to the current directory by dropping the file in the metadata panel of the directory, or by selecting the metadata upload button. +\nBy hovering over the metadata upload button, a link to a _metadata template file_ becomes available:\n\nimage:docs/images/screenshots/Download metadata template.png[Download metadata template,role=\"th\",align=\"center\"]\n\nThe file describes the format in commented lines and\ncontains the available properties in the header row.\n\n.Example metadata file\n====\nAn example comma-separated values file with metadata about the current directory ``./``,\nwhich is annotated with a description and two key words (``sample`` and ``lab``),\nand the file ``test.txt`` which is linked to Subject 1 by the unique subject label\nand to the RNA-seq analysis type by the analysis type identifier (``O6-12``).\n[source, csv]\n----\nPath,Is about subject,Type of analysis,Description,Keywords\n./,,,Directory with samples,sample|lab,\ntest.txt,Subject 1,https://institut-curie.org/analysis#O6-12,,\n----\nThis specifies the table:\n[%header,format=csv]\n|===\nPath,Is about subject,Type of analysis,Description,Keywords\n./,,,Directory with samples,sample|lab\ntest.txt,Subject 1,https://institut-curie.org/analysis#O6-12,,\n|===\n====\n\n==== Metadata\n\nExplore metadata and find associated collections and files.\nimage:docs/images/screenshots/Metadata view.png[Metadata]\n\n\n\n=== Interfaces for accessing and querying data (API)\n\nThe data in Fairspace can be accessed via Application Programming Interfaces (APIs).\nThe user interfaces application uses those APIs, but also other programs can use them,\ne.g., for automated data uploading or for exporting data for further processing\nor for synchronisation with other systems.\n\n==== Authentication\n\nAll API endpoints require authentication via an authorisation header.\nTo enable WebDAV clients to connect to Fairspace, also so-called _Basic authentication_ is supported.\n\nFor secure authentication, it is strongly advised to use the _OpenID Connect (OIDC) / OAuth2_ workflow.\nThe user interface application also uses this workflow.\n\nWhen using the APIs in automated scripts, ensure that an account is used with only the required\nprivileges (conform the _principle of least privilege_). I.e., when an admin account is not needed, use a non-admin account.\nFor adding shared metadata, an account with\n_Add shared metadata_ role is required, see \u003c\u003cUploading metadata\u003e\u003e.\n\nWhen an action is done on behalf of a specific user,\ndo not use a service account or system account for the action directly, but obtain a token for that user first, e.g.,\nby using the https://www.keycloak.org/docs/latest/securing_apps/#impersonation[impersonation] feature of Keycloak.\nThat way the audit logging still captures which user did what.\n\n===== OpenID Connect (OIDC) / OAuth2 workflow\n\nFairspace supports OpenID Connect authentication via Keycloak.\nThe workflow for API access is roughly as follows.\n\n* The client authenticates with the token endpoint of the identity provider (Keycloak) and obtains a signed access token\n* The client uses the access token in the request header when connecting to the Fairspace API\n* Fairspace receives the request with the access token and validates if the token is valid,\n  using the public key of the identity provider.\n\nThe token endpoint of Keycloak supports refreshing the token if it is close to expiry.\nHowever, checking the token expiration and refreshing make the authentication logic quite complex.\n\nYou can either obtain a fresh token before every API request or use an existing library\nthat implements the authentication workflow.\nFor finding available client-side libraries,\ncheck the https://www.keycloak.org/docs/latest/securing_apps/[Securing applications and services guide] of Keycloak.\n\nFor use in scripts, it is advised to obtain a token for offline access, using the https://www.keycloak.org/docs/latest/server_admin/#_offline-access[Offline access] feature of OpenID Connect.\n\n.Code to obtain the OpenID Connect authorisation header (Python)\n[%collapsible]\n====\n[source, python]\n----\nimport logging\nimport os\nimport requests\n\nlog = logging.getLogger()\n\ndef fetch_access_token(keycloak_url: str = os.environ.get('KEYCLOAK_URL'),\n                       realm: str = os.environ.get('KEYCLOAK_REALM'),\n                       client_id: str = os.environ.get('KEYCLOAK_CLIENT_ID'),\n                       client_secret: str = os.environ.get('KEYCLOAK_CLIENT_SECRET'),\n                       username: str = os.environ.get('KEYCLOAK_USERNAME'),\n                       password: str = os.environ.get('KEYCLOAK_PASSWORD')) -\u003e str:\n    \"\"\"\n    Obtain access token from Keycloak\n    :return: the access token as string.\n    \"\"\"\n    params = {\n        'client_id': client_id,\n        'client_secret': client_secret,\n        'username': username,\n        'password': password,\n        'grant_type': 'password'\n    }\n    headers = {\n        'Content-type': 'application/x-www-form-urlencoded',\n        'Accept': 'application/json'\n    }\n    response = requests.post(f'{keycloak_url}/realms/{realm}/protocol/openid-connect/token',\n                             data=params,\n                             headers=headers)\n    if not response.ok:\n        log.error('Error fetching token!', response.json())\n        raise Exception('Error fetching token.')\n    data = response.json()\n    token = data['access_token']\n    log.info(f\"Token obtained successfully. It will expire in {data['expires_in']} seconds\")\n    return token\n\ndef auth():\n    return f'Bearer {fetch_access_token()}'\n----\n====\n\n.Code to obtain the OpenID Connect authorisation header (bash, curl)\n[%collapsible]\n====\nRequires the https://stedolan.github.io/jq/[jq] JSON parser.\n[source, bash]\n----\nfetch_access_token() {\n  curl -s \\\n    --data-urlencode \"client_id=${KEYCLOAK_CLIENT_ID}\" \\\n    --data-urlencode \"client_secret=${KEYCLOAK_CLIENT_SECRET}\" \\\n    --data-urlencode \"username=${KEYCLOAK_USERNAME}\" \\\n    --data-urlencode \"password=${KEYCLOAK_PASSWORD}\" \\\n    -d 'grant_type=password' \\\n    \"${KEYCLOAK_URL}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token\" | jq -r '.access_token'\n}\nACCESS_TOKEN=$(fetch_access_token)\n----\n====\n\n===== Basic authentication\n\nFor WebDAV client access and for a simpler authentication method\nduring testing, Fairspace also supports _Basic authentication_,\nwhich means that the ``base64`` encoded ``username:password`` string is sent in the ``Authorization`` header together with a prefix ``Basic ``.\n\nThis authentication method is considered to be less secure than\ntoken based authentication, because it requires scripts to have\na plain text password stored somewhere.\nAlso, users may have to retype their passwords when logging in, tempting them to choose less secure, easier to remember, passwords.\n\n.Code to generate the Basic authorisation header (Python)\n[%collapsible]\n====\n[source, python]\n----\nimport base64\nimport os\n\ndef auth():\n    username = os.environ.get('KEYCLOAK_USERNAME')\n    password = os.environ.get('KEYCLOAK_PASSWORD')\n    return f\"Basic {base64.b64encode(f'{username}:{password}'.encode()).decode()}\"\n----\n====\n\n.Code to generate the Basic authorisation header (bash)\n[%collapsible]\n====\n[source, bash]\n----\nAUTH_HEADER=\"Basic $(echo -n \"${KEYCLOAK_USERNAME}:${KEYCLOAK_PASSWORD}\" | base64)\"\n----\n====\n\n===== Examples\n\nIn the examples in this documentation, we assume one of both methods to be available.\n\nThis means for the Python examples that a function ``auth()`` should be implemented that returns the authorisation header value, see the examples above.\n\n[source, python]\n----\nimport os\nfrom requests import Response, Session\n\ndef auth():\n    \"\"\" Returns authorisation header\n    Replace this with an implementation from one of the sections above.\n    \"\"\"\n    pass\n\nserver_url = os.environ.get('FAIRSPACE_URL')\nheaders = {\n    'Authorization': auth()\n}\nresponse = Session().get(f'{server_url}/api/users/current', headers=headers)\nif not response.ok:\n    raise Exception(f\"Error fetching current user: {response.status_code} {response.reason}\")\nprint(response.json())\n----\n\nFor examples using curl, an authorisation header needs to be passed using the ``-H`` option.\n\nFor Basic authentication:\n[source, bash]\n----\nAUTH_HEADER=\"Basic $(echo -n \"${KEYCLOAK_USERNAME}:${KEYCLOAK_PASSWORD}\" | base64)\"\ncurl -i -H \"Authorization: ${AUTH_HEADER}\" \"${FAIRSPACE_URL}/api/users/current\"\n----\n\nFor OpenID Connect:\n[source, bash]\n----\n# ACCESS_TOKEN=...\nAUTH_HEADER=\"Bearer ${ACCESS_TOKEN}\"\ncurl -i -H \"Authorization: ${AUTH_HEADER}\" \"${FAIRSPACE_URL}/api/users/current\"\n----\n\n\n\n===== Automatic authentication in Jupyter Hub\n\nIn Jupyter Hub, users are automatically authenticated and can directly connect to the\nlocal API address without adding authentication headers.\n\n\n==== WebDAV\n\nA file storage API is exposed via the WebDAV protocol for accessing the file system via the web. It runs on ``/api/webdav/``.\n\nThis endpoint can be used by many file explorers,\nincluding Windows Explorer,\nand by tools like https://filezilla-project.org/[FileZilla] and https://cyberduck.io/[Cyberduck].\nUse ``\\https://fairspace.example.com/api/webdav/`` or\n``davs://fairspace.example.com/api/webdav/`` as location, with\n``fairspace.example.com`` replaced by the server name.\n\nAll visible collections in the system are exposed as top-level directories.\nCreating a top-level directory via WebDAV will result in an error message, see \u003c\u003cCreate collection or directory\u003e\u003e.\n\nThe {WebDAV}[Web-based Distributed Authoring and Versioning (WebDAV)] protocol allows users to operate on collections and files.\nFairspace exposes a WebDAV API for accessing the file systems, while restricting access to only the files accessible by the user.\n\nThe WebDAV API allows to upload and download files and to perform standard file operations such as copying or moving,\nas well as custom operations, such as collection lifecycle management\nand advanced data loss prevention features such as versioning and undeletion.\n\nBe aware that the _move_ operation moves both file content and all its metadata (e.g. linked metadata entities),\nwhereas _copy_ includes only the file content and standard webdav properties, like file size.\n\n===== Directory listing and path properties\n\n|===\n2+| ``PROPFIND /api/webdav/{path}``\n\n2+| _Request headers_:\n| `Depth`\n| When ``0`` only the information about the path is returned,\n  when ``1`` the contents of the directory is returned, if the path is a directory.\n| `Show-Deleted`\n| Include deleted paths when the value is `on`. (_Optional_)\n| `Version`\n| Specify a version number to request properties of a specific file version.\n  The first version has number `1`. If not specific, the current version is returned.\n| `With-Metadata-Links`\n| Include list of metadata entities that are linked to the resource, when value `true`.\n2+| _Request body_:\n2+| To include also custom Fairspace attributes in the response, like the collection description, send the following request body: +\n  ``\u003cpropfind\u003e\u003callprop /\u003e\u003c/propfind\u003e``\n|===\n\n====== Code examples\n\n.Check if path exists (Python)\n[%collapsible]\n====\n[source, python]\n----\nimport logging\nimport os\nfrom requests import Request, Response, Session\n\nlog = logging.getLogger()\n\nserver_url = os.environ.get('FAIRSPACE_URL')\n\ndef exists(path):\n    \"\"\" Check if a path exists\n    \"\"\"\n    headers = {\n        'Depth': '0',\n        'Authorization': auth()\n    }\n    session = Session()\n    req = Request('PROPFIND', f'{server_url}/api/webdav/{path}/', headers=headers, cookies=session.cookies)\n    response: Response = session.send(req.prepare())\n    return response.ok\n----\n====\n\n.Fetch directory listing (Python)\n[%collapsible]\n====\n[source, python]\n----\nimport logging\nimport os\nfrom requests import Request, Response, Session\nfrom xml.etree.ElementTree import fromstring\n\nlog = logging.getLogger()\n\nserver_url = os.environ.get('FAIRSPACE_URL')\n\ndef ls(path: str):\n    \"\"\" List contents of path\n    \"\"\"\n    headers = {\n        'Depth': '1',\n        'Authorization': auth()\n    }\n    session = Session()\n    req = Request('PROPFIND', f'{server_url}/api/webdav/{path}', headers=headers, cookies=session.cookies)\n    response: Response = session.send(req.prepare())\n    if not response.ok:\n        raise Exception(f\"Error fetching directory '{path}': {response.status_code} {response.reason}\")\n    tree = fromstring(response.content.decode())\n    for item in tree.findall('{DAV:}response'):\n        print(item.find('{DAV:}href').text)\n----\n====\n\n.Fetch directory listing (curl)\n[%collapsible]\n====\nRequires the http://xmlstar.sourceforge.net/[xmlstarlet] tool.\n[source, bash]\n----\ncurl -s -H \"Authorization: ${AUTH_HEADER}\" -X PROPFIND -H \"Depth: 1\" \"${FAIRSPACE_URL}/api/webdav/${path}\" -d '\u003cpropfind\u003e\u003callprop /\u003e\u003c/propfind\u003e' \\\n| xmlstarlet sel -T -t -m d:multistatus/d:response -v d:propstat/d:prop/d:displayname -n\n----\n====\n\n====== Example response\n.Example ``PROPFIND`` response\n[%collapsible]\n====\nExample response using ``PROPFIND`` on the root location ``https://fairspace.ci.fairway.app/api/webdav`` with ``Depth: 1`` and request body ``\u003cpropfind\u003e\u003callprop /\u003e\u003c/propfind\u003e``.\nAdding the ``\u003callprop /\u003e`` in the request results in custom Fairspace properties,\nlike the description (``ns1:comment``), to be included in the WebDAV response.\n[source, xml]\n----\n\u003c?xml version=\"1.0\" encoding=\"utf-8\" ?\u003e\n\u003cd:multistatus xmlns:ns1=\"https://fairspace.nl/ontology#\" xmlns:d=\"DAV:\"\u003e\n    \u003cd:response\u003e\n        \u003cd:href\u003e/api/webdav/\u003c/d:href\u003e\n        \u003cd:propstat\u003e\n            \u003cd:prop\u003e\n                \u003cd:getcontenttype\u003e\u003c/d:getcontenttype\u003e\n                \u003cd:getetag\u003e\"https://fairspace.ci.fairway.app/api/webdav\"\u003c/d:getetag\u003e\n                \u003cd:iscollection\u003eTRUE\u003c/d:iscollection\u003e\n                \u003cd:displayname\u003e\u003c/d:displayname\u003e\n                \u003cd:isreadonly\u003eTRUE\u003c/d:isreadonly\u003e\n                \u003cd:name\u003e\u003c/d:name\u003e\n                \u003cd:supported-report-set\u003e\u003c/d:supported-report-set\u003e\n                \u003cd:resourcetype\u003e\n                    \u003cd:collection/\u003e\n                \u003c/d:resourcetype\u003e\n            \u003c/d:prop\u003e\n            \u003cd:status\u003eHTTP/1.1 200 OK\u003c/d:status\u003e\n        \u003c/d:propstat\u003e\n    \u003c/d:response\u003e\n    \u003cd:response\u003e\n        \u003cd:href\u003e/api/webdav/Demonstration/\u003c/d:href\u003e\n        \u003cd:propstat\u003e\n            \u003cd:prop\u003e\n                \u003cns1:access\u003eWrite\u003c/ns1:access\u003e\n                \u003cns1:canRead\u003eTRUE\u003c/ns1:canRead\u003e\n                \u003cns1:userPermissions\u003ehttp://fairspace.ci.fairway.app/iri/user-iri Manage\n                \u003c/ns1:userPermissions\u003e\n                \u003cns1:accessMode\u003eRestricted\u003c/ns1:accessMode\u003e\n                \u003cns1:availableStatuses\u003eActive\u003c/ns1:availableStatuses\u003e\n                \u003cns1:canDelete\u003eFALSE\u003c/ns1:canDelete\u003e\n                \u003cns1:iri\u003ehttps://fairspace.ci.fairway.app/api/webdav/Demonstration\u003c/ns1:iri\u003e\n                \u003cns1:canWrite\u003eTRUE\u003c/ns1:canWrite\u003e\n                \u003cns1:ownedByCode\u003eDemo\u003c/ns1:ownedByCode\u003e\n                \u003cns1:canManage\u003eFALSE\u003c/ns1:canManage\u003e\n                \u003cns1:canUndelete\u003eFALSE\u003c/ns1:canUndelete\u003e\n                \u003cns1:workspacePermissions\u003ehttp://fairspace.ci.fairway.app/iri/workspace-iri\n                    Write\n                \u003c/ns1:workspacePermissions\u003e\n                \u003cns1:createdBy\u003ehttp://fairspace.ci.fairway.app/iri/user-iri\u003c/ns1:createdBy\u003e\n                \u003cns1:comment\u003eDemonstration collection\u003c/ns1:comment\u003e\n                \u003cns1:availableAccessModes\u003eRestricted\u003c/ns1:availableAccessModes\u003e\n                \u003cns1:ownedBy\u003ehttp://fairspace.ci.fairway.app/iri/workspace-iri\u003c/ns1:ownedBy\u003e\n                \u003cns1:status\u003eActive\u003c/ns1:status\u003e\n                \u003cd:getcreated\u003e2021-02-02T12:12:33Z\u003c/d:getcreated\u003e\n                \u003cd:creationdate\u003e2021-02-02T12:12:33Z\u003c/d:creationdate\u003e\n                \u003cd:getcontenttype\u003etext/html\u003c/d:getcontenttype\u003e\n                \u003cd:getetag\u003e\"https://fairspace.ci.fairway.app/api/webdav/Demonstration\"\u003c/d:getetag\u003e\n                \u003cd:iscollection\u003eTRUE\u003c/d:iscollection\u003e\n                \u003cd:displayname\u003eDemonstration collection\u003c/d:displayname\u003e\n                \u003cd:isreadonly\u003eFALSE\u003c/d:isreadonly\u003e\n                \u003cd:name\u003eDemonstration collection\u003c/d:name\u003e\n                \u003cd:supported-report-set\u003e\u003c/d:supported-report-set\u003e\n                \u003cd:resourcetype\u003e\n                    \u003cd:collection/\u003e\n                \u003c/d:resourcetype\u003e\n            \u003c/d:prop\u003e\n            \u003cd:status\u003eHTTP/1.1 200 OK\u003c/d:status\u003e\n        \u003c/d:propstat\u003e\n    \u003c/d:response\u003e\n\u003c/d:multistatus\u003e\n----\n====\n\n===== Create collection or directory\n\n|===\n2+| ``MKCOL /api/webdav/{path}``\n\n2+| Create collection or directory\n2+| _Request headers_:\n| `Owner`\n| Specify the identifier of the owner workspace when creating a collection.\n|===\n\n\n.Example create collection or directory (Python)\n[%collapsible]\n====\n[source, python]\n----\nimport logging\nimport os\nfrom requests import Request, Response, Session\n\nlog = logging.getLogger()\n\nserver_url = os.environ.get('FAIRSPACE_URL')\n\ndef mkdir(path: str, workspace_iri: str=None):\n    # Create directory\n    headers = {\n        'Authorization': auth()\n    }\n    if workspace_iri is not None:\n        headers['Owner'] = workspace_iri\n    req = Request('MKCOL', f'{server_url}/api/webdav/{path}/', headers=headers, cookies=self.session().cookies)\n    response: Response = Session().send(req.prepare())\n    if not response.ok:\n        raise Exception(f\"Error creating directory '{path}': {response.status_code} {response.reason}\")\n----\n====\n\n.Example create collection or directory (curl)\n[%collapsible]\n====\n[source, bash]\n----\n# Create a new collection, owned by workspace WORKSPACE_IRI\nNEW_COLLECTION=New collection\nWORKSPACE_IRI=http://fairspace.ci.fairway.app/iri/workspace-iri\ncurl -i -H \"Authorization: ${AUTH_HEADER}\" -X MKCOL -H \"Owner: ${WORKSPACE_IRI}\" \"${FAIRSPACE_URL}/api/webdav/${NEW_COLLECTION}\"\n# Create a new directory in the newly created collection\ncurl -i -H \"Authorization: ${AUTH_HEADER}\" -X MKCOL \"${FAIRSPACE_URL}/api/webdav/${NEW_COLLECTION}/Test directory\"\n----\n====\n\n===== Upload files\n\n|===\n2+| ``POST /api/webdav/{path}`` +\n  ``action=upload_files``\n\n2+| _Request data_:\n| ``action``\n| ``upload_files``\n| ``files``\n| Send files with the target file names as keys, see the examples below.\n|===\n\n.Example uploading files (Python)\n[%collapsible]\n====\n[source, python]\n----\nimport logging\nimport os\nfrom requests import Response, Session\n\nlog = logging.getLogger()\n\nserver_url = os.environ.get('FAIRSPACE_URL')\n\ndef upload_files(path: str, files: Dict[str, any]):\n    # Upload files\n    response: Response = Session().post(f'{server_url}/api/webdav/{path}/',\n            headers={'Authorization': auth()},\n            data={'action': 'upload_files'},\n            files=files)\n    if not response.ok:\n        raise Exception(f\"Error uploading files into '{path}': {response.status_code} {response.reason}\")\n----\n====\n\n.Example uploading files (curl)\n[%collapsible]\n====\n[source, bash]\n----\n# Upload files 'coffee.jpg' and 'coffee 2.jpg' to a collection\npath=\"new collection\"\ncurl -i -H \"Authorization: ${AUTH_HEADER}\" -X POST -F 'action=upload_files' -F 'coffee.jpg=@coffee.jpg' -F 'coffee 2.jpg=@coffee 2.jpg'\"${FAIRSPACE_URL}/api/webdav/${path}\"\n----\n====\n\n===== Copy and move a directory or file\n\n|===\n2+| ``COPY /api/webdav/{path}``\n\n2+| Copy a directory or file. Metadata linked to the file/directory is not copied.\n2+| _Request headers_:\n| ``Destination``\n| The destination path relative to the server, URL encoded, e.g., ``/api/webdav/collection%20abc/test.txt``.\n|===\n\n.Example copy path (curl)\n[%collapsible]\n====\n[source, bash]\n----\n# Copy 'Examples/Test dir/test 1.txt' to 'Examples/Test dir/test 2.txt'\npath=\"Examples/Test dir/test 1.txt\"\ntarget=\"/api/webdav/Examples/Test%20dir/test%202.txt\"\ncurl -i -H \"Authorization: ${AUTH_HEADER}\" -X COPY -H \"Destination: ${target}\" \"${FAIRSPACE_URL}/api/webdav/${path}\"\n----\n====\n\n|===\n2+| ``MOVE /api/webdav/{path}``\n\n2+| Move or rename a directory or file. Metadata linked to the file/directory is also moved along with it.\n2+| _Request headers_:\n| ``Destination``\n| The destination path relative to the server, URL encoded, e.g., ``/api/webdav/collection%20abc/test.txt``.\n|===\n\n.Example move path (curl)\n[%collapsible]\n====\n[source, bash]\n----\n# Move 'Examples/Test dir/test 1.txt' to 'Examples/Test dir/test 2.txt'\npath=\"Examples/Test dir/test 1.txt\"\ntarget=\"/api/webdav/Examples/Test%20dir/test%202.txt\"\ncurl -i -H \"Authorization: ${AUTH_HEADER}\" -X MOVE -H \"Destination: ${target}\" \"${FAIRSPACE_URL}/api/webdav/${path}\"\n----\n====\n\n===== Undelete a directory or file\n\n|===\n2+| ``POST /api/webdav/{path}`` +\n    ``action=undelete``\n\n2+| Undelete a directory or file\n2+| _Request headers_:\n| ``Show-deleted``\n| ``on``\n2+| _Request data_:\n| ``action``\n| ``undelete``\n|===\n\n.Example undelete path (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -i -H \"Authorization: ${AUTH_HEADER}\" -X POST -F \"action=undelete\" \"${FAIRSPACE_URL}/api/webdav/${path}\"\n----\n====\n\n===== Delete directory content\n\n|===\n2+| ``POST /api/webdav/{path}`` +\n    ``action=delete_all_in_directory``\n\n2+| Delete directory content\n2+| _Request data_:\n| ``action``\n| ``delete_all_in_directory``\n|===\n\n.Example delete all in directory (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -i -H \"Authorization: ${AUTH_HEADER}\" -X POST -F \"action=delete_all_in_directory\" \"${FAIRSPACE_URL}/api/webdav/${path}\"\n----\n====\n\n===== Revert to a file version\n\n|===\n2+| ``POST /api/webdav/{path}`` +\n``action=revert``\n\n2+| Restore a previous file version\n2+| _Request data_:\n| ``action``\n| ``revert``\n| ``version``\n| The version number to restore.\n|===\n\n.Example revert file version (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -i -H \"Authorization: ${AUTH_HEADER}\" -X POST -F \"action=revert\" -F \"version=${version}\" \"${FAIRSPACE_URL}/api/webdav/${path}\"\n----\n====\n\n===== Other collection actions\n\nOn collections, a number of actions is available.\nThese are not documented here in detail, but can be used from the user interface instead.\n\n[cols=\"1,1\"]\n|===\n| Action\n\n| Description\n\n| ``set_access_mode``\n| Change the access mode of a collection.\n| ``set_status``\n| Change the status of a collection.\n| ``set_permission``\n| Change the permission of the specified user or workspace on a collection.\n| ``set_owned_by``\n| Transfer ownership of a collection to another workspace.\n| ``unpublish``\n| Unpublish a published collection.\n|===\n\n\n==== SPARQL\nThe {SPARQL}[SPARQL] API is a standard API for querying RDF databases.\nThis endpoint is read-only and can be used for advanced search, analytics, data extraction, etc.\nIt is only accessible for users with the _canQueryMetadata_ role.\n\n|===\n| ``POST /api/rdf/query``\n\n| Execute SPARQL query\n| _Request body:_\n| The SPARQL query.\n|===\n\n.Example SPARQL query (Python)\n[%collapsible]\n====\nQuery for the first 500 samples.\n[source, python]\n----\nimport logging\nimport os\nfrom requests import Response, Session\n\nlog = logging.getLogger()\n\nserver_url = os.environ.get('FAIRSPACE_URL')\n\ndef query_sparql(query: str):\n    headers = {\n        'Authorization': auth(),\n        'Content-Type': 'application/sparql-query',\n        'Accept': 'application/json'\n    }\n    response: Response = Session().post(f\"{server_url}/api/rdf/query\", data=query, headers=headers)\n    if not response.ok:\n        raise Exception(f'Error querying metadata: {response.status_code} {response.reason}')\n    return response.json()\n\nquery_sparql(\"\"\"\n    PREFIX example: \u003chttps://example.com/ontology#\u003e\n    PREFIX fs: \u003chttps://fairspace.nl/ontology#\u003e\n\n    SELECT DISTINCT ?sample\n    WHERE {\n        ?sample a example:BiologicalSample .\n        FILTER NOT EXISTS { ?sample fs:dateDeleted ?anyDateDeleted }\n    }\n    # ORDER BY ?sample\n    LIMIT 500\n\"\"\")\n----\n====\n\n.Example SPARQL query (curl)\n[%collapsible]\n====\nQuery for the first 500 samples.\n[source, bash]\n----\ncurl -X POST -H \"Authorization: ${AUTH_HEADER}\" -H 'Content-Type: application/sparql-query' -H 'Accept: application/json' \\\n-d \"\n    PREFIX example: \u003chttps://example.com/ontology#\u003e\n    PREFIX fs: \u003chttps://fairspace.nl/ontology#\u003e\n\n    SELECT DISTINCT ?sample\n    WHERE {\n        ?sample a example:BiologicalSample .\n        FILTER NOT EXISTS { ?sample fs:dateDeleted ?anyDateDeleted }\n    }\n    # ORDER BY ?sample\n    LIMIT 500\n\" \\\n\"${FAIRSPACE_URL}/api/rdf/query\"\n----\n====\n\n\n==== RDF metadata\n\nFor reading and writing metadata to the database,\nthe ``/api/metadata`` endpoint supports a number of operations:\n\n* ``GET``: Retrieve metadata for a specified subject, predicate or object.\n* ``PUT``: Add metadata\n* ``PATCH``: Update metadata\n* ``DELETE``: Delete specified triples or all metadata linked to a subject.\n\nThe metadata is stored as subject-predicate-object triples.\nThe API supports several serialisation formats for sending :\n\n* {turtle}[Turtle] (``text/turtle``)\n* {json-ld}[JSON-LD] (``application/ld+json``, {jsonld-schema}[JSON schema])\n* {n-triples}[N-Triples] (``application/n-triples``)\n\nAfter any update, the metadata must be consistent with the data model, see \u003c\u003cData model and view configuration\u003e\u003e.\nIf an update would violate the data model constraints,\nthe request is rejected with a status ``400`` response, with a message indicating the violation.\n\n===== Uploading metadata\n\nShared metadata entities will in most cases come from other systems and will be added to Fairspace exclusively by an ETL process which will extract data from the laboratory and clinical systems, perform pseudonymization of identifiers, convert the metadata to some RDF-native format conforming the data model and send them to Fairspace.\n\nFairspace will validate the uploaded metadata against the constraints defined in the data model and returns a detailed error message in case of violations.\nThe validations include all the necessary type checks, referential consistency (validity of identifiers) checks, validation of mandatory fields, etc.\nIf any entity violates the constraints, the entire bulk upload will be rejected.\n\nThe ETL process will use a special technical account with the _Add shared metadata_ role.\nRegular users will not be able to add or modify shared metadata entities.\nRegular users can link files to shared metadata entities,\nsee \u003c\u003cMetadata forms\u003e\u003e and \u003c\u003cMetadata upload\u003e\u003e.\n\nIn addition to the main ETL workflow, data managers needs a possibility to add or modify certain properties of top-level metadata entities. This can be done using the RDF-based metadata API.\n\nA number of guidelines for uploading shared metadata:\n\n* Entities must have a type, a globally unique identifier,\n  and a unique label for the type. +\nIt is advised to use a unique identifier from an existing reference system for this purpose.\n* Because of the nature of linked data, it is advised\n  to add shared metatdata entities in an append-only fashion: only adding entities and avoid updating or deleting entities.\n* By nature of RDF, metadata is typically added on the level of triples.\nE.g., when adding a property ``dcat:keyword`` to a file, this will add a key word to the (possibly) already existing list of key words. +\nIf you want to completely replace (or remove)\na property from an entity, use the ``PATCH`` method instead of ``PUT``.\n\nExample metadata file in turtle format: ``testdata.ttl``:\n[source, turtle]\n----\n@prefix example: \u003chttps://example.com/ontology#\u003e .\n@prefix rdfs: \u003chttp://www.w3.org/2000/01/rdf-schema#\u003e .\n@prefix subject: \u003chttp://example.com/subjects#\u003e .\n@prefix file: \u003chttp://example.com/api/webdav/\u003e .\n@prefix gender: \u003chttp://hl7.org/fhir/administrative-gender#\u003e .\n@prefix ncbitaxon: \u003chttps://bioportal.bioontology.org/ontologies/NCBITAXON/\u003e .\n@prefix dcat: \u003chttp://www.w3.org/ns/dcat#\u003e .\n\nsubject:s1 a example:Subject ;\n    rdfs:label \"Subject 1\" ;\n    example:isOfSpecies ncbitaxon:9606 .\n\nfile:coll1\\/coffee.jpg\n    dcat:keyword \"fairspace\", \"java\" ;\n    example:aboutSubject example:s1 .\n----\n\n.Example uploading metadata file using Python.\n[%collapsible]\n====\n[source, python]\n----\nimport logging\nimport os\nfrom requests import Response, Session\n\nlog = logging.getLogger()\n\nserver_url = os.environ.get('FAIRSPACE_URL')\n\nwith open('testdata.ttl') as testdata:\n    response: Response = Session().put(f\"{server_url}/api/metadata/\",\n        data=testdata.read(),\n        headers={\n            'Authorization': auth(),\n            'Content-type': 'text/turtle'\n        })\n    if not response.ok:\n        raise Exception(f\"Error uploading metadata: {response.status_code} {response.reason}\")\n----\n====\n\n.Example uploading metadata file (curl).\n[%collapsible]\n====\n[source, bash]\n----\ncurl -v -X PUT -H \"Authorization: Basic $(echo -n \"${KEYCLOAK_USERNAME}:${KEYCLOAK_PASSWORD}\" | base64)\" \\\n  -H \"Content-type: text/turtle\" --data @testdata.ttl \"${FAIRSPACE_URL}/api/metadata/\"\n----\n====\n\n===== API specification\n\n|===\n3+| ``GET /api/metadata``\n\n3+| Retrieve metadata\n3+| _Parameters:_\n| ``subject``\n| string\n| IRI of the subject to filter on.\n| ``predicate``\n| string\n| The predicate to filter on, not required.\n| ``object``\n| string\n| The object to filter on, not required.\n| ``includeObjectProperties``\n| boolean\n| If set, the response will include several properties for the included objects.\n  The properties to be included are marked with ``fs:importantProperty`` in the vocabulary.\n3+| _Response:_\n3+| Returns serialised triples matching the query parameters.\n|===\n\n.Example of fetching metadata in turtle format (curl)\n[%collapsible]\n====\nRequest metadata for a subject 'a'.\n[source, bash]\n----\ncurl -G -H \"Accept: text/turtle\" \\\n--data-urlencode \"subject=a\" \\\n--data-urlencode \"withValueProperties=true\" \\\n\"http://localhost:8080/api/metadata/\"\n----\n====\n\n.Example of fetching metadata in json-ld format (curl)\n[%collapsible]\n====\nRequest metadata for the triple with subject 'a', predicate 'b' and object 'c'.\n[source, bash]\n----\ncurl -G -H \"Accept: application/ld+json\" \\\n--data-urlencode \"subject=a\" \\\n--data-urlencode \"predicate=b\" \\\n--data-urlencode \"object=c\" \\\n--data-urlencode \"withValueProperties=true\" \\\n\"http://localhost:8080/api/metadata/\"\n----\n====\n\n|===\n3+| ``PUT /api/metadata``\n\n3+| Add metadata. Existing metadata is left untouched.\n    The data must be consistent with the data model after the update (see \u003c\u003cData model and view configuration\u003e\u003e),\n    otherwise ``400`` is returned.\n  Only available for users with _Add shared metadata_ role.\n3+| _Parameters:_\n| ``doViewsUpdate``\n| boolean\n| Flag to switch on and off materialized views refresh once metadata updated. The materialized views are used to speed up the metadata search. ``true`` by default. Recommended to set to ``false`` for batch update (e.g. for metadata uploading via pipeline). (_Optional_)\n3+| _Request body:_\n3+| Serialised RDF triples.\n|===\n\n.Example of adding metadata in turtle format (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X PUT -H \"Content-type: text/turtle\" -d \\\n'\n@prefix example: \u003chttps://example.com/ontology#\u003e .\n@prefix rdfs: \u003chttp://www.w3.org/2000/01/rdf-schema#\u003e .\n@prefix test: \u003chttps://test.com/ontology#\u003e .\nexample:Study_001 a test:Study ;\n    rdfs:label \"Project study #001\" ;\n    test:studyIdentifier \"STUDY-001\" ;\n    test:studyTitle \"Project study #001\" ;\n    test:studyDescription \"This is a description of the study.\" .\n' \\\n\"http://localhost:8080/api/metadata/\"\n----\n====\n\n|===\n3+| ``PATCH /api/metadata/``\n\n3+| Update metadata.\n    Any existing metadata for a given subject/predicate combination will be overwritten with the provided values.\n    The data must be consistent with the data model after the update (see \u003c\u003cData model and view configuration\u003e\u003e),\n    otherwise ``400`` is returned.\n  Only available for users with _Add shared metadata_ role.\n3+| _Parameters:_\n| ``doViewsUpdate``\n| boolean\n| Flag to switch on and off materialized views refresh once metadata updated. The materialized views are used to speed up the metadata search. ``true`` by default. Recommended to set to ``false`` for batch update (e.g. for metadata uploading via pipeline). (_Optional_)\n3+| _Request body:_\n3+| Serialised RDF triples.\n|===\n\n.Example of updating metadata in turtle format (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X PATCH -H \"Content-type: text/turtle\" -d \\\n'\n@prefix example: \u003chttps://example.com/ontology#\u003e .\n@prefix test: \u003chttps://test.com/ontology#\u003e .\nexample:Study_001 a test:Study ;\n    test:studyTitle \"Updated project study #001\" ;\n' \\\n\"http://localhost:8080/api/metadata/\"\n----\n====\n\n|===\n3+| ``DELETE /api/metadata/``\n\n3+| Delete metadata.\nIf a request body is provided, the triples specified in the body will be deleted.\nOtherwise, the subject specified in the subject parameter will be marked as deleted.\nPlease note that the subject will still exist in the database.\nOnly available for users with _Add shared metadata_ role.\n3+| _Parameters:_\n| ``subject``\n| string\n| The subject to filter on. (_Optional_)\n| ``doViewsUpdate``\n| boolean\n| Flag to switch on and off materialized views refresh once metadata updated. The materialized views are used to speed up the metadata search. ``true`` by default. Recommended to set to ``false`` for batch update (e.g. for metadata uploading via pipeline). (_Optional_)\n3+| _Request body:_\n3+| Serialised RDF triples. (_Optional_)\n|===\n\n.Example of deleting triples in turtle format (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X DELETE -H \"Content-Type: text/turtle\" -d \\\n'\n@prefix example: \u003chttps://example.com/ontology#\u003e .\n@prefix test: \u003chttps://test.com/ontology#\u003e .\nexample:Study_001 a test:Study ;\n    test:studyDescription \"This is a description of the study.\" .\n' \\\n\"http://localhost:8080/api/metadata/\"\n----\n====\n\n.Example of marking an entity as deleted (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X DELETE -G --data-urlencode \"subject=https://example.com/ontology#tpe1\" \"http://localhost:8080/api/metadata/\"\n----\n====\n\n\n==== Metadata views\n\nMetadata views endpoint used for metadata-based search.\n\n|===\n| ``GET /api/views/``\n\n| List all views with available columns per each view.\n|===\n\n.Example list view (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H \"Accept: application/json\" \"http://localhost:8080/api/views/\"\n----\n====\n\n|===\n3+| ``POST /api/views/``\n\n3+| Fetch page of rows of a view matching the request filters.\n3+| _Parameters:_\n| ``view``\n| string\n| Name of the view.\n| ``filters``\n2+| List of filters, based on available facets and their values.\nEach filter has to contain a \"field\" property, matching the name of a facet, and list of values to filter on.\n| ``page``\n| integer\n| Requested page\n| ``size``\n| integer\n| Page size\n|===\n\n.Example fetching page of view rows (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X POST -H 'Content-type: application/json' -H 'Accept: application/json' -d \\\n'{\n  \"view\":\"Resource\",\n  \"filters\":[\n    {\n      \"field\":\"Resource_type\",\n      \"values\":[\"https://fairspace.nl/ontology#Collection\"]\n    }\n  ],\n  \"page\":1,\n  \"size\":100\n}' \\\n\"http://localhost:8080/api/views/\"\n----\n====\n\n|===\n3+| ``POST /api/views/count``\n\n3+| Count rows of a view matching request filters. If `maxDisplayCount` configured in the `views.yaml` for a view, then the count for the view is limited by this value if total count exceeds it. Otherwise, the total count is returned.\n3+| _Parameters:_\n| ``view``\n| string\n| Name of the view.\n| ``filters``\n2+| List of filters, based on available facets and their values.\nEach filter has to contain a \"field\" property, matching the name of a facet, and list of values to filter on.\n|===\n\n.Example counting view rows (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X POST -H 'Content-type: application/json' -H 'Accept: application/json' -d \\\n'{\n  \"view\":\"Resource\",\n  \"filters\":[\n    {\n      \"field\":\"Resource_type\",\n      \"values\":[\"https://fairspace.nl/ontology#Collection\"]\n    }\n  ]\n}' \\\n'http://localhost:8080/api/views/count'\n----\n====\n\n|===\n| ``GET /api/views/facets``\n\n| List all facets with available values per each facet.\n|===\n\n.Example retrieving facets with values (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H \"Accept: application/json\" \"http://localhost:8080/api/views/facets\"\n----\n====\n\n\n==== Text search\n\nSearch endpoint used for text search on labels or comments.\n\n|===\n3+| ``POST /api/search/files``\n\n3+| Find files, directories or collections based on a label or a comment.\n3+| _Parameters:_\n| ``query``\n| string\n| Text fragment to search on.\n| ``parentIRI``\n| string\n| IRI of the parent directory or collection to limit the search area.\n\n3+| _Response_\n3+| Object in JSON format, with `query` and `results` properties.\nResults contain a list of files (and/or directories, collections) with the following properties:\n| ``id``\n| string\n| File (or directory) identifier (IRI).\n| ``label``\n| string\n| File (or directory) name.\n| ``type``\n| string\n| Type of the resource as defined in the vocabulary, e.g. \"https://fairspace.nl/ontology#File\", \"https://fairspace.nl/ontology#Directory\"\n| ``comment``\n| string\n| File (or directory) description. Optional.\n|===\n\n.Example text search (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X POST -H 'Content-type: application/json' -H 'Accept: application/json' -d \\\n'{\n  \"query\":\"test folder\",\n  \"parentIRI\":\"http://localhost:8080/api/webdav/dir1\"\n}' \\\n'http://localhost:8080/api/search/files'\n----\n====\n\n.Example text search response\n[%collapsible]\n====\n[source, json]\n----\n{\n  \"results\": [\n    {\n      \"id\": \"https://fairspace.example.com/api/webdav/col1/test\",\n      \"label\": \"test\",\n      \"type\": \"https://fairspace.nl/ontology#File\",\n      \"comment\": \"Description of the test file from col1.\"\n    },\n    {\n      \"id\": \"https://fairspace.example.com/api/webdav/col2/new_test_folder\",\n      \"label\": \"new_test_folder\",\n      \"type\": \"https://fairspace.nl/ontology#Directory\",\n      \"comment\": null\n    }\n  ],\n  \"query\": \"test\"\n}\n----\n====\n\n|===\n3+| ``POST /api/search/lookup``\n\n3+| Metadata entities lookup search by entity labels or description.\n3+| _Parameters:_\n| ``query``\n| string\n| Text fragment to search on.\n| ``resourceType``\n| string\n| Type of the entity in request.\n|===\n\n.Example lookup search (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X POST -H 'Content-type: application/json' -H 'Accept: application/json' -d \\\n'{\n  \"query\":\"test\",\n  \"resourceType\":\"https://example.com/ontology#TumorPathologyEvent\"\n}' \\\n'http://localhost:8080/api/search/lookup'\n----\n====\n\n\n==== Workspace management\n\nOperations on workspace entities.\n\n|===\n2+| ``GET /api/workspaces/``\n\n2+| List all available workspaces.\n2+| _Response_ contains the following data:\n| ``iri``\n| Unique workspace IRI.\n| ``code``\n| Unique workspace code.\n| ``title``\n| Workspace title.\n| ``managers``\n| List of  workspace managers.\n| ``summary``\n| Short summary on the workspace - how many collections and how many users it has.\n| ``canCollaborate``\n| If a current user is added to the workspace as a collaborator.\n| ``canManage``\n| If a current user is a workspace manager.\n|===\n\n.Example of listing available workspaces (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H \"http://localhost:8080/api/workspaces/\"\n----\n====\n\n|===\n3+| ``PUT /api/workspaces/``\n\n3+| Add a workspace. Available only to administrators.\n3+| _Parameters:_\n|``code``\n|string\n|Unique workspace code.\n3+| _Response:_\n3+| Response contains the workspace name and newly assigned IRI.\n|===\n\n.Example of adding a workspace (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X PUT -H \"Content-type: application/json\" -d '{\"name\": \"test workspace\"}' \"http://localhost:8080/api/workspaces/\"\n----\n====\n\n|===\n3+| ``DELETE /api/workspaces/``\n\n3+| Delete a workspace. Available only to administrators.\n3+| _Parameters:_\n| ``workspace``\n| string\n| Workspace IRI (URL-encoded).\n|===\n\n.Example of deleting a workspace (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X DELETE --data-urlencode \"workspace=http://fairspace.com/iri/123\" \"http://localhost:8080/api/workspaces/\"\n----\n====\n\n===== Workspace users\n\n|===\n3+| ``GET /api/workspaces/users/``\n\n3+| List all workspace users with workspace roles.\n3+| _Parameters:_\n| ``workspace``\n| string\n| Workspace IRI (URL-encoded).\n3+| _Response:_\n3+| Response contains list of workspace users with their workspace roles.\n|===\n\n.Example of listing workspace users (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -G --data-urlencode \"workspace=http://fairspace.com/iri/123\" \"http://localhost:8080/api/workspaces/users/\"\n----\n====\n\n|===\n3+| ``PATCH /api/workspaces/users/``\n\n3+| Assign a workspace role to a user (``Member`` or ``Manager``) or revoke\na workspace role (by assigning role ``None``).\n3+| _Parameters:_\n| ``workspace``\n| string\n| Workspace IRI.\n| ``user``\n| string\n| User IRI\n| ``role``\n| string\n| ``None`` (to remove), ``Member`` or ``Manager``\n|===\n\n.Example of updating workspace users (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X PATCH -H \"Content-type: application/json\" -d '{\"workspace\":\"http://fairspace.com/iri/123\",\"user\":\"http://fairspace.com/iri/456\",\"role\":\"Member\"}' \"http://localhost:8080/api/workspaces/users/\"\n----\n====\n\n\n==== Users and permissions\n\n|===\n| ``GET /api/users/``\n\n| List all organisation users.\n| _Response:_\n| Returns list of users with user's unique ID, name, email, username and user's organisation-level permissions:\nif a user is an administrator, super-administrator or can view public metadata, view public data or add shared metadata.\n|===\n\n.Example listing users (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H 'Accept: application/json' 'http://localhost:8080/api/users/'\n----\n====\n\n|===\n3+| ``PATCH /api/users/``\n\n3+| Update user roles.\n3+| _Parameters:_\n| ``id``\n| string\n| UUID of the user for which roles will be updated.\n| \"role name\"\n| boolean\n| Role name is any of ``isAdmin``, ``canViewPublicData``, ``canViewPublicMetadata``, ``canAddSharedMetadata``\nor ``canQueryMetadata``. The value determines whether the user has the role or not.\n|===\n\n.Example updating user roles (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X PATCH -H \"Accept: application/json\" -H \"Content-Type: application/json\" -d \\\n'{\n  \"id\": \"123e4567-e89b-12d3-a456-426614174000\",\n  \"canViewPublicData\": false,\n  \"canViewPublicMetadata\": true\n}' \\\n\"http://localhost:8080/api/users/\"\n----\n====\n\n|===\n| ``GET /api/users/current``\n\n| Get current user.\n| _Response:_\n| Returns current user's unique ID, name, email, username and user's organisation-level permissions:\nif the user is an administrator, super-administrator or can view public metadata,\nview public data or add shared metadata.\n|===\n\n.Example getting current user (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H \"Accept: application/json\" \"http://localhost:8080/api/users/current\"\n----\n====\n\n|===\n| ``POST /api/users/current/logout``\n\n| logout the current user.\n|===\n\n.Example logging out (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X POST \"http://localhost:8080/api/users/current/logout\"\n----\n====\n\n\n==== Configuration endpoints\n\n===== Vocabulary\n\nThe vocabulary contains a description of the structure of the metadata.\nIt contains the types of entities that can be created, along with the data types for the fields.\nIt is stored in {SHACL}[SHACL] format.\n\n|===\n| ``GET /api/vocabulary/``\n\n| Retrieve a representation of the vocabulary.\n|===\n\n.Example fetching the vocabulary in turtle format (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H 'Accept: text/turtle' 'http://localhost:8080/api/vocabulary/'\n----\n====\n\n.Example fetching the vocabulary in json-ld format (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H 'Accept: application/json+ld' 'http://localhost:8080/api/vocabulary/'\n----\n====\n\n===== Features\n\n|===\n| ``GET /api/features/``\n\n| List available application features.\n|===\n\nResponse contains list of additional features that are currently available in the application.\n\n.Example listing features (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H 'Accept: application/json' 'http://localhost:8080/api/features/'\n----\n====\n\n===== Icons (SVG)\n\n|===\n| ``GET /api/iconsvg/{icon_name}``\n\n| Retrieve an SVG icon by name.\n|===\n\nResponse contains an SVG icon (if configured).\n\n.Example retrieving icon (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H 'Accept: image/svg+xml' 'http://localhost:8080/api/iconsvg/{icon_name}'\n----\n====\n\n===== Services\n\n|===\n| ``GET /api/services/``\n\n| List linked services.\n|===\n\nResponse contains list of external services linked to Fairspace,\ne.g. JupyterHub, cBioPortal, etc with their configuration details: name, url and icon name that can be used to retrieve\nthe icon (using ``GET /api/iconsvg/{icon_name}``).\n\n.Example listing services (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H 'Accept: application/json' 'http://localhost:8080/api/services/'\n----\n====\n\n===== Server configuration\n\n|===\n| ``GET /api/config``\n\n| View server configuration properties.\n|===\n\nResponse contains a list of server configuration properties,\ncurrently limited to a max file size for uploads.\n\n.Example listing properties (curl)\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H 'Accept: application/json' 'http://localhost:8080/api/config/'\n----\n====\n\n===== External storages\n\n|===\n| ``GET /api/storages/``\n\n| List linked data storages.\n|===\n\nResponse contains list of external data storages linked to Fairspace.\n\n.Example listing external storages using curl\n[%collapsible]\n====\n[source, bash]\n----\ncurl -H 'Accept: application/json' 'http://localhost:8080/api/storages/'\n----\n====\n\n===== Maintenance\n\n|===\n2+| ``POST /api/maintenance/reindex``\n\n2+| Recreate the view database from the RDF database.\n\nStarts an asynchronous task to clean the PostgreSQL database with the data used for the metadata views, and to repopulate the database with the data from the RDF database.\n\nThis can be used after a change in the data model or view configuration to ensure\nthat all data is properly indexed.\n\nOnly available when the application is configured with ``viewDatabase.enabled: true``.\n\nOnly allowed for administrators.\n2+| _Response:_\n| ``204``\n| Asynchronous task to recreate the index has started.\n| ``403``\n| Operation not allowed. The current user is not an administrator.\n| ``409``\n| Maintenance (reindexing or compacting) is already in progress.\n| ``503``\n| Service not available. This means that the application is configured not to use a view database.\n|===\n\n|===\n2+| ``POST /api/maintenance/compact``\n\n2+| Compact the Jena TDB database files.\n\nJena database files grow fast when using transactions. This operation will compact the database files to reduce their size. If data is inserted using many small transactions the files will be reduced to 10-20% of their original size.\n\nOnly allowed for administrators.\n2+| _Response:_\n| ``204``\n| Asynchronous task to compact Jena files has started.\n| ``403``\n| Operation not allowed. The current user is not an administrator.\n| ``409``\n| Maintenance (reindexing or compacting) is already in progress.\n| ``503``\n| Service not available. This means that the application is configured not to use a view database.\n|===\n\n\n.Example of compacting Jena using curl\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X POST 'http://localhost:8080/api/maintenance/compact'\n----\n====\n\n|===\n2+| ``GET /api/maintenance/status``\n\n2+| Get the status of maintenance tasks.\n\nIt is not possible to run more than one maintenance task at the same time. If you start a task while another task is running, the new task will be rejected. If you want to know in advance whether a task is running, you can use this endpoint.\n\nA text is return\n\n2+| _Response:_\n| ``200``\n| Returns \"active\" or \"inactive\"\n| ``403``\n| Operation not allowed. The current user is not an administrator.\n| ``409``\n| Reindexing is already in progress.\n| ``503``\n| Service not available. This means that the application is configured not to use a view database.\n|===\n\n\n.Example of getting the maintenance status using curl\n[%collapsible]\n====\n[source, bash]\n----\ncurl -X POST 'http://localhost:8080/api/maintenance/status'\n----\n====\n\n=== External file system integration\n\nAs Fairspace supports the \u003c\u003cWebDAV\u003e\u003e protocol, it can be configured to connect to external data storages that implement a WebDAV interface.\nAn overview of external files is integrated into Fairspace user interface. Currently, a read-only interaction is supported.\nUsers can browse through the external file system, read the data and metadata (e.g. creation date, description).\nFiles from the external storage will be also made available for analysis in Jupyter Hub.\n\n\n==== Access policy\n\nAccess policies differ between systems. To avoid inconsistencies, permissions validation and management are expected to be under\ncontrol of the external storage system. Each storage component is responsible for its own policy and needs to perform\nthe required checks to ensure that users only get to see the data they are supposed to see.\n\nIt is assumed that a user requesting files from a storage using WebDAV has at least \"read\" access to all the files included in the WebDAV response.\nAccess can be further limited by using a custom `access` property. If a value of this property on a resource is set to \"List\",\nthe resource's metadata will be readable, but it will not be possible to read the resource's content.\n\nAnother assumption is that the Fairspace client can authenticate in the external storage via the same Keycloak and the same realm\nas configured for Fairspace, so that the same bearer token can be used for all storages.\nSee the \u003c\u003cAuthentication\u003e\u003e section for more information.\n\n==== API specification and supported parameters\n\nA subset of default WebDAV properties is used and displayed as a resource metadata in the Fairspace user interface.\nThese properties are presented in the table below.\n\n|===\n| WebDAV property | Description\n\n| ``DAV:creationdate``\n| Creation date\n| ``DAV:iscollection``\n| Flag determining whether a resource is a file or directory\n| ``DAV:getlastmodified``\n| Last modification date\n| ``DAV:getcontentlength``\n| Size of the file (0 for directories)\n|===\n\nThere is also a set of custom Fairspace properties, some of which are required to be returned from the WebDAV request.\n\n|===\n| WebDAV property | Description\n\n| ``iri``\n| IRI of the resource. Required.\n| ``createdBy``\n| Id of a user that created the resource.\n| ``comment``\n| Resource description.\n| ``access``\n| By default, users are granted ``Read`` access to the resource returned from WebDAV endpoint.\nOther supported value is ``List``, which means that users can see the resource and its metadata, but cannot read its content.\n| ``metadataEntities``\n| List of IRIs in a form of comma-separated string. IRIs represent all metadata entities linked to the resource.\nIf the IRI matches a metadata entity stored in Fairspace, such an entity will be displayed in the user interface.\n|===\n\nIt is also supported to specify any other custom property in the WebDAV response body, as WebDAV responses are easily extendable.\nAll these properties (if not specifically marked as excluded in Fairspace), will be displayed in the user interface\nin a form of key-value pairs.\n\n==== Text search in external storages\n\nText based search on external file system can be enabled in the Fairpsace user interface,\nif the external system exposes a search endpoint, following the specification from the \u003c\u003cText search\u003e\u003e section.\nTo enable finding files based on name or description, ``searchUrl`` has to be specified in the storage configuration.\n\n==== Configuration\n\nMultiple external storages can be configured simultaneously. A list of configuration parameters is presented below.\n\n|===\n| Parameter | Description\n\n| ``name``\n| Unique name of the storage.\n| ``label``\n| String to be used as a display name of the storage.\n| ``url``\n| WebDAV endpoint to connect to.\n| ``searchUrl``\n| Optional search endpoint URL. If specified, a text based search on file name or description will be enabled in the user interface.\n| ``rootDirectoryIri``\n| Optional IRI of the root directory. If not specified, ``url`` will be used as a default root directory.\n|===\n\nSample configuration of storages in YAML format:\n[source, bash]\n----\nstorages:\n  exStorage1:\n    name: exStorage1\n    label: \"External storage 1\"\n    url: https://exstorage1/api/webdav\n    searchUrl: https://exstorage1/api/search/\n    rootDirectoryIri: http://ex1/api/webdav/\n  exStorage2:\n    name: exStorage2\n    label: \"External storage 2\"\n    url: https://exstorage2/api/webdav\n----\n\n=== Multi-Fairspace metadata views integration\n\nFairspace works with a single data model configured. However, if there is a need to have data from multiple models,\nrepresenting multiple domains that would be combined in single user interface, it is possible to integrate multiple Fairspace instances together.\n\n\nThe main Fairspace instance can be configured as a view interface of metadata from another (external) Fairspace instance.\nExternal Fairspace can have a different metadata model configured than the main Fairspace.\nAn important precondition is that the external instance has to be connected to the same Keycloak realm as the main instance.\n\n\nExternal metadata can be searched and browsed in the same way as the internal metadata views. When configured, there is an extra page,\nseparate from internal metadata views, that allows to explore external metadata.\nCurrently, cross-instance metadata search is not supported.\n\n\nFor the external metadata pages, the tables and columns are created basing on the views configuration specified in that instance configuration.\n\n\n==== Access policy\n\nConnection to the same Keycloak realm allows to authenticate a user in all integrated Fairspace instances with a single set of login credentials (single sign-on).\nEach Fairspace instance is responsible for controlling the access to its own metadata and perform\nthe required checks to ensure that users only get to see metadata of an instance, if has a view public metadata role assigned within that instance.\n\n==== Configuration\n\nMultiple external Fairspace metadata pages can be configured simultaneously. A list of configuration parameters is presented below.\n\n|===\n| Parameter | Description\n\n| ``name``\n| Unique name of the metadata source.\n| ``label``\n| String to be used as a display name of the metadata source.\n| ``url``\n| Fairspace instance to connect to. If the url is not specified, the metadata source will be treated as the internal one.\n\n*Important!* There should only be a single configuration of internal metadata (only the first one will not be ignored).\n| ``icon-name``\n| Name of an icon configured in the \"icons\" section of values.yaml file. If the name is not specified, there will be a default icon used.\n|===\n\nSample configuration of *external* metadata sources in YAML format:\n[source, bash]\n----\nfairspace:\n    ...\n    metadata-sources:\n      internal:\n        name: internal\n        label: \"Internal metadata\"\n        icon-name: \"icon-internal-metadata\"\n      additionalMetaSource1:\n        name: metadataSource1\n        label: \"Test metadata 1\"\n        url: https://fairspace-test1/api/\n        icon-name: \"icon-1\"\n      metaSource2:\n        name: metadataSource2\n        label: \"Test metadata 2\"\n        url: https://fairspace-test2/api/\n----\n\nConfiguration of gateway redirection in `values.yaml`:\n[source, bash]\n----\npluto:\n  ...\n  backends:\n    storageRoutes:\n      - id: additional-metadata-domain1\n        uri: https://fairspace-test1\n        predicates:\n        - Path=/api/metadata-sources/metadataSource1/**\n        filters:\n        - RewritePath=/api/metadata-sources/metadataSource1/(?\u003csegment\u003e(views|vocabulary|metadata).*),/api/$\\{segment}\n        - RewriteResponseHeader=Set-Cookie, ^([^=]+)=, DO_$1=\n      - id: additional-metadata-domain2\n        uri: https://fairspace-test2\n        predicates:\n        - Path=/api/metadata-sources/metadataSource2/**\n        filters:\n        - RewritePath=/api/metadata-sources/metadataSource2/(?\u003csegment\u003e(views|vocabulary|metadata).*),/api/$\\{segment}\n        - RewriteResponseHeader=Set-Cookie, ^([^=]+)=, DO_$1=\n----\n\nwhere **RewriteResponseHeader** is important filter that needs to be added not to overwrite an existing browser session, which would lead to authorization errors.\n\nConfiguration above gets transformed to the following Spring Cloud Gateway config in the `application.yaml` configuration of Pluto:\n\n[source, bash]\n----\nspring:\n  cloud:\n    gateway:\n      routes:\n      - id: additional-metadata-domain1\n        uri: https://fairspace-test1\n        predicates:\n        - Path=/api/metadata-sources/metadataSource1/**\n        filters:\n        - RewritePath=/api/metadata-sources/metadataSource1/(?\u003csegment\u003e(views|vocabulary|metadata).*),/api/$\\{segment}\n        - RewriteResponseHeader=Set-Cookie, ^([^=]+)=, DO_$1=\n      - ...\n----\n\n== Structure and terminology\n\nIn this section we describe in detail the main concepts and components of the\nFairspace data repository and how they relate to each other.\n\nThe core entities of the data repository are:\n\n* _Users_: individual users in the organisation, looking for data,\ncontributing to data collections or managing data.\n* _Workspaces_ (for projects, teams): entities in the system linked, representing a group of users,\nto organise data collections and data access.\n* _Collections_: entities in the system to group data files.\nThese are the minimal units of data for data access and data modification rules.\n* _Files_: The smallest units of data that the system processes.\nFiles always belong to a single collection.\nFiles can be added, changed and deleted, but not in all collection states.\nChanging a file creates a new version.\nAccess to a file is based on access to the collection the file belongs to.\nFiles can be organised in _Directories_, which we will leave out of most descriptions for brevity.\n\nimage:docs/images/diagrams/Collections access model.png[Diagram]\n\nThe diagram above sketches the relevant entities and actors.\nThe basic structure consists of users, workspaces, collections and files as represented in the system.\nCollections are the basic units of data access management.\nA collection is owned by a workspace.\nThe responsibility for a collection is organised via the owner workspace:\nmembers of the owner workspace can be assigned as editors or managers of the collection.\nThis reflects the situation where in an organisation, a data collection belongs to a project or a research team.\nThis way the workspace represents the organisational unit that is responsible for a number of data collections\n(e.g., a research team or project).\nData can be shared with other workspaces or individual users (for reading)\nand ownership may be transferred to another workspace\n(e.g., in the case the workspace is temporary, or when the organisation changes).\n\nFairspace provides a _data catalogue_, containing all the metadata,\nwhich is visible for all users with catalogue access (_View public metadata_).\nUsers with metadata write access (_Add shared metadata_) can add metadata to the catalogue.\nPreferably this is done by an automated process that ensures the consistency\nof the metadata and uniqueness of metadata entities.\nMetadata on collection and file level is protected by the access policy of the collections.\n\n_User administration_ is organised in an external component ([Keycloak]),\nbut user permissions are stored in Fairspace.\nA back end application is responsible for storing the data and metadata,\nand for providing APIs for securely retrieving and adding data and metadata using standard data formats and protocols.\nA user interface application provides an interactive file manager and (meta)data browser\nand data entry forms based on the back end APIs.\nBesides the data storage and data management, Fairspace offers _analysis environments_ using {JupyterHub}[Jupyter Hub].\nIn Jupyter Hub, the data repository is accessible. Every user has a private working directory.\nWe do no assumptions on the structure of the data or on the permissions of the external file systems\nthat are connected to the data repository and referenced in the data catalogue.\nThe organisation structure may be replicated in the different systems in incompatible ways,\nand the permissions may not be aligned.\n\n=== Workflow and access modes\n\nDuring the lifetime of a collection, different rules may be applicable for data modification and data access.\nIn Fairspace, collections follow a workflow with the following statuses:\n\n* _Active_: for the phase of data collection, data production and data processing;\n* _Read-only_: for when the data set is complete and is available for reuse;\n* _Archived_: for when the data set should not be available for reading, but still needs to be preserved;\n* _Deleted_: for when the data set needs to be permanently made unavailable (non-readable and non-searchable).\nThis status is irreversible. There is one exception to this rule – for the sake of data loss prevention, in special cases, administrators can still undelete a collection that was already deleted.\n\nIn these different statuses, different actions on the data are enabled or disabled. Also, visibility of the data and linked metadata depends partly on the collection status.\nWe also distinguish three access modes for reading and listing files in a collection (where listing also includes seeing the metadata):\n\n* _Restricted_: only access to explicitly selected workspaces and users;\n* _Metadata published_: the collection and its files are visible, metadata linked to them is visible for all users;\n* _Data published_: the files in the collection are readable for all users.\nThis mode is irreversible. There is one exception to this rule – there might be a special situation, resulting from, e.g., a legal reason, when a collection has to be unpublished. This action is available to administrators, but it is highly discouraged, since the collection (meta)data may already be referenced in other systems.\n\nThe statuses and access modes, and the transitions between them\nare shown in the following diagram.\n\nimage:docs/images/diagrams/Dataset workflow and visibility modes.png[Collection editing and publication workflow]\n\n=== Roles and permissions\n\nWe distinguish the following roles in the solution:\n\n* _User_: regular users can only view their own workspaces and collections.\n* _View public metadata_: the user can view public metadata, workspaces, collections and files;\n* _View public data_: the user can read public files;\n* _Admin_: can create workspaces, assign roles and permissions;\n* _Add shared metadata_: can add, modify and delete shared metadata entities.\n* _Query metadata_: can run \u003c\u003cSPARQL\u003e\u003e queries to query metadata.\n\nMost users should have the _View public data_ role.\nOnly when the shared metadata may contain sensitive information that should\nnot be visible for some users, the public data and public metadata roles should be discarded for\nthose users.\n\nWorkspaces are used to organise collections in a hierarchy. On workspace level there are two access levels:\n\n* _Manager_: can edit workspace details, manage workspace access and manage access to all collections that belong to the workspace;\n* _Member_: can create a collection in the workspace.\n\nAccess to collections and files is managed on collection level. We distinguish the following access levels on collections:\n\n* _List_: see collection, directory and file names and metadata properties/relations\n(only applicable for collections shared via the _Metadata published_ access mode);\n* _Read_: read file contents;\n* _Write_: add files, add new file versions, mark files as deleted;\n* _Manage_: grant, revoke access to the collection, change collection status and modes.\n\nAccess levels are hierarchical: the _Read_ level includes the _List_ level;\nthe _Edit_ level includes _Read_ level; the _Manage_ level includes _Edit_ and _Read_ level access.\nThe user that creates the collection gets _Manage_ access.\n\n\n\n== Data model and view configuration\n\n=== Metadata\n\nMetadata is data about data.\nMetadata is used to describe data assets, e.g., for making it easier to find or use certain data.\nBecause metadata is data itself, it can be difficult to make a proper distinction between data and metadata in a system.\n\n==== Types of metadata\n\nIn a digital archive, _technical metadata_ is linked to data assets, like file type, location, size, creation or modification dates, checksums for checking data integrity, ownership.\nSuch metadata is essential for a system to store and retrieve data files.\nTechnical metadata can also include data format specific properties, like encoding, data layout, resolution, etc., required to correctly read the data. +\nWith most publications, _bibliographic metadata_ is associated, such as author, title, abstract, publication details, keywords and subject categories.\nSuch metadata makes it possible to find relevant publications.\nThis is the kind of metadata used by libraries and archives and numerous standards exist for such data, such as https://www.dublincore.org/[Dublin Core] and https://www.loc.gov/standards/mets/[METS].\n\nMore detailed _descriptive metadata_ provides information about the contents of the data,\ne.g., description of rows and columns, summary statistics, project information, geographical information, results, study design, methods, materials or equipment.\nIn the extreme case, the entire content of the file is captured in descriptive metadata.\n\nWe can distinguish different kinds of descriptive metadata, such as:\n\n* Description of the _contents_ (rows, columns, values, summary statistics)\n* Description of the _subject_, what the data is about (subject, topic, project, study design, object of study, time, location)\n* Description of _data sources_ (for derived or processed data)\n* Description of the _methods_ or technology used to produce or capture the data, such as scripts and versions.\n\nIn the context of health research data, it is essential to link data to research subjects, i.e., patients and samples.\n\nThe values of the metadata can be of any type, numerical, free text, date,\nconform to a controlled vocabulary (e.g., ICD or SNOMED codes, units, file types)\nor a reference to a typed entity within the database, or external entities.\n\nLikewise, the data the metadata is about can be of any type, a file system, a tabular file, image, genomic data, a relational database, etc.\n\n==== Purpose\nMetadata is used for several purposes:\n\n* Descriptors to enable use of the data (file type, file format, encoding, how it was created/generated).\nThe metadata may be used by users or scripts to read or interpret a particular file or data set.\n* Finding relevant data for analysis:\n** Metadata may be used to organise data within a data set that a researcher is working on, by using (study specific) categories linked to individual files.\n** Metadata may be used in search queries or navigation to find out if data is available that meets certain selection criteria (e.g., data types, categories, cohort characteristics), for inclusion in a new analysis.\n** Metadata may be used to identify data that is linked to a specific entity, such as a patient or a sample, to determine if such data has already been analysed, in order to avoid duplicate analysis.\n\nIt is important to identify for which purpose metadata is collected and used, as it may affect which types of metadata are collected, how they are navigated and if access control on metadata is desired or required.\n\n=== Data model\n\nTo enable validation of (meta)data, and to enable intuitive navigation and search within the metadata, it is essential to have a good data model. +\nThe data model consists of the entity types (classes), their properties (with types) and relationships between entities that can be represented in the system.\n\nThe data model needs to be broad (expressive) enough to allow users to express all relevant facts about data sets conveniently and accurately, but it needs to be specific enough to allow validation and the generation of useful overviews and information pages.\nInternational data standards should be used as much as possible to enable interoperability between systems.\n\nE.g., it is probably better to use a specific field ‘disease’ where the value must be a valid ICD-10 code, than using a generic ‘description’ field where a disease is described in a free text field.\n\n==== Data domains\n\nWe distinguish different data domains in order to clearly separate the data that is system specific and the metadata that is more flexible.\n\n===== Workspaces and collection-level data\n\nUsers, workspaces, collections, directories and files are system-level entities,\nrepresenting the file system of the system.\nAccess to these entities is restricted by the workspace-level and collection-level access control.\nThese entities cannot be changed on demand, but are inherent to the system.\nHowever, custom properties and relations may be added, e.g., to link files to patients.\n\n===== Metadata\n\nThe data model for the other (non system-level) entities, the shared metadata, can be configured,\nin order to make the metadata suitable for the environment where it is used.\nThese metadata are used to link entities in the file system to entities in the research domain,\nsuch as samples, patients, diseases, diagnoses,\nor to entities in the organisation domain, such as projects.\nThese entities may be displayed and navigated in the application and can be explored through the API (for technical users).\n\n====== Controlled vocabularies\n\nThe data model may contain controlled vocabularies (e.g., disease codes, file types, project phases) that can be used as values in the metadata.\nEvery value in a controlled vocabulary has a unique identifier and a label.\nUsing such vocabularies enables standardisation and validation of metadata values.\n\n====== Reference data\n\nThe data model may support domain specific entity types (patients, samples, genes, treatments, studies, etc.)\nor generic entity types (project, organisation, person, etc), defining the metadata objects that collection-level data assets can refer to.\nThe reference data can also be linked.\n\nEvery entity has a unique identifier, a type, a label, and the properties and relations as specified by the type.\nThese entities do not belong to a particular space that is owned by a specific group or user.\n\n=== Data model configuration\n\nFairspace uses an {Jena}[Apache Jena] database to store system metadata\nand the custom domain specific metadata.\nThe data models for these metadata are defined using the {SHACL}[Shapes Constraint Language (SHACL)].\n\n* The system metadata includes workspaces, collections, directories, files, file versions, users and access rights.\n  The system data model is defined in  link:projects/saturn/src/main/resources/system-vocabulary.ttl[system-vocabulary.ttl]\n* The customisable data model includes the custom (shared)\n  metadata entities, custom controlled vocabulary types,\n  and custom properties of the system entities.\n  The default custom data model is defined in link:projects/saturn/src/main/resources/vocabulary.ttl[vocabulary.ttl].\n  This data model can be overriden by a data more suitable for your organisation.\n\nA schematic overview of the default data model in link:projects/saturn/src/main/resources/vocabulary.ttl[vocabulary.ttl]:\n\nimage:docs/images/diagrams/CDR data model.png[CDR data model]\n\nThe data model defines an entity-relationship model, specifying\nthe entity types that are relevant to describe your data assets,\nthe properties of the entities, and the relationships between entities.\n\n\n.Example data model\n====\n\nIn this example data model, the following custom entity types are defined:\n\n * ``example:Gender`` with property _Label_;\n * ``example:Species`` with property _Label_;\n * ``example:Subject`` with properties _Gender_, _Species_, _Age at last news_ and _Files_.\n\nThe system class ``fs:File`` is extended with the _Is about subject_ property.\n\n[source, turtle]\n----\n@prefix owl: \u003chttp://www.w3.org/2002/07/owl#\u003e .\n@prefix rdf: \u003chttp://www.w3.org/1999/02/22-rdf-syntax-ns#\u003e .\n@prefix rdfs: \u003chttp://www.w3.org/2000/01/rdf-schema#\u003e .\n@prefix sh: \u003chttp://www.w3.org/ns/shacl#\u003e .\n@prefix xsd: \u003chttp://www.w3.org/2001/XMLSchema#\u003e .\n@prefix dash: \u003chttp://datashapes.org/dash#\u003e .\n@prefix fs: \u003chttps://fairspace.nl/ontology#\u003e .\n@prefix example: \u003chttps://example.com/ontology#\u003e .\n\nexample:Gender a rdfs:Class, sh:NodeShape ;\n    sh:closed false ;\n    sh:description \"The gender of the subject.\" ;\n    sh:name \"Gender\" ;\n    sh:ignoredProperties ( rdf:type owl:sameAs ) ;\n    sh:property\n    [\n        sh:name \"Label\" ;\n        sh:description \"Unique gender label.\" ;\n        sh:datatype xsd:string ;\n        sh:maxCount 1 ;\n        dash:singleLine true ;\n        fs:importantProperty true ;\n        sh:path rdfs:label\n    ] .\n\nexample:Species a rdfs:Class, sh:NodeShape ;\n    sh:closed false ;\n    sh:description \"The species of the subject.\" ;\n    sh:name \"Species\" ;\n    sh:ignoredProperties ( rdf:type owl:sameAs ) ;\n    sh:property\n    [\n        sh:name \"Label\" ;\n        sh:description \"Unique species label.\" ;\n        sh:datatype xsd:string ;\n        sh:maxCount 1 ;\n        dash:singleLine true ;\n        fs:importantProperty true ;\n        sh:path rdfs:label\n    ] .\n\nexample:isOfGender a rdf:Property .\nexample:isOfSpecies a rdf:Property .\nexample:ageAtLastNews a rdf:Property .\n\nexample:Subject a rdfs:Class, sh:NodeShape ;\n    sh:closed false ;\n    sh:description \"A subject of research.\" ;\n    sh:name \"Subject\" ;\n    sh:ignoredProperties ( rdf:type owl:sameAs ) ;\n    sh:property\n    [\n        sh:name \"Label\" ;\n        sh:description \"Unique subject label.\" ;\n        sh:datatype xsd:string ;\n        sh:maxCount 1 ;\n        dash:singleLine true ;\n        fs:importantProperty true ;\n        sh:path rdfs:label;\n        sh:order 0\n    ],\n    [\n        sh:name \"Gender\" ;\n        sh:description \"The gender of the subject.\" ;\n        sh:maxCount 1 ;\n        sh:class example:Gender ;\n        sh:path example:isOfGender\n    ],\n    [\n        sh:name \"Species\" ;\n        sh:description \"The species of the subject.\" ;\n        sh:maxCount 1 ;\n        sh:class example:Species ;\n        sh:path example:isOfSpecies\n    ],\n    [\n        sh:name \"Age at last news\" ;\n        sh:description \"The age at last news.\" ;\n        sh:datatype xsd:integer ;\n        sh:maxCount 1 ;\n        sh:path example:ageAtLastNews\n    ],\n    [\n        sh:name \"Files\" ;\n        sh:description \"Linked files\" ;\n        sh:path [sh:inversePath example:aboutSubject];\n    ] .\n\nexample:aboutSubject a rdf:Property .\n\n# Augmented system class shapes\nfs:File sh:property\n    [\n        sh:name \"Is about subject\" ;\n        sh:description \"Subjects that are featured in this collection.\" ;\n        sh:class example:Subject ;\n        sh:path example:aboutSubject\n    ] .\n----\nAll entity types have a unique label, specified using the ``rdfs:label`` predicate.\nThe _Gender_ and _Species_ properties link the subject to an entity from\nthe respective controlled vocabularies.\nThe _Age at last news_ property is a numerical (integer) value property. +\nThe _Files_ property of the _Subject_ entity type is an example of an inverse relation.\nThe link is defined on the file, but the link will be visible on the subject as well, because of this inverse relation.\n====\n\nThe following guidelines should be followed when creating a custom data model.\n\n* Define a namespace for your custom entities and properties,\n  like ``@prefix example: \u003chttps://example.com/ontology#\u003e .`` in the example.\n* Each custom entity type must have types ``rdfs:Class`` and ``sh:NodeShape``, the properties ``sh:closed false`` and\n  ``sh:ignoredProperties ( rdf:type owl:sameAs )``,\n  and a valid value for ``sh:name``.\n  The ``sh:description`` property is optional.\n* Controlled vocabulary or terminology types are modelled as entity types as well, having only the _Label_ (``rdfs:label``) property, see ``example:Gender`` and ``example:Species``.\n* Properties are specified using the ``sh:property`` property.\n** Every entity type must have a property _Label_ (``sh:path rdfs:label``)\n   of data type ``xsd:string``.\n   The label of an entity must be unique for that type.\n   The label property should be singleton and marked ``fs:importantProperty true``. If there are multiple properties, the label should have ``sh:order: 0``.\n** Properties must have a valid value for ``sh:name``.\n  The ``sh:description`` property is optional.\n** A property must either have a ``sh:datatype`` property,\nspecifying one of ``xsd:string``, ``xsd:integer`` or ``xsd:date``,\n   or a property ``sh:class`` specifying an entity type as the target of a relationship.\n** The predicate used for the property (the middle part of the RDF triple)\nis specified with the ``sh:path`` property, e.g., ``example:aboutSubject``\nfor the _Is about subject_ relation.\n** If a relationship is bidirectional, the path of the inverse relation is specified using ``sh:inversePath``, see the _Files_ property on the _Subject_ entity type.\n** A property can be marked _mandatory_ by specifying ``sh:minCount 1``.\n   A property can be marked _singleton_ by specifying ``sh:maxCount 1``.\n** A text property (with ``sh:datatype xsd:string``) can be limited\n   to a single line text field using ``dash:singleLine true``.\n\n==== Limitations\nAlthough assigning multiple types to an entity is easy in RDF, Fairspace assumes entities to have a single type.\n\nInheritance is possible in SHACL, but not supported by Fairspace.\nInstead of specifying an entity type as a subtype of another,\na single type can be specified with a _type_ property,\nindicating the sub type of the entity.\n\nE.g., instead of defining entity types _DNASeqAssay_ and _RNASeqAssay_\nas sub types of _Assay_, a property type _assayType_ can be defined on _Assay_,\nusing a controlled vocabulary type _AssayType_ with the assay types as values.\n\nAlthough there are many RDF-compatible XSD datatypes, it is recommended to reuse the types\nthat are already used in the default vocabularies.ttl file as a value of ``sh:datatype`` property.\nOther types may not be handled properly in the user interface and may cause some unexpected issues.\nSame recommendation is for SHACL constraints that can be added for an entity or its properties - reuse the constraints described\nin the custom data model creation guidelines.\n\n==== Changing existing data model\n\nFlexible, configurable data model is one of the key features of Fairspace.\nData model evolution is possible, but needs to be applied carefully as well:\nmake sure that new versions of data models are consistent with previous versions,\nin order to prevent inconsistencies for existing data.\n\nWARNING: _Editing a data model is specialized work for data modellers/information architects. Use with care.\nThe system is flexible, but the system cannot compensate for poor data modelling choices.\nBad modelling will make it hard for users to enter data and to interact with the data._\n\nIt is recommended to only add properties to existing entities or add new entities.\nChanging existing entities will cause inconsistencies.\n\nList of data model changes that can be considered safe:\n\n* Adding new entity,\n* Adding new property to an existing entity,\n* Removing constraints on properties,\n* Changing description of an existing entity or property.\n\nDangerous actions (not recommended):\n\n* Changing or removing existing entities,\n* Adding or changing constraints,\n* Removing or changing existing properties (property type, name),\n* Changing relations between entities.\n\n.In order to change the model:\n\n. Update the vocabularies.ttl file, defining the custom model. Follow the guides specified in \u003c\u003cData model configuration\u003e\u003e section).\n. Update views configuration file (see views \u003c\u003cView configuration\u003e\u003e section),\nif applicable - only if there is a change that needs to be reflected in metadata search views.\n. Apply the changes\n+\nFor the deployment with Helm, run an upgrade command with _saturn.vocabulary_ and _saturn.views_ parameters\npointing to a new vocabularies and views definitions (see \u003c\u003cInstallation and configuration\u003e\u003e),\nuse `--set-file` option:\n+\n[source, bash]\n----\n~bin/helm/helm upgrade … --set-file saturn.vocabulary=/path/to/vocabulary.ttl --set-file saturn.views=/path/to/views.yaml\n----\n+\nThis should also restart the Saturn pod. If not, trigger the restart manually.\n+\nFor local development - replace vocabulary file in link:projects/saturn/src/main/resources/vocabulary.ttl[vocabulary.ttl]\nand views configuration in link:projects/saturn/src/main/resources/views.yaml[views.yaml].\nRestart Saturn run.\n+\n. Load data for new entities or properties.\n. Reindex Postgres database using `/api/maintenance/reindex` API endpoint (see \u003c\u003cMaintenance\u003e\u003e API) to apply the changes for metadata search.\n\n\n==== Controlled vocabularies\n\nFor controlled vocabulary types, e.g., _Gender_ and _Species_ in the example, you should insert the allowed values in the database by uploading\na taxonomies file using the \u003c\u003cRDF metadata\u003e\u003e API.\nAn example taxonomy is in link:projects/saturn/src/main/resources/taxonomies.ttl[taxonomies.ttl].\n\nIt is preferred to use existing standard taxonomies and labels.\nIf that is not possible, please define your own namespaces for\nyour custom taxonomies.\n\n.Example taxonomy\n====\nIn this example we use existing standard ontologies for the _Gender_ and _Species_ controlled vocabulary types.\n\n* The https://hl7.org/fhir/R4/codesystem-administrative-gender.html[HL7 FHIR AdministrativeGender code system] for _Gender_.\n* The https://bioportal.bioontology.org/ontologies/NCBITAXON/[NCBI Organismal Classification] for _Species_.\n\n[source, turtle]\n----\n@prefix rdfs: \u003chttp://www.w3.org/2000/01/rdf-schema#\u003e .\n@prefix example: \u003chttps://example.com/ontology#\u003e .\n@prefix gender: \u003chttp://hl7.org/fhir/administrative-gender#\u003e .\n@prefix ncbitaxon: \u003chttps://bioportal.bioontology.org/ontologies/NCBITAXON/\u003e .\n\ngender:male a example:Gender ;\n  rdfs:label \"Male\" .\ngender:female a example:Gender ;\n  rdfs:label \"Female\" .\n\nncbitaxon:562 a example:Species ;\n  rdfs:label \"Escherichia coli\" .\nncbitaxon:1423 a example:Species ;\n  rdfs:label \"Bacillus subtilis\" .\nncbitaxon:4896 a example:Species ;\n  rdfs:label \"Schizosaccharomyces pombe\" .\nncbitaxon:4932 a example:Species ;\n  rdfs:label \"Saccharomyces cerevisiae\" .\nncbitaxon:6239 a example:Species ;\n  rdfs:label \"Caenorhabditis elegans\" .\nncbitaxon:7227 a example:Species ;\n  rdfs:label \"Drosophila melanogaster\" .\nncbitaxon:7955 a example:Species ;\n  rdfs:label \"Zebrafish\" .\nncbitaxon:8355 a example:Species ;\n  rdfs:label \"Xenopus laevis\" .\nncbitaxon:9606 a example:Species ;\n  rdfs:label \"Homo sapiens\" .\nncbitaxon:10090 a example:Species ;\n  rdfs:label \"Mus musculus\" .\n----\n====\n\n=== View configuration\n\nFor the metadata pages in the user interface, a view configuration needs to be created\nthat specifies the tables and columns.\nAn example can be found in link:projects/saturn/src/main/resources/views.yaml[views.yaml]\n\n\n\n== Installation and configuration\n\n=== Local development\n\nRequires:\n\n* yarn\n* docker\n* Java 21\n\nOn MacOS the docker logging driver needs to be configured, because the default  is not available (``journald``).\nOverride the logging driver by setting the ``DOCKER_LOGGING_DRIVER`` environment variable or adding a line the ``.env`` file in ``local-development``:\n[source, shell]\n----\nDOCKER_LOGGING_DRIVER=json-file\n----\n\nTo run the development version, checkout this repository,\nnavigate to ``projects/mercury`` and run the following commands (``yarn install`` only has to be ran the first time running fairspace).\n\n[source, shell]\n----\nyarn install\nyarn dev\n----\n\n\nThis will start a Keycloak instance for authentication at port ``5100``,\nthe backend application named Saturn at port ``8080`` and the\nuser interface at port ``3000``.\n\nAt first run, you need to configure the service account in Keycloak. If you cannot log in, you might need to restart fairspace by closing it and running ``yarn dev`` again.\n\n* Navigate to link:http://localhost:5100[http://localhost:5100]\n* Login with credentials ``keycloak``, ``keycloak``\n* In the top-left drop down menu, select the fairspace realm\n* Grant the ``view-users`` role to the client service account:\n** Click `Clients` in the left menu -\u003e Select 'workspace-client'\n** Choose tab `Service Account Roles`\n** Click `Assign Role`\n** Select `Filter by clients` from the drop down menu and search for role name `view-users`. Then click `Assign`.\n\nNow everything should be ready to start using Fairspace:\n\n* Navigate to link:http://localhost:3000/dev[http://localhost:3000/dev] to open the application.\n* Login with one of the following credentials:\n+\n[cols=\"1, 1\"]\n|===\n| Username | Password\n\n| organisation-admin\n| fairspace123\n\n| user\n| fairspace123\n|===\n\n=== Kubernetes and helm\n\nRequires:\n\n* Helm \u003e= 3.14.x\n* kubectl \u003e= 1.27.x\n\nYou can deploy Fairspace on a Kubernetes cluster using link:https://helm.sh/[Helm].\nHelm charts for Fairspace are published to the public helm repository at\nghcr.io/thehyve/helm-charts/fairspace (GitHub Packages of the Fairspace repository)\n\nWe provide a number of charts for various components that can be used in combination, or separately:\n\n- _Fairspace Keycloak_: Installs Keycloak and configures an ingress node for Keycloak. This chart is not required if Keycloak is already installed separately. You still need to \u003c\u003cConfigure a Keycloak realm for Fairspace,configure a Keycloak realm for Fairspace\u003e\u003e\n(chart source: https://github.com/thehyve/fairspace-keycloak[]).\n- _Fairspace_: Installs the Fairspace application, including the _saturn_ backend, _pluto_ proxy, _mercury_ frontend and a PostgreSQL database, and configures an ingress node for Fairspace (chart source: https://github.com/thehyve/fairspace[]).\n- _Jupyter_: Installs a version of Jupyter Hub that uses Keycloak for authentication and launches a link:https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-datascience-notebook[jupyter-datascience-notebook] based Jupyter notebook with the Fairspace collections file system mounted automatically (chart source: https://github.com/thehyve/fairspace-jupyter[]).\n\n==== Instructions for deploying to Google Cloud\n\n===== Download and install helm and gcloud\n\n* Download ``helm 3.14.3`` (or higher) from https://github.com/helm/helm/releases/tag/v3.14.3\n* Extract the downloaded archive to ``~/bin/helm`` and check with:\n+\n[source, shell]\n----\n~/bin/helm/helm version\n----\n\n* Install link:https://kubernetes.io/docs/tasks/tools/install-kubectl/[kubectl] (for Helm 3.14.x install version \u003e 1.27.x).\n* Download and install the link:https://cloud.google.com/sdk/docs/install[Google Cloud SDK] (requires Python).\n* Obtain credentials for Kubernetes:\n+\n[source, shell]\n----\ngcloud iam service-accounts keys create credentials.json --iam-account \u003ciam account id, e.g. fairspace-207108@appspot.gserviceaccount.com\u003e\nexport GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json\ngcloud container clusters get-credentials \u003ccluster id, e.g. fairspacecicluster\u003e --zone europe-west1-b --project \u003cproject id, e.g. fairspace-207108\u003e\n----\n\n* Check if all tools are correctly installed:\n+\n[source, shell]\n----\n# List available clusters\ngcloud container clusters list\n# List Kubernetes namespaces\nkubectl get ns\n# List helm releases (deployments)\n~/bin/helm/helm list -A\n----\n\n===== Configure DNS\n\nFind the address of the Kubernetes cluster:\n[source, shell]\n----\nkubectl cluster-info\n----\nCreate DNS records for the ``keycloak.example.com``, ``fairspace.example.com`` and (optionally) ``jupyterhub.example.com`` domains, pointing to the cluster.\n\n===== Fetch charts\n[source, shell]\n----\n# Fetch the fairspace-keycloak chart\n~/bin/helm/helm pull oci://ghcr.io/thehyve/fairspace/helm-charts/fairspace-keycloak --version 0.7.0\n# Fetch the fairspace chart\n~/bin/helm/helm pull oci://ghcr.io/thehyve/fairspace/helm-charts/fairspace --version 2.0.2\n----\n\n===== Deploy Keycloak\nCreate a new Kubernetes namespace:\n[source, shell]\n----\nkubectl create namespace keycloak-new\n----\nCreate a new deployment (called _release_ in helm terminology) and\ninstall the Fairspace Keycloak chart:\n[source, shell]\n----\n~/bin/helm/helm install keycloak-new fairspace-keycloak-0.7.0.tgz --namespace=keycloak-new \\\n-f /path/to/fairspace-keycloak-values.yaml\n----\nYou can pass values files with ``-f`` or ``--values``.\n\nExample ``fairspace-keycloak-values.yaml`` file:\n[source, yaml]\n----\nfairspaceKeycloak:\n  name: keycloak-new\n  postgresql:\n    postgresPassword: # choose a strong database password\n\nkeycloak:\n  extraEnv: |\n    - name: KEYCLOAK_USER\n      value: keycloak\n    - name: KEYCLOAK_PASSWORD\n      value: # choose a strong Keycloak admin password\n    - name: PROXY_ADDRESS_FORWARDING\n      value: \"true\"\n\ningress:\n  domain: keycloak.example.com\n  tls:\n    certificate:\n      force: true\n----\n\nYou can pass values files with ``-f`` or ``--values``.\n\n===== Configure a Keycloak realm for Fairspace\n\n* Navigate to ``https://keycloak.example.com`` and select _Administration Console_:\n+\nimage:docs/images/screenshots/Keycloak administration console.png[Keycloak administration console, role=\"th\", align=\"center\"]\n\n* Create a realm, e.g., _fairspace_:\n+\nimage:docs/images/screenshots/Add realm.png[Add realm, role=\"th\", align=\"center\"]\n\n* Configure the realm:\n+\nimage:docs/images/screenshots/Realm settings.png[Realm settings, role=\"th\", align=\"center\"]\n* Add a client to the realm, e.g., _fairspace-example-private_:\n+\nimage:docs/images/screenshots/Add client.png[Add client, role=\"th\", align=\"center\"]\n\n* Configure the client:\n+\nimage:docs/images/screenshots/Client settings.png[Client settings, role=\"th\", align=\"center\"]\n\n** Set _Access Type_ to _confidential_;\n** Set _Service Accounts Enabled_ to _On_;\n** Ensure that ``https://fairspace.example.com`` is added to the _Valid Redirect URIs_ and _Web Origins_;\n** Optionally (if you intend to add Jupyter Hub), ensure that the Jupyter Hub domain is added as well.\n\n* Assign the _view-users_ role for client _realm-management_ to the client service account:\n+\nimage:docs/images/screenshots/Service account permissions.png[Service account permissions, role=\"th\", align=\"center\"]\n\n* Copy the client secret from the _Credentials_ tab, for use in the Fairspace configuration:\n+\nimage:docs/images/screenshots/Client credentials.png[Client credentials, role=\"th\", align=\"center\"]\n\n===== Deploy Fairspace\nCreate a new Kubernetes namespace:\n[source, shell]\n----\nkubectl create namespace fairspace-new\n----\nCreate a new deployment (called _release_ in helm terminology) and\ninstall the Fairspace chart:\n[source, shell]\n----\n~/bin/helm/helm install fairspace-new fairspace-2.0.2.tgz --namespace=fairspace-new \\\n-f /path/to/values.yaml --set-file saturn.vocabulary=/path/to/vocabulary.ttl --set-file saturn.views=/path/to/views.yaml\n----\nYou can pass values files with ``-f`` and provide a file for a specified\nvalue with ``--set-file``.\n\nExample ``values.yaml`` file:\n[source, yaml]\n----\n# External dependencies for running the fairspace\nexternal:\n  keycloak:\n    baseUrl: https://keycloak.example.com\n    realm: fairspace\n    clientId: fairspace-example-private\n    clientSecret: # Copy the client secret from Keycloak\n\n# Settings for fairspace\nfairspace:\n  name: \"Example Fairspace\"\n  description: \"Example Fairspace\"\n  ingress:\n    domain: fairspace.example.com\n  features: []\n  icons:\n    jupyter: \"/icons/jupyter.svg\" # path to the icon svg file\n    extra-icon: \"extra-icon.svg\" # path to the custom svg file\n  services:\n    jupyterhub:\n        name: \"JupyterHub\"\n        url: https://jupyterhub.example.com/user/${username}/lab\n        icon-name: \"jupyter\"\n  storages:\n    external:\n      name: external\n      label: \"External storage\"\n      url: https://storage.example.com/api/webdav\n      search-url: https://storage.example.com/api/search/files\n      root-directory-iri: https://storage.example.com/api/webdav\n\n# Specific settings for Saturn subchart\nsaturn:\n  persistence:\n    files:     # stores transaction logs and files\n      size: 60Gi\n      storageClass: expandable\n    database:  # stores RDF database\n      size: 60Gi\n      storageClass: expandable\n    audit:     # stores the audit log\n      size: 10Gi\n      storageClass: expandable\n  resources:\n    limits:\n      cpu: 1\n      memory: 16Gi\n    requests:\n      cpu: 500m\n      memory: 512Mi\n  image:\n    pullPolicy: Always\n  customStorageClass:\n    create: true\n    name: expandable\n    type: pd-sta","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthehyve%2Ffairspace","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthehyve%2Ffairspace","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthehyve%2Ffairspace/lists"}