{"id":19402731,"url":"https://github.com/cscfi/hpcs","last_synced_at":"2025-08-20T23:10:37.473Z","repository":{"id":228608748,"uuid":"774316648","full_name":"CSCfi/HPCS","owner":"CSCfi","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-21T05:37:10.000Z","size":134,"stargazers_count":7,"open_issues_count":12,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-05-21T06:32:46.044Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSCfi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-03-19T10:38:16.000Z","updated_at":"2025-05-08T16:07:58.000Z","dependencies_parsed_at":"2024-11-10T11:26:18.135Z","dependency_job_id":"6ecd33fd-4146-4398-99de-4d232791e0c3","html_url":"https://github.com/CSCfi/HPCS","commit_stats":null,"previous_names":["cscfi/hpcs"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/CSCfi/HPCS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2FHPCS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2FHPCS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2FHPCS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2FHPCS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSCfi","download_url":"https://codeload.github.com/CSCfi/HPCS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2FHPCS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271400259,"owners_count":24752830,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T11:25:30.929Z","updated_at":"2025-08-20T23:10:37.449Z","avatar_url":"https://github.com/CSCfi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HPCS High Performance Computing Secured\n\n## Main goal\n\n### Project\n\nThis partnership project involving CSC and Hewlett Packard Enterprise aims to enable HPC users to run secured jobs. It provides tools to enable anyone running secured jobs with encrypted data and specific confidential containers on a supercomputing site, leveraging (non exhaustively) :\n\n- [SPIFFE/SPIRE](https://github.com/spiffe/spire)\n- [Hashicorp Vault](https://github.com/hashicorp/vault)\n- [Singularity / Apptainer encryption](https://github.com/apptainer/apptainer)\n- [age encryption](https://github.com/FiloSottile/age)\n\n### Architecture\n\nIn-depth architecture documentation is available [here](/docs/architecture/architecture.md)\n\n## Demonstration\n\n[![asciicast](https://asciinema.org/a/PWDzxlaVQmfGjbh30KKNTRO1f.svg)](https://asciinema.org/a/PWDzxlaVQmfGjbh30KKNTRO1f)\n\n## Quickstart\n\n### Client\n\nAssuming that the HPCS server is setup. (See [Server](#Server)).\n\nTo start using HPCS Client, the supported method has two requirements, docker and docker-compose.\n\n#### Pulling images\n\nStart by pulling the three images.\n\n```bash\ndocker pull ghcr.io/cscfi/hpcs/[container/data/job]-prep:['branch-name'/latest]\n```\n\n```bash\ndocker pull ghcr.io/cscfi/hpcs/container-prep:latest\ndocker pull ghcr.io/cscfi/hpcs/data-prep:latest\ndocker pull ghcr.io/cscfi/hpcs/job-prep:latest\n```\n\n#### Configuring the client\n\nInformation about the Spire-Server, the HPCS-Server and the Vault depends on the respective server installation setup choices. For more information on the configuration, please contact your HPCS-Server service provider.\n\nThe client configuration is made of 4 main sections, in INI format. In depth description of the configuration files is available [here](/docs/configuration).\n\nExample of client configuration :\n\n```ini\n[spire-server]\naddress = spire.lumi.csc.fi\nport = port\ntrust-domain = spire-trust-domain\n\n[hpcs-server]\nurl = https://hpcs-server:port\n\n[vault]\nurl = https://vault-provider:port\n\n[supercomputer]\naddress = lumi.csc.fi\nusername = etellier\n```\nPlease replace the `spire-server` section configuration with the settings relative to your Spire Server.\nYou will also need to replace `hpcs-server` with the address of your HPCS server, eventually the `port` with the port on which the HPCS server is exposed.\nThe `vault` is the same as the `hpcs-server` section, please complete it with your vault settings.\nFinally, configure the supercomputer to use in the `supercomputer` section, specifying it's address under `address` and your `username` on the system. Your SSH Key needs to be setup.\n\n#### Prepare the runtime\n\nWe recommend using docker-compose to run the process. Here is an example of a docker-compose file :\nIt's composed of 3 sections, covering the three main steps of the process.\n- Prepare, encrypt and ship the `talinx/jp2a` image for the user `etellier` on node `nid003044`\n- Prepare, encrypt and ship the input data under `./workdir/jp2a_input` for the user `etellier` on node `nid003044`\n- Run the `jp2a` preapred image on the supercomputer `nid003044` node, specifying application args `/sd-container/encrypted/jp2a_input/image.png --width=100 --output=/sd-container/output/result` (the `jp2a_input` prepared dataset will be available under `/sd-container/encrypted/jp2a_input` at runtime). You also can specify your verifications scripts to run before and after the application here : `[-i/-o] /pfs/lustrep4/scratch/project_462000031/etellier/verification_scripts`\n\nIn depth documentation of the cli of the 3 parts are available [here](/docs/cli).\n\n```yaml\nversion: '2.4'\n\nservices:\n  container-prep:\n    image: ghcr.io/cscfi/hpcs/container-prep:latest\n    command:\n      - \"--config\"\n      - \"/tmp/hpcs-client.conf\"\n      - \"-b\"\n      - \"talinx/jp2a\"\n      - \"-s\"\n      - \"/Users/telliere/secure_job/workdir\"\n      - \"-e\"\n      - \"-u\"\n      - \"etellier\"\n      - \"-c\"\n      - \"nid003044\"\n      - \"--data-path\"\n      - \"/tmp/encrypted_prepared_jp2a.sif\"\n      - \"--data-path-at-rest\"\n      - \"/scratch/project_462000031/etellier/\"\n      - \"--username\"\n      - \"etellier\"\n    volumes:\n      - /etc/group:/etc/group\n      - /etc/passwd:/etc/passwd\n      - /var/run/docker.sock:/var/run/docker.sock\n      - $PWD/workdir:/tmp\n      - $HOME/.ssh:/tmp/.ssh\n    environment:\n      - PWD=${PWD}\n      - HOME=${HOME}\n    user: \"1001\" # On Linux : Your uid (gathered using `id`). Remove on MacOS\n    group_add:\n     -  \"120\"    # The docker daemon group. Remove on MacOS\n\n\n  data-prep:\n    image: ghcr.io/cscfi/hpcs/data-prep:latest\n    command:\n      - \"--config\"\n      - \"/tmp/hpcs-client.conf\"\n      - \"-i\"\n      - \"/tmp/jp2a_input\"\n      - \"-o\"\n      - \"/tmp/\"\n      - \"-u\"\n      - \"etellier\"\n      - \"-c\"\n      - \"nid003044\"\n      - \"--data-path\"\n      - \"/tmp/encrypted_jp2a_input.tgz\"\n      - \"--data-path-at-rest\"\n      - \"/scratch/project_462000031/etellier/\"\n      - \"--username\"\n      - \"etellier\"\n    volumes:\n      - /var/run/docker.sock:/var/run/docker.sock\n      - $PWD/workdir:/tmp\n      - $HOME/.ssh:/tmp/.ssh\n    environment:\n      - PWD=${PWD}\n      - HOME=${HOME}\n\n  job-prep:\n    image: ghcr.io/cscfi/hpcs/job-prep:latest\n    command:\n      - \"--config\"\n      - \"/tmp/hpcs-client.conf\"\n      - \"-N\"\n      - \"1\"\n      - \"-p\"\n      - \"standard\"\n      - \"-t\"\n      - \"\\\"00:60:00\\\"\"\n      - \"-A\"\n      - \"project_462000031\"\n      - \"--nodelist\"\n      - \"nid003044\"\n      - \"--workdir\"\n      - \"/scratch/project_462000031/etellier\"\n      - \"-ai\"\n      - \"/scratch/project_462000031/etellier/encrypted_prepared_jp2a.sif.info.yaml\"\n      - \"-di\"\n      - \"/scratch/project_462000031/etellier/encrypted_jp2a_input.tgz.info.yaml\"\n      - \"-args\"\n      - \"\\\"\\\\\\\" /sd-container/encrypted/jp2a_input/image.png --width=100 --output=/sd-container/output/result \\\\\\\"\\\"\"\n      - \"-i\"\n      - \"/pfs/lustrep4/scratch/project_462000031/etellier/verification_scripts\"\n      - \"-o\"\n      - \"/pfs/lustrep4/scratch/project_462000031/etellier/verification_scripts\"\n      - \"-flags\"\n      - \"--env TERM=xterm-256color\"\n      - \"-f\"\n    volumes:\n      - $PWD/workdir:/tmp\n      - $HOME/.ssh:/tmp/.ssh\n    environment:\n      - PWD=${PWD}\n      - HOME=${HOME}\n```\n\n#### Run the preparations and the job\n\nTo run one of the containers :\n\n```bash\ndocker compose run --rm [data/container/job]-prep\n```\n\nIf you want to run the whole process by yourself :\n\n```bash\ndocker compose run --rm data-prep\ndocker compose run --rm container-prep\ndocker compose run --rm job-prep\n```\n\nAn example demonstration is available [here](https://asciinema.org/a/PWDzxlaVQmfGjbh30KKNTRO1f).\n\n### Server\n\nHPCS Server is an API, interfacing HPCS client with Vault and Spire. This section needs basic knowledge of [SPIFFE/SPIRE](https://spiffe.io/) and [HashiCorp Vault](https://www.vaultproject.io/).\n\nFor k8s, we only consider `kubectl` and `ansible` as available tools and that `kubectl` can create pods. Vault roles, spire identities are created automatically.\nFor development and demonstrative purposes we provide a `terraform` definition of a VM with an operational kubernetes cluster, for documentation and deployment instructions go [there](terraform).\n\nFor docker-compose, we consider the Vault and the Spire Server as setup and the Spire-OIDC provider implemented to allow login to the vault using SVID identity. We also consider that proper roles are created in Vault to authorize HPCS Server to write roles and policies to the Vault, using a server SPIFFEID.\n\n#### K8s\n\nHPCS' serverside consists in an web API. The difficulty in the installation of it comes with the underlying services that needs to be available for it to run properly. Here is a scheme of the architecture of a fully-installed HPCS server side.\n\n```mermaid\nflowchart LR\n\nsubgraph SSP[Spire Server pod]\n  SSC[Spire Server container]\n  SOC[Spire OIDC container]\n  NC[Nginx proxy]\nend\n\nSSS[Spire server service]\nVS[Vault service]\n\nVP[Vault pod]\n\nHP[HPCS server pod]\n\nSSC \u003c-- TCP:8081--\u003e SSS\nNC \u003c-- HTTPS:443--\u003e SSS\nSSC \u003c--UNIX Socket--\u003e SOC\nSOC \u003c--UNIX Socket--\u003e NC\nSSS \u003c--HTTPS:443--\u003e VP\nHP \u003c--TCP:8081--\u003e SSS\nHP \u003c--UNIX Socket--\u003e SSC\nVS \u003c--HTTP:8200--\u003e VP\nHP \u003c--HTTP:8200--\u003e VS\n\nOutside \u003c--HTTP:anyport--\u003e VS\nOutside \u003c--TCP:anyport--\u003e SSS\n```\n(Ports are specified for the serverside, clients ports used for communication doesn't matter)\n\nThis architecture comes in 3 different main parts :\n- HPCS Server (1 Container, containing Spire-Agent and HPCS Server)\n- Spire Server (3 Containers, spire-server, spire-oidc, and hpcs-nginx)\n- Vault (Helm chart, not managed by HPCS)\n\nIn order to proceed to the deployment of this architecture, k8s is the supported method, all the code associated is available under `/k8s`.\n\n##### Pre-requisite\n\nBefore proceeding to HPCS' deployment, an original setup is required including :\n- A ready-to-run k8s cluster\n- `kubectl` and `helm` available and able to run kubernetes configurations (`.yaml`)\n- `rbac`, `storage` and `dns` and `helm` kubernetes capabilities, f.e : `microk8s enable rbac storage dns helm` with microk8s.\n\nPlease note down the name of your k8s cluster in order to run later deployments.\n\n##### Configuration\n\n##### Bash\n\nThis part of the documentation walks you through the different steps necessary in order to run a manual deployment of HPCS' serverside (including Vault, Spire-Server and HPCS Server).\n\nGenerate your certificate :\n```bash\nopenssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout hpcs-stack/charts/spire/files/spire-oidc.key -out hpcs-stack/charts/spire/files/spire-oidc.crt -addext \"subjectAltName = DNS:spire-oidc\"\n```\n\nAdd hashicorp repo and run installation :\n```bash\nhelm repo add hashicorp https://helm.releases.hashicorp.com\nhelm install vault hashicorp/vault --version 0.27.0 --namespace=hpcs\n```\n\nInstall `hpcs-stack` chart :\n```bash\nhelm upgrade --install --namespace hpcs --create-namespace hpcs-stack $(git rev-parse --show-toplevel)/k8s/hpcs-stack\n```\n\nInitialize the Vault :\n```bash\nkubectl exec -it vault-0 -n hpcs -- vault operator init -n 1 -t 1\n```\nNote unseal token and root token.\n\nUnseal vault :\n```bash\nkubectl exec -it vault-0 -n hpcs -- vault operator unseal [seal token]\n```\n\nNow double check that `spire-server` installed as part of `hpcs-stack` is actually ready already :\n```bash\n-\u003e kubectl get --namespace hpcs pods/spire-server-0\nNAME             READY   STATUS    RESTARTS   AGE\nspire-server-0   3/3     Running   0          73m\n```\n\nIf it's ready you can connect to the vault to enable jwt auth and kvv2 secrets, register oidc as a source :\n```bash\nkubectl exec -it vault-0 -n hpcs -- sh\nexport VAULT_TOKEN=\"[root token]\"\n\n# Enable kvv2\nvault secrets enable -version=2 kv\n\n# Enable jwt auth\nvault auth enable jwt\n\n# Connect OIDC authority (spire-oidc)\nvault write auth/jwt/config oidc_discovery_url=https://spire-oidc oidc_discovery_ca_pem=\"\n-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n\"\n```\n\n__Next step is to create a spire identity and it's vault role in order to be able to identify HPCS-Server against Vault__\n\nGet your kubernetes node uid (repeat this and the following spire identity creation for every nodes):\n```bash\nkubectl get nodes -o json | grep uid\n```\n\nCheck wether you run containers using cgroupsv2 :\n```bash\ngrep cgroup2 /proc/filesystems\n```\n\nCreate the spire identity (If not using cgroupsv2):\n```bash\nkubectl exec -n hpcs spire-server-0 -c spire-server -- ./bin/spire-server entry create -spiffeID spiffe://hpcs/hpcs-server/workload -parentID spiffe://hpcs/spire/agent/k8s_psat/[Cluster Name]/[UID] -selector k8s:pod-name:hpcs-server\n```\n\nCreate the spire identity (If using cgroupsv2):\n```bash\nkubectl exec -n hpcs spire-server-0 -c spire-server -- ./bin/spire-server entry create -spiffeID spiffe://hpcs/hpcs-server/workload -parentID spiffe://hpcs/spire/agent/k8s_psat/[Cluster Name]/[UID] -selector unix:uid:0\n```\n\nConnect to vault and create hpcs-server policy :\n```bash\n\u003e kubectl exec -it vault-0 -n hpcs -- sh\n/ $ vi /tmp/policy\npath \"auth/jwt/role/*\" {\n  capabilities = [\"sudo\",\"read\",\"create\",\"delete\",\"update\"]\n}\npath \"sys/policies/acl/*\" {\n  capabilities = [\"sudo\",\"read\",\"create\",\"delete\",\"update\"]\n}\n/ $ vault policy write hpcs-server /tmp/policy\n/ $ vault write auth/jwt/role/hpcs-server role_type=jwt user_claim=sub bound_audiences=TESTING bound_subject=spiffe://hpcs/hpcs-server/workload token_ttl=24h tok\nen_policies=hpcs-server\n```\n\nYou can now wait for HPCS server to finish setting up :\n\n```bash\n-\u003e kubectl get --namespace hpcs pods/hpcs-server-0\nNAME            READY   STATUS    RESTARTS      AGE\nhpcs-server-0   1/1     Running   3 (75m ago)   75m\n```\n\nThat's it, you can now use HPCS server as you please.\n\n##### Ansible\n\nThe previously explained steps can be automatically run using an ansible [playbook](k8s/deploy-all.yaml).\n\nAll the pre-requisites listed before are necessary to run this playbook. If you are running kubernetes using `microk8s`, you will need to create aliases or fake commands for `helm`, for example using a script :\n```bash\n#!/bin/bash\n\nmicrok8s helm3 $@\n```\nWritten as `/usr/bin/helm`.\n\nYou will also need ansible k8s and openssl plugins :\n```bash\nansible-galaxy collection install kubernetes.core\nansible-galaxy collection install community.crypto\n```\n\nYou can now run the ansible playbook :\n```bash\ncd k8s\nansible-playbook deploy-all.yaml\n```\n\n#### Docker-compose\n\n:warning: This method is not the officially supported method for HPCS Server and merely intended for testing purposes.\n\nPull server's image using Docker pull :\n\n```bash\ndocker pull ghcr.io/cscfi/hpcs/server:latest\n```\n\nThe server configuration is made of 2 main sections, in INI format. In depth description of the configuration files is available [here](/docs/configuration).\n\nYou'll be able to configure your Spire Server interfacing specifying :\n- Address and port of the spire-server API.\n- Spire trust domain.\n- pre-command and spire-server-bin : e.g pre-command = \"`kubectl exec -n spire spire-server-0 -- `\" and spire-server-bin = \"`spire-server`\" will then be used to create cli interactions with the Spire Server socket (i.e : `kubectl exec -n spire spire-server-0 -- spire-server entry show`). Please keep this part as-it-is when running docker standalone and mount the spire-server directory at it's default path (`/tmp/spire-server`).\n\nVault configuration will work the same as for the client (using a base `url` config). The main difference is that you need to specify the name of the spire-server role in the Vault. This role needs to be created manually and needs to be bound to a policy allowing it to create policies and roles for clients (data/container preparation) and workloads (accessing data/container).\n\n```ini\n[spire-server]\naddress = \"spire.lumi.csc.fi\"\nport = PORT\ntrust-domain = TRUST_DOMAIN\npre-command = \"\"\nspire-server-bin = spire-server\n\n[vault]\nurl = https://vault-provider:port\nserver-role = hpcs-server\n```\n\nThe server image comes with a non-configured agent, it's configuration is assumed to be mounted under `/tmp/spire-agent.conf`. Several [node attestation methods](https://github.com/spiffe/spire/tree/main/doc) (see `plugin_agent_nodeattestor`) are available, the goal is to register the HPCS server Spire agent under a specific identity to provide workload identities for the server (then used to write policies and roles in vault).\n\nAn example configuration, using `join_token` attestation method :\n\n```hcl\nagent {\n    data_dir = \"./data/agent\"\n    log_level = \"DEBUG\"\n    trust_domain = \"TRUST_DOMAIN\"\n    server_address = \"\"\n    server_port = PORT\n    socket_path = \"/tmp/spire-agent/public/api.sock\"\n    join_token = \"TOKEN\"\n    insecure_bootstrap = true\n}\n\nplugins {\n   KeyManager \"disk\" {\n        plugin_data {\n            directory = \"./data/agent\"\n        }\n    }\n\n    NodeAttestor \"join_token\" {\n        plugin_data {}\n    }\n\n    WorkloadAttestor \"unix\" {\n        plugin_data {\n            discover_workload_path = true\n        }\n    }\n}\n```\n\n\nTo run the server as a standalone Docker, we recommend using docker-compose.\n\nAn in depth documentation of the server's cli is available [here](/docs/configuration).\n\nThis docker-compose file specifies the proper spire-agent configuration to use, the mountpoint of the spire-server directory and the path to the mounted hpcs configuration.\n\n```yaml\nversion: '2.4'\n\nservices:\n  server:\n    image: ghcr.io/cscfi/hpcs/server:dockerfile_everywhere\n    command:\n      - \"--config\"\n      - \"/tmp/hpcs-server.conf\"\n    ports:\n      - 10080:10080\n    volumes:\n      - $PWD:/tmp\n      - $PWD/spire-agent.conf:/tmp/agent.conf\n      - /tmp/spire-server:/tmp/spire-server\n    environment:\n      - PWD=${PWD}\n```\n\nYou can then run it using :\n```bash\ndocker-compose run server\n```\n\n## Limitations\n\nThis project has been developed to work on LUMI and is currently (03/2024) still under development. The goal was to use LUMI as-is, without the need for changes by administrators. Even though this makes it easier to adapt to other environments, it also means introducing limitations that can prevent HPCS to achieve its full potential. These limitations are discussed below.\n\n### Node attestation\n\nThis project enables users to chose who can read their data or containers based on UNIX identities on the super-computer platform. Another important feature is the possibility for them to limit this access to a specific set of nodes on the supercomputer site. However, this feature requires the attestation of the nodes.\n\n[Several methods of attestation](https://github.com/spiffe/spire/tree/main/doc) exist using Spire. The following are most relevant for HPCS:\n- Token based attestation (user provides a token that is pre-registered to attest the node using it).\n- Slurm based attestation (not in use at the moment, needs first to make sure that slurm is a trustable source of information to attest the node).\n- TPM based attestation ([with DevID](https://github.com/spiffe/spire/blob/main/doc/plugin_agent_nodeattestor_tpm_devid.md) or [without](https://github.com/boxboat/spire-tpm-plugin)).\n- Other hardware based key management based attestation (ex : [sev-snp](https://github.com/ufcg-lsd/spire-amd-sev-snp-node-attestor), in the future).\n\nUsing TPM, for example, it is very easy to run automatic node attestation, based on hardware managed keys that can't be easily spoofed. Unfortunately, LUMI does not provide TPM at the moment and for this reason, node attestation is currently made using a dummy endpoint providing join tokens to anyone. However, this behaviour could easily be modified to strengthen the node attestation with very low code modification for other supercomputers. For example, the node attestation could be performed by admins instead of the present user-initiated attestation.\n\n### Encrypted container\n\nThe goal of this project was to leverage Singularity/Apptainer's [encrypted containers](https://docs.sylabs.io/guides/3.4/user-guide/encryption.html). This feature enables the end user to protect the runtime of the container, allowing it to confine unencrypted data within the encrypted container, adding an extra layer of security.\n\nUnfortunately for LUMI, this feature relies on different technologies, depending the permission level at which the container is encrypted, this behaviour is documented in the following table for usage on LUMI :\n\n| Build \\ Run | root ? | singularity-ce version 3.11.4-1 (LUMI) | apptainer version 1.2.5 (Binary able to be shipped to LUMI) |\n| --- | --- | --- | --- |\n| singularity-ce version 3.11.4 | yes | Unable to decrypt filesystem (no dm_crypt) | Failure (says user namespaces are needed) |\n| singularity-ce version 3.11.4 | no | doesn’t build | doesn’t build |\n| apptainer version 1.2.5 | yes | Unable to decrypt filesystem (no dm_crypt) | Failure (says user namespaces are needed) |\n| apptainer version 1.2.5 | no | Filesystem not recognized | Failure (says user namespaces are needed) |\n\nTwo main reasons for the issues with the encrypted containers :\n- Cannot run as root on a node (no workaround, as this is a feature of HPC environments).\n- User namespaces are disabled on LUMI (for secure reason, [this stackexchange](https://security.stackexchange.com/questions/267628/user-namespaces-do-they-increase-security-or-introduce-new-attack-surface) has some explanations).\n\nTo run encrypted containers as described above, we would need to enable user namespaces on the platform. This would require a thorough risk/benefit assessment, since it introduces new attack surfaces and therefore will not be introduced lightly, at least not on on LUMI in the near future.\n\nWe mitigate the unavailability of encrypted containers in two steps :\n- Encryption of the container at rest (encryption of the image file while stored on the supercomputer, decryption right before runtime)\n- Usage of encrypted FUSE Filesystems in the container. This is achieved using `gocryptfs` (actually the same way as Singularity does it for encrypted containers) but only for some mountpoints. This for example allows us to certify that the input dataset won't ever be written as plaintext on the node as well as the output data.\n\nHowever, this limitation has known solutions (cf. user namespaces) that will be leveraged or not on the platforms. The code was originally written to work with encrypted containers and this code is currently commented out but still available in case of usage on platform supporting user namespaces. Another lead that hasn't been explored as of today is [the newest version of Apptainer](https://github.com/apptainer/apptainer/releases/tag/v1.3.0), introducing new behaviour based on setuid.\n\n### Client attestation\n\nWhen a client wants to encrypt its data or container and to give access to it to someone, it's automatically attested, based on it's public IP. A workload identity is then automatically created, based on the `sha256sum` of the binary calling the workload API or the image_id of the container where the workload is running (See #5). This behaviour represents a problem because this attestation method isn't applicable to every client:\n- Client runs containers using cgroupsv1\n  - Fine, the docker image_id can be used. However, this image_id can be spoofed\n- Client runs containers using cgroupsv2\n  - Client runs on Linux\n    - `spire-agent api fetch` can be attested using spire-agent binary's `sha256sum`\n    - `python3 ./utils/spawn_agent.py` can't be attested since the `sha256sum` recognised by the workload API is `python3`'s. A mitigation to that would be to compile the code, if possible. This would potentially provide a unique binary that would then be able to be attested using `sha256sum`\n  - Client runs on MacOS\n    - No attestation is doable at the moment since MacOS doesn't support docker and runs container inside of a Linux VM\n      - Using cgroupsv2\n      - Replacing every calling binary by the hypervisor\n\nSince this limitation doesn't represent a confidentiality issue (a client isn't ever provided more than a write-only permission), current mitigations are more practical than secure (again, see #5).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fhpcs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcscfi%2Fhpcs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fhpcs/lists"}