{"id":18825484,"url":"https://github.com/grycap/hpcmeetsdocker","last_synced_at":"2026-03-17T19:24:16.715Z","repository":{"id":149748423,"uuid":"81338119","full_name":"grycap/HPCmeetsDocker","owner":"grycap","description":null,"archived":false,"fork":false,"pushed_at":"2018-07-16T11:08:29.000Z","size":31,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2026-02-12T07:04:24.232Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grycap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-02-08T14:27:42.000Z","updated_at":"2020-07-06T22:56:26.000Z","dependencies_parsed_at":"2023-04-25T03:39:14.274Z","dependency_job_id":null,"html_url":"https://github.com/grycap/HPCmeetsDocker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/grycap/HPCmeetsDocker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grycap%2FHPCmeetsDocker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grycap%2FHPCmeetsDocker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grycap%2FHPCmeetsDocker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grycap%2FHPCmeetsDocker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grycap","download_url":"https://codeload.github.com/grycap/HPCmeetsDocker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grycap%2FHPCmeetsDocker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30629196,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-17T17:32:55.572Z","status":"ssl_error","status_checked_at":"2026-03-17T17:32:38.732Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T00:59:41.864Z","updated_at":"2026-03-17T19:24:16.708Z","avatar_url":"https://github.com/grycap.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HPCmeetsDocker\n\nRunning HPC workloads using Docker.\n\n## Create the HPCmeetsDocker image\n\nFirst we get HPCmeetsDocker from git:\n\n```bash\ngit clone https://github.com/grycap/HPCmeetsDocker\n``` \n\nNow we create a container and push the files needed to create the installation\n\n```bash\nlxc launch local:ubuntu:16.04 HPCmD -p default -p docker\nlxc exec HPCmD -- mkdir -p /opt/HPCmeetsDocker/install \ntar c -C ./HPCmeetsDocker/install/ . | lxc exec HPCmD -- tar xf - -C /opt/HPCmeetsDocker/install\n```\n\nEnter the container and execute the installation\n\n```bash\nlxc exec HPCmD -- bash -c 'cd /opt/HPCmeetsDocker/install\n./01install-docker\n./02install-munge\n./03compile-slurm\n./04install-mpi-python\n'\n```\n\nNow stop the container and publish the golden image\n\n```bash\nlxc stop HPCmD\nlxc publish HPCmD local: --alias HPCmD:0.1\n```\n\n## Using with MCC\n\n[MCC](https://github.com/grycap/mcc) is a tool that automates the process of creating virtual clusters whose nodes are containers. At this point, it is based on LXD. In order to test _HPCmeetsDocker_, we suggest to create a virtual cluster with _MCC_ prior to use it in production. Let's proceed.\n\nIn first place, we create a profile for mcc and I make it privileged for my purposes\n\n```bash\nlxc profile create HPCmD\nlxc profile device add HPCmD aadisable disk path=/sys/module/apparmor/parameters/enabled source=/dev/null\nlxc profile set HPCmD security.nesting true\nlxc profile set HPCmD linux.kernel_modules 'overlay, nf_nat'\nlxc profile set HPCmD security.privileged true\n```\n\nYou can issue the whole profile, instead:\n\n```bash\ncat \u003c\u003c\\EOF | lxc profile edit HPCmD\nname: HPCmD\nconfig:\n  linux.kernel_modules: overlay, nf_nat\n  security.nesting: \"true\"\n  security.privileged: \"true\"\ndescription: Profile for HPCmD, that supports Docker inside the containers and is privileged (it is a copy of the docker profile, setting the privilege to true)\ndevices:\n  aadisable:\n    path: /sys/module/apparmor/parameters/enabled\n    source: /dev/null\n    type: disk\nEOF\n```\n\nAnd then we use the next command to launch the cluster:\n\n```bash\nMCC_LXC_LAUNCH_OPTS=\"-p HPCmD\" mcc --verbose create --front-end-image local:HPCmD:0.1 --context-folder ./HPCmeetsDocker/ -n 2 -e -d home\n```\n\nAnd now we have a cluster that consists of 2 computing nodes, with a shared home folder. The computing nodes have docker and slurm installed, and a single user called _ubuntu_ that is able to launch docker containers.\n\n## Testing the cluster\n\n### Basic testing\n\nYou can issue a command like this one, that runs a docker container on each of the two nodes. And the best of it is that Slurm managed the execution and allocation of nodes.\n\n```bash\nsrun -N 2 docker run alpine sh -c 'hostname'\n```\n\n### OpenFOAM\n\nFirst grab the openfoam image (can be done as root or as any other user)\n\n```bash\ndocker pull openfoam/openfoam4-paraview50\nssh node1 docker pull openfoam/openfoam4-paraview50\nssh node2 docker pull openfoam/openfoam4-paraview50\n```\nThen, as a user, we create a script named ```job.sh``` that we will use to run openfoam in slurm\n\n```bash\n#!/bin/bash\n\nsrun docker run --rm --entrypoint '/bin/bash' -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v /home/ubuntu:/home/openfoam -u $(id -u):$(id -g) openfoam/openfoam4-paraview50 -c '. /opt/openfoam4/etc/bashrc\nmkdir -p $FOAM_RUN \nrun \ncp -r $FOAM_TUTORIALS/incompressible/icoFoam/cavity/cavity . \ncd cavity \nblockMesh \nicoFoam'\n```\n\nIf we had our folder with our OpenFOAM case, we could change that ```job.sh``` command by the next fragment\n\n```bash\n#!/bin/bash\n\ncd cavity\nsrun docker run --rm --entrypoint '/bin/bash' -w \"$(pwd)\" -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v /home:/home -u $(id -u):$(id -g) openfoam/openfoam4-paraview50 -c '. /opt/openfoam4/etc/bashrc\nblockMesh \nicoFoam'\n```\n\nThen we could submit the job to slurm:\n\n```bash\nsbatch job.sh\n```\n\n## Using MPI\n\n### Ensuring that MPI is running\n\nFirst of all, you must ensure that you are able to run MPI. In order to make it, a basic set-up is included here:\n\nInstalling MPICH:\n```bash\napt-get install -y mpich openssh-client openssh-server python python-pip\npip install --upgrade pip mpi4py\n```\n\nPreparing the passwordless ssh access. We assume that the /home folder is shared between the different hosts\n```bash\nsu - ubuntu\nssh-keygen\ncat .ssh/id_rsa.pub \u003e\u003e .ssh/authorized_keys\nchmod 400 .ssh/authorized_keys\ncat \u003e .ssh/config \u003c\u003c EOF\nHost node*\nStrictHostKeyChecking no\nUserKnownHostsFile /dev/null\nLogLevel QUIET\nEOF\n```\n\nRunning a basic test in python:\n```bash\ncat \u003e mpi_hello.py \u003c\u003c\\EOF\nfrom mpi4py import MPI\ncomm = MPI.COMM_WORLD\nrank = comm.Get_rank()\nprint \"hello world from process \", rank\nimport socket\nprint(socket.gethostname())\nEOF\n\nmpirun -np 2 -H node1,node2 python mpi_hello.py\n```\n\nAnd this is our basic setup for a MPI platform.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrycap%2Fhpcmeetsdocker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrycap%2Fhpcmeetsdocker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrycap%2Fhpcmeetsdocker/lists"}