{"id":20164079,"url":"https://github.com/southernmethodistuniversity/singularity_docker","last_synced_at":"2026-02-19T05:31:36.266Z","repository":{"id":110252085,"uuid":"251387643","full_name":"SouthernMethodistUniversity/singularity_docker","owner":"SouthernMethodistUniversity","description":"Creating Portable Environments with Singularity and Docker","archived":false,"fork":false,"pushed_at":"2020-04-01T08:34:03.000Z","size":25,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-10-26T05:31:15.888Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SouthernMethodistUniversity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-30T18:04:16.000Z","updated_at":"2020-04-01T08:43:35.000Z","dependencies_parsed_at":"2023-04-05T23:53:05.282Z","dependency_job_id":null,"html_url":"https://github.com/SouthernMethodistUniversity/singularity_docker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SouthernMethodistUniversity/singularity_docker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SouthernMethodistUniversity%2Fsingularity_docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SouthernMethodistUniversity%2Fsingularity_docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SouthernMethodistUniversity%2Fsingularity_docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SouthernMethodistUniversity%2Fsingularity_docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SouthernMethodistUniversity","download_url":"https://codeload.github.com/SouthernMethodistUniversity/singularity_docker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SouthernMethodistUniversity%2Fsingularity_docker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29604552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T05:11:50.834Z","status":"ssl_error","status_checked_at":"2026-02-19T05:11:38.921Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T00:33:10.221Z","updated_at":"2026-02-19T05:31:36.251Z","avatar_url":"https://github.com/SouthernMethodistUniversity.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Creating Portable Environments with Singularity and Docker\n\n## Center for Scientific Computation (CSC)\n\n* Maintains our primary shared resource for research computing, ManeFrame II (M2),\n  in collaboration with OIT\n* Provides research computing tools, support, and training to all faculty, staff,\n  and students using research computing resources\n  [www.smu.edu/csc](https://www.smu.edu/csc) has documentation and news\n* [help@smu.edu](mailto:help@smu.edu) or\n  [rkalescky@smu.edu](mailto:rkalescky@smu.edu) for help\n\n## CSC Workshop Series\n\n|Date         |Workshop                                                     |\n|-------------|-------------------------------------------------------------|\n|January 21   |M2 Introduction                                              |\n|January 28   |Introduction to LAPACK and BLAS                              |\n|February 4   |Text Mining with Python on M2 (Lead by Dr. Eric Godat)       |\n|February 11  |Using the New HPC Portal                                     |\n|February 18  |Using GitHub                                                 |\n|February 25  |Writing Portable Accelerator Code with KOKKOS, RAJA, and OCCA|\n|March 3      |M2 Introduction                                              |\n|March 10     |Introduction to Parallelization Using MPI                    |\n|March 17     |No Workshop Spring Break                                     |\n|March 24     |Writing High Performance Python Code                         |\n|March 31     |Creating Portable Environments with Docker and Singularity   |\n|April 7      |M2 Introduction                                              |\n|April 14     |Introduction to Parallelization Using OpenMP and OpenACC     |\n|April 21     |Profiling Applications on M2                                 |\n|April 28     |Improving Code Vectorization                                 |\n\n## Accessing ManeFrame II (M2) for this Workshop\n\n* Via Terminal or Putty as usual (see [here](http://faculty.smu.edu/csc/documentation/access.html) for details)\n* Via the HPC Portal (Note that this doesn't support X11 forwarding)\n    1. Go to [hpc.smu.edu](https://hpc.smu.edu/).\n    2. Sign in using your SMU ID and SMU password.\n    3. Select \"ManeFrame II Shell Access\" from the \"Clusters\" drop-down menu.\n\n## Introduction to Containers\n\nContainers are a type of operating system virtualization where a single kernel is shared by all operating systems. This is contrasted with virtual machines that provide a virtualized hardware environment upon which an operating system can run. Containers, as such, are light weight compared to virtual machines, but are restricted to a single kernel type, *e.g.* Linux. The primary benefits on containers are:\n\n* Isolation\n* Reproducibility\n* Portability\n\n[Singularity](https://sylabs.io) and [Docker](https://www.docker.com) are popular toolsets used to create and manage containers on Linux. For security reasons, Docker is not usually allowed on HPC systems and this includes M2. M2 does have Singularity, which is largely compatible with Docker containers.\n\n### Singularity and Docker Documentation\n\nThis workshop provides an introduction to containers, specifically Singularity\nand Docker. Additional information for each of these tools can be found at\ntheir respective sites, [Singularity\ndocumentation](https://sylabs.io/guides/3.5/user-guide/) and [Docker\ndocumentation](https://docs.docker.com).\n\n## Singularity\n\n### Definition File Structure\n\n1. Header: The header describes the core operating system upon which\n   everything else will be built. Most frequently the header is simply used to declare\n   the Linux distribution and version that you're interested in using.\n    * **Bootstrap:** Describes from where the initial container image will be sourced\n        * **library** From the [Singularity Container Services](https://cloud.sylabs.io/home)\n        * **docker** From the [Docker Hub](https://hub.docker.com)\n        * **localimage** From a local container image\n        * Others: There are several other options available described [here](https://sylabs.io/guides/3.5/user-guide/definition_files.html#other-bootstrap-agents).\n    * **From:** The name or tag of the initial container image\n    * **Stage:** Support of multi-stage container builds, *e.g.* `devel` and `final`.\n2. Sections: The sections part is compressed of several sections that define\n   what you want to do inside the container.\n   * **%setup** Commands to be run outside the container before the build. The\n     container's root file system can be access via `${SINGULARITY_ROOTFS}`.\n   * **%files** Copy files to the container of the form `\u003csource\u003e \u003cdestination\u003e`\n   * **%environment** Environment variables that will be set at runtime.\n   * **%post** Commands to be run during the containers build process.\n   * **%runscript** Commands to be run via `singularity run` or running the container directly.\n   * **%test** Commands to run at the end of the build process to validate the container image.\n   * **%labels** Add metadata to the container, *e.g.* `Author`, `Version`, and `Label`.\n   * **%help** Text that will be displayed via `singularity run-help`.\n   * **%app** Commands for enabling multiple \"apps\" from a single container.\n\n#### Best Practices\n\n* Install files into operating system standard locations.\n* All files should be owned by system accounts.\n* Document and script the build process rather than making manual changes to a\n  container.\n\n### Building Containers\n\n`singularity build --fakeroot --force \u003ccontainer_file_name\u003e \u003ccontainer_definition_file\u003e`\n\nUntil recently, building Singularity containers required root access. However, recent versions have the `--fakeroot` flag, which enables non-root accounts to build containers. On M2 this requires the `singularity` command from `module load singularity/3.5.3` and it \nmust be run via the \"container\" queue, *e.g.*:\n\n`srun -p container -c 1 --mem=6G --pty singularity build --fakeroot --force \u003ccontainer_file_name\u003e \u003ccontainer_definition_file\u003e`\n\n### Moving Containers\n\n#### Copying\n\nSingularity containers can simply be moved as normal files, *i.e.* the\ncontainers built on one machine and simply copied to and run on another.\n\n#### Singularity Container Services\n\nContainers can be uploaded to the [Singularity Container\nServices](https://cloud.sylabs.io/home). This allows for easy versioning of\ncontainers as you would with files in a Git repository.\n\n`singularity push \u003ccontainer\u003e library://\u003ccontainer_library\u003e`\n\nFree access to [Singularity Container Services](https://cloud.sylabs.io/home) is available.\n\n### Running Containers\n\n* **run** Run the container, which is execute the `%runscript` section of the definition file.\n* **exec** Execute arbitrary commands with in the container.\n* **app** Run the apps defined in the `%apps` section of the definition file.\n* **shell** Execute the container's default shell.\n\n### Running Containers on ManeFrame II (M2)\n\n#### Submitting Jobs Using Containers\n\nSingularity containers can be run similarly to many other applications on M2.\nFor executables defined in the `%runscript` section of the definition file, the\ncontainer can simply be executed as any other executable by giving the path to\nthe container file. Alternatively, the Singularity verbs above can be used to\nrun various executables within the container.\n\n```sh\nmodule load singularity\nsingularity exec \u003cpath_to_container\u003e python3 python_script.py\n```\n\nHere, a full or relative container file path can be given. The executable is\n`python3`, which resides inside the container. The Python script\n`python_script.py` resides outside the container.\n\nIf `python3` is defined to run inside the `%runscript` section of the definition\nfile the following would be equivalent to the example above.\n\n```sh\nmodule load singularity\n\u003cpath_to_container\u003e python_script.py\n```\n\nBoth of these methods can be used with `srun` and `sbatch`.\n\n### Examples\n\nTo build and run:\n\n```sh\nmodule load singularity/3.5.3\ngit clone https://github.com/SouthernMethodistUniversity/singularity_docker.git\ncd singularity_docker/examples\nsrun -p container -c 1 --mem=6G --pty singularity build --fakeroot --force compliance.sif compliance.singularity\nsrun -p container -c 1 --mem=6G --x11=first --pty ./compliance.sif # Won't work from HPC portal shell access, no X11\nsrun -p container -c 1 --mem=6G --pty singularity build --fakeroot --force anaconda.sif anaconda.singularity\nsrun -p container -c 1 --mem=6G --pty ./anaconda.sif matrix_multiplication.py\nsbatch anaconda.sbatch\n```\n\n### Tasks\n\n1. Run the above examples.\n2. Create a Singularity definition file to run gnuplot. Hint: `apt-get -y install gnuplot` for Debian-based Linux distributions such as Ubuntu.\n\n## Docker\n\n### Definition File Structure\n\n* **FROM** The name or tag of the initial container image.\n* **COPY** Copy files to the container of the form `\u003csource\u003e \u003cdestination\u003e`.\n* **RUN** Commands to be run during the containers build process.\n* **ENV** Environment variables that will be set during build and at runtime.\n* **CMD** Command to be executed at runtime.\n* **LABEL** Add metadata to the container.\n\n### Building Containers\n\n`docker build -t \u003ccontainer_tag\u003e -f \u003ccontainer_definition_file\u003e \u003ccontainer_definition_path\u003e`\n\n### Moving Containers\n\n`docker push \u003ccontainer_tag\u003e`\n\n### Running Containers\n\n`docker run \u003ccontainer_tag\u003e`\n\n### Examples\n\nTo build and run on your own computer as Docker is not supported on M2, but\nDocker images can be consumed by Singularity on M2.\n\n```sh\ngit clone https://github.com/SouthernMethodistUniversity/singularity_docker.git\ncd singularity_docker/examples\ndocker build -t \u003cdocker_username\u003e/anaconda -f anaconda.docker .\ndocker run -v $(pwd)/matrix_multiplication.py:/input_file \u003cdocker_username\u003e/anaconda:latest\ndocker push \u003cdocker_username\u003e/anaconda:latest\n```\n\nTo use the container on M2, log into M2 and then:\n\n```sh\nmodule load singularity/3.5.3\ngit clone https://github.com/SouthernMethodistUniversity/singularity_docker.git\ncd singularity_docker/examples\nsrun -p container -c 1 --mem=6G --pty singularity build --fakeroot --force anaconda_from_docker.sif anaconda_from_docker.singularity\nsrun -p container -c 1 --mem=6G --pty ./anaconda_from_docker.sif matrix_multiplication.py\n```\n\n### Tasks\n\n1. Run the above examples.\n2. Create a Docker definition file to run gnuplot and run via Singularity on M2. Hint: `apt-get -y install gnuplot` for Debian-based Linux distributions such as Ubuntu.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsouthernmethodistuniversity%2Fsingularity_docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsouthernmethodistuniversity%2Fsingularity_docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsouthernmethodistuniversity%2Fsingularity_docker/lists"}