https://github.com/southernmethodistuniversity/singularity_docker
Creating Portable Environments with Singularity and Docker
https://github.com/southernmethodistuniversity/singularity_docker
Last synced: 4 months ago
JSON representation
Creating Portable Environments with Singularity and Docker
- Host: GitHub
- URL: https://github.com/southernmethodistuniversity/singularity_docker
- Owner: SouthernMethodistUniversity
- Created: 2020-03-30T18:04:16.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2020-04-01T08:34:03.000Z (over 6 years ago)
- Last Synced: 2025-10-26T05:31:15.888Z (8 months ago)
- Homepage:
- Size: 24.4 KB
- Stars: 1
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Creating Portable Environments with Singularity and Docker
## Center for Scientific Computation (CSC)
* Maintains our primary shared resource for research computing, ManeFrame II (M2),
in collaboration with OIT
* Provides research computing tools, support, and training to all faculty, staff,
and students using research computing resources
[www.smu.edu/csc](https://www.smu.edu/csc) has documentation and news
* [help@smu.edu](mailto:help@smu.edu) or
[rkalescky@smu.edu](mailto:rkalescky@smu.edu) for help
## CSC Workshop Series
|Date |Workshop |
|-------------|-------------------------------------------------------------|
|January 21 |M2 Introduction |
|January 28 |Introduction to LAPACK and BLAS |
|February 4 |Text Mining with Python on M2 (Lead by Dr. Eric Godat) |
|February 11 |Using the New HPC Portal |
|February 18 |Using GitHub |
|February 25 |Writing Portable Accelerator Code with KOKKOS, RAJA, and OCCA|
|March 3 |M2 Introduction |
|March 10 |Introduction to Parallelization Using MPI |
|March 17 |No Workshop Spring Break |
|March 24 |Writing High Performance Python Code |
|March 31 |Creating Portable Environments with Docker and Singularity |
|April 7 |M2 Introduction |
|April 14 |Introduction to Parallelization Using OpenMP and OpenACC |
|April 21 |Profiling Applications on M2 |
|April 28 |Improving Code Vectorization |
## Accessing ManeFrame II (M2) for this Workshop
* Via Terminal or Putty as usual (see [here](http://faculty.smu.edu/csc/documentation/access.html) for details)
* Via the HPC Portal (Note that this doesn't support X11 forwarding)
1. Go to [hpc.smu.edu](https://hpc.smu.edu/).
2. Sign in using your SMU ID and SMU password.
3. Select "ManeFrame II Shell Access" from the "Clusters" drop-down menu.
## Introduction to Containers
Containers are a type of operating system virtualization where a single kernel is shared by all operating systems. This is contrasted with virtual machines that provide a virtualized hardware environment upon which an operating system can run. Containers, as such, are light weight compared to virtual machines, but are restricted to a single kernel type, *e.g.* Linux. The primary benefits on containers are:
* Isolation
* Reproducibility
* Portability
[Singularity](https://sylabs.io) and [Docker](https://www.docker.com) are popular toolsets used to create and manage containers on Linux. For security reasons, Docker is not usually allowed on HPC systems and this includes M2. M2 does have Singularity, which is largely compatible with Docker containers.
### Singularity and Docker Documentation
This workshop provides an introduction to containers, specifically Singularity
and Docker. Additional information for each of these tools can be found at
their respective sites, [Singularity
documentation](https://sylabs.io/guides/3.5/user-guide/) and [Docker
documentation](https://docs.docker.com).
## Singularity
### Definition File Structure
1. Header: The header describes the core operating system upon which
everything else will be built. Most frequently the header is simply used to declare
the Linux distribution and version that you're interested in using.
* **Bootstrap:** Describes from where the initial container image will be sourced
* **library** From the [Singularity Container Services](https://cloud.sylabs.io/home)
* **docker** From the [Docker Hub](https://hub.docker.com)
* **localimage** From a local container image
* Others: There are several other options available described [here](https://sylabs.io/guides/3.5/user-guide/definition_files.html#other-bootstrap-agents).
* **From:** The name or tag of the initial container image
* **Stage:** Support of multi-stage container builds, *e.g.* `devel` and `final`.
2. Sections: The sections part is compressed of several sections that define
what you want to do inside the container.
* **%setup** Commands to be run outside the container before the build. The
container's root file system can be access via `${SINGULARITY_ROOTFS}`.
* **%files** Copy files to the container of the form ` `
* **%environment** Environment variables that will be set at runtime.
* **%post** Commands to be run during the containers build process.
* **%runscript** Commands to be run via `singularity run` or running the container directly.
* **%test** Commands to run at the end of the build process to validate the container image.
* **%labels** Add metadata to the container, *e.g.* `Author`, `Version`, and `Label`.
* **%help** Text that will be displayed via `singularity run-help`.
* **%app** Commands for enabling multiple "apps" from a single container.
#### Best Practices
* Install files into operating system standard locations.
* All files should be owned by system accounts.
* Document and script the build process rather than making manual changes to a
container.
### Building Containers
`singularity build --fakeroot --force `
Until recently, building Singularity containers required root access. However, recent versions have the `--fakeroot` flag, which enables non-root accounts to build containers. On M2 this requires the `singularity` command from `module load singularity/3.5.3` and it
must be run via the "container" queue, *e.g.*:
`srun -p container -c 1 --mem=6G --pty singularity build --fakeroot --force `
### Moving Containers
#### Copying
Singularity containers can simply be moved as normal files, *i.e.* the
containers built on one machine and simply copied to and run on another.
#### Singularity Container Services
Containers can be uploaded to the [Singularity Container
Services](https://cloud.sylabs.io/home). This allows for easy versioning of
containers as you would with files in a Git repository.
`singularity push library://`
Free access to [Singularity Container Services](https://cloud.sylabs.io/home) is available.
### Running Containers
* **run** Run the container, which is execute the `%runscript` section of the definition file.
* **exec** Execute arbitrary commands with in the container.
* **app** Run the apps defined in the `%apps` section of the definition file.
* **shell** Execute the container's default shell.
### Running Containers on ManeFrame II (M2)
#### Submitting Jobs Using Containers
Singularity containers can be run similarly to many other applications on M2.
For executables defined in the `%runscript` section of the definition file, the
container can simply be executed as any other executable by giving the path to
the container file. Alternatively, the Singularity verbs above can be used to
run various executables within the container.
```sh
module load singularity
singularity exec python3 python_script.py
```
Here, a full or relative container file path can be given. The executable is
`python3`, which resides inside the container. The Python script
`python_script.py` resides outside the container.
If `python3` is defined to run inside the `%runscript` section of the definition
file the following would be equivalent to the example above.
```sh
module load singularity
python_script.py
```
Both of these methods can be used with `srun` and `sbatch`.
### Examples
To build and run:
```sh
module load singularity/3.5.3
git clone https://github.com/SouthernMethodistUniversity/singularity_docker.git
cd singularity_docker/examples
srun -p container -c 1 --mem=6G --pty singularity build --fakeroot --force compliance.sif compliance.singularity
srun -p container -c 1 --mem=6G --x11=first --pty ./compliance.sif # Won't work from HPC portal shell access, no X11
srun -p container -c 1 --mem=6G --pty singularity build --fakeroot --force anaconda.sif anaconda.singularity
srun -p container -c 1 --mem=6G --pty ./anaconda.sif matrix_multiplication.py
sbatch anaconda.sbatch
```
### Tasks
1. Run the above examples.
2. Create a Singularity definition file to run gnuplot. Hint: `apt-get -y install gnuplot` for Debian-based Linux distributions such as Ubuntu.
## Docker
### Definition File Structure
* **FROM** The name or tag of the initial container image.
* **COPY** Copy files to the container of the form ` `.
* **RUN** Commands to be run during the containers build process.
* **ENV** Environment variables that will be set during build and at runtime.
* **CMD** Command to be executed at runtime.
* **LABEL** Add metadata to the container.
### Building Containers
`docker build -t -f `
### Moving Containers
`docker push `
### Running Containers
`docker run `
### Examples
To build and run on your own computer as Docker is not supported on M2, but
Docker images can be consumed by Singularity on M2.
```sh
git clone https://github.com/SouthernMethodistUniversity/singularity_docker.git
cd singularity_docker/examples
docker build -t /anaconda -f anaconda.docker .
docker run -v $(pwd)/matrix_multiplication.py:/input_file /anaconda:latest
docker push /anaconda:latest
```
To use the container on M2, log into M2 and then:
```sh
module load singularity/3.5.3
git clone https://github.com/SouthernMethodistUniversity/singularity_docker.git
cd singularity_docker/examples
srun -p container -c 1 --mem=6G --pty singularity build --fakeroot --force anaconda_from_docker.sif anaconda_from_docker.singularity
srun -p container -c 1 --mem=6G --pty ./anaconda_from_docker.sif matrix_multiplication.py
```
### Tasks
1. Run the above examples.
2. Create a Docker definition file to run gnuplot and run via Singularity on M2. Hint: `apt-get -y install gnuplot` for Debian-based Linux distributions such as Ubuntu.