Using Singularity on Quest
Container technology offers a way to package software along with its dependencies in a format that can be run by any computer. The advantages of containers include portability, reproducibility, and freedom for users to run their software of choice on any system, without having to rely on system administrators to install, configure, and maintain software. Singularity is a software container technology designed for scientific workloads on traditional HPC environments such as Quest. In addition to the benefits offered by other container technologies, Singularity has security features that limit the privileges of a container to be the same as the user running the container. This prevents containers from affecting shared resources such as hardware or networking that could impact other users. Singularity also uses a simple image format that makes moving images as easy as copying a single file.
Load the Singularity Module
Singularity is installed as a module on Quest. To use it, either interactively or in a submission script, you must first load the module:
module load singularity
Getting Singularity Images On Quest
Downloading Container Images From Singularity Hub
The Singularity project offers a free service called the Singularity Hub for hosting and building container images.
Images on the Singularity Hub are grouped into "collections," which consist of one or more images that are built from the same GitHub repository. In general, most collections consist of a single image that represents a single software application.
Use the singularity pull command to pull images from the Singularity Hub. For example:
singularity pull shub://nuitrcs/biobakery
This will create a file called nuitrcs-biobakery-master-latest.simg that is a Singularity image. It is possible to move or copy this file like any other file, and run it on any computer with Singularity installed.
Importing Images From Docker
Singularity can also pull images from Docker registries, including the public Docker Hub. For example:
singularity pull docker://biocontainers/blast
By default Singularity will pull the image with the tag latest but it is possible to pull a specific tag:
singularity pull docker://biocontainers/blast:v2.2.31_cv1.12
This will create blast-v2.2.31_cv1.12.simg.
Building Custom Singularity Images
The singularity build command is used to create custom containers. In general, singularity build requires sudo privileges and thus cannot be run on Quest. You will need to run it on a Linux computer with sudo privileges. A virtual machine created with the free Virtualbox software or a cloud computing instance will work.
Singularity can build images in a variety of ways, including creating a "writable" image that can be set up and configured interactively and then saved. However the preferred way to create Singularity images is with a Singularity Recipe file. Refer to the Singularity documentation for detailed description of the Recipe file format: https://www.sylabs.io/guides/2.5/user-guide/container_recipes.html
One advantage of the Recipe file format is that they can be used to build images automatically via the Singularity Hub if hosted in a GitHub repository.
Running Singularity Containers
There are multiple ways to run a Singularity container.
To run an interactive shell in a container (to explore the container's filesystem or experiment with it), use the singularity shell command. For example to run the blast.simg container created above:
singularity shell blast.simg
By default all environment variables (such as $HOME) from the host shell session where you execute singularity shell will be passed into the container shell environment unless they have been defined in the container already (e.g. by a Singularity recipe file or Dockerfile). The -e flag to singularity shell will run a shell in the container with a "clean" environment, that is without passing in environment variables from the host. For more details, see the Singularity documentation.
Run The Default Command With run
Singularity recipes have a %runscript section which defines exactly which command will run when the container is run with the singularity run command or directly as an executable file. When an image is imported from Docker, the Dockerfile's ENTRYPOINT instruction is used as the runscript. For example, the nuitrcs/hello-world container's runscript is a shell script that prints "Hello world!":
singularity run shub://nuitrcs/hello-world Hello world!
Specify The Command To Run With exec
Some containers do not have a runscript or may have multiple executables that you need to choose from. The singularity exec command is used to specify the exact command to run inside the container and is the preferred way to run singularity containers:
singularity exec blast.simg blastp -version blastp: 2.2.31+ Package: blast 2.2.31, build Apr 23 2016 15:49:47
Singularity allows you to map directories on the host system to directories within your container using bind mounts. This allows you to read and write data on the host system with ease. On Quest, $HOME, /tmp, and /proc are mapped by default from the host into the container.
The --bind or -B option to the Singularity shell, run, and exec commands also allows the user to specify additional directories to map into the container. This option's argument is a comma-delimited string of bind path specifications in the format src[:dest[:opts]], where src and dest are outside (that is, on the host) and inside (in the container) paths. If dest is not given, it is set equal to src. Mount options (opts) may be specified as ro (read-only) or rw (read/write, which is the default). The --bind/-B option can be specified multiple times, or a comma-delimited string of bind path specifications can be used:
singularity shell -B /projects/pXXXXX:/project shub://nuitrcs/hello-world
In this example, the contents of /projects/pXXXXX on Quest would be accessible inside the container as /project.
Singularity Containers In Batch Jobs
Singularity commands can be run in a batch submission script just like any other command. One approach to Singularity is to pull images at runtime and run them from a temporary directory, so they do not take up space on the system when not being used. Here is one such example job submission script:
#!/bin/bash #SBATCH -A <allocation ID> #SBATCH -p <partition name> #SBATCH -t 02:00:00 #SBATCH -N 1 #SBATCH -n 1 #SBATCH --mem=4G #SBATCH --job-name="<job name>" module load singularity # Use a temporary directory when pulling container image export SINGULARITY_CACHEDIR=$TMPDIR # Run the container singularity exec -B /project/pXXXXX/data:/data docker://biocontainers/blast blastx -query /data/seqs.fasta -out /data/output.blast.txt
This script assumes the seqs.fasta file is in the /project/pXXXXX/data directory, which is mapped into the container as /data using the -B option. The output will be written to /project/pXXXXX/data/output.blast.txt.
Singularity supports GPU hardware via the --nv option, which uses the Nvidia drivers installed on the host system inside the container. If you have access to a GPU allocation on Quest, you can utilize it within a Singularity container. For example:
git clone https://github.com/tensorflow/models.git singularity exec --nv docker://tensorflow/tensorflow:latest-gpu \ python ./models/tutorials/image/mnist/convolutional.py
Singularity does integrate with the Message Passing Interface (MPI) and OpenMPI version 2.1 is supported out of the box. Quest's mpi/openmpi-2.1.1-gcc-5.1.0 module should be used for best results, and OpenMPI will need to be installed inside the container. Assuming OpenMPI is installed in the container, running MPI programs within a Singularity container is as simple as running mpirun on the host:
mpirun -np 4 singularity exec my_mpi_container.simg /script/to/run
- Custom container images should contain a single application. Do not try to bundle many different executables in your image.
- If you do need to bundle multiple applications into an image (to contain an entire pipeline for example), use the Scientific Filesystem integration and %appenv directives in your Singularity recipe file to keep the image as modular as possible. See the Singularity documentation for more details: https://www.sylabs.io/guides/2.5/user-guide/reproducible_scif_apps.html
- Use singularity exec to specify which command to run in your container rather than relying on singularity run.
- When creating custom Singularity images, don't put anything in $TMP or $HOME as those directories will be bind mounted at runtime.
- Read the Singularity user documentation for more detail on all Singularity functionality.