Examples of Jobs on Quest
On May 1st, 2019 the Quest scheduler changed from Moab to Slurm. Researchers using Quest will have to update their job submission scripts and use different commands to run under the new scheduler.
Additional details and examples of Quest job submission scripts and commands.
See Research Computing's GitHub repository of example jobs for language/software specific examples and scripts you can copy and modify.
Interactive Jobs
Example: Interactive Job with More than One Node
If you submit an interactive job requiring more than one node, you will land on the Head Node for your job once the scheduler finds the resources. For more information see Submitting an Interactive Job.
[abc123@quser22 ~]$ srun -A p12345 -p normal -N 2 --tasks-per-node=10 --mem-per-cpu=1G --time=01:00:00 --pty bash -l ---------------------------------------- srun job start: Thu Feb 28 16:31:27 CST 2019 Job ID: 557781 Username: abc123 Queue: normal Account: p12345 ---------------------------------------- The following variables are not guaranteed to be the same in prologue and the job run script ---------------------------------------- PATH (in prologue) : /hpc/slurm/usertools:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/home/abc123/bin WORKDIR is: /home/abc123 ---------------------------------------- [abc123@qnode4217 ~]$
This interactive job requests 2 nodes and 10 cores in each of those nodes. In total you will have access to 20 cores. Since the job reserved 1 GB per core, 20 GB of RAM is allocated in total (i.e. 10 GB in each node).
To find the name of the other compute node(s) that you have been assigned as part of your job, you can use the squeue command in a from the same terminal session. Look under NODELIST in the squeue output:
[abc123@qnode4217 ~]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 557781 normal bash abc123 R 5:51 2 qnode[4217-4218]
You can then ssh to the other node(s) if needed as long as your job is active.
ssh qnode4218
Batch Jobs
Job Array
Job arrays can be used to submit multiple jobs at once that use the same application script. This can be useful if you want to run the same script multiple times with different input parameters.
In the example below, the --array option defines the job array, with a specification of the index numbers you want to use (in this case, 1 through 50 with 10 elements running simultaneously anytime). The $SLURM_ARRAY_TASK_ID variable takes on the values of the job array index for each job (so here, integer values 1 through 50, one value for each job). In this example, the value of $SLURM_ARRAY_TASK_ID is used to print out a line from a CSV input file, whose fields are then passed on to a script as command line arguments.
jobsubmission.sh
#!/bin/bash #SBATCH -A <allocationID> #SBATCH -p <partitionName> #SBATCH -n 1 #SBATCH -t <hh:mm:ss>
#SBATCH --mem=<total memory>G # Reserving a total amount of memory for each job within the array #SBATCH --array=1-50%10 # make sure the array number matches the number of lines in your input file! #SBATCH --error=arrayJob_%A_%a.err #SBATCH --output=arrayJob_%A_%a.out #SBATCH -J <jobName> module purge module load anaconda3/2018.12 python ./myscript.py $(sed -n ${SLURM_ARRAY_TASK_ID}p input.csv | sed 's/,/ /g')
myscript.py
import sys
first_argument = sys.argv[1]
second_argument = sys.argv[2]
print("The first argument is ", first_argument)
print("The second argument is ", second_argument)
In this example, myscript.py will receive the values in input.csv as arguments: the first field will be sys.argv[1], the second field will be sys.argv[2], etc.
Note: make sure the number you specify for the --array parameter matches the number of lines in your input file!
Also, note that in this example standard output and error files are printed separately for each element of the job array with the --output and --error options. To avoid each element overwriting these files, tag them with jobID (%A) and elementID (%a) variables (which are automatically assigned by the scheduler) so elements have their own distinct output and error files.
Submit this script with:
sbatch jobsubmission.sh
The job array will then be submitted to the scheduler.
Software Applications
Below are examples of importing modules and calling various software programs in a submission script. These lines come below the SBATCH directives, with at least a single empty line between the end of the scheduler options and the start of commands specifying the work of the job.
There may be other ways to run these programs as well. These commands are only provided as a convenient reference.
Python
module load python/anaconda python myscript.py
There are multiple versions of Python available on Quest. module load python/anaconda will load Python 2.7. For other versions of Python, use the appropriate Anaconda Python module.
R
module load R/3.3.1 Rscript mycode.R
See Using R on Quest for more information and examples.
MATLAB
module load matlab matlab -nosplash -nodesktop -singleCompThread -r 'commands;exit'
or
module load matlab matlab -nosplash -nodesktop -singleCompThread -r myscript
Where myscript is a .m file. Note that there are multiple versions of MATLAB. You should specify the version you want, as the default may change over time.
Stata
module load stata/14 stata-mp < myprog.do > myprog.log
or
module load stata/14 stata-mp -b do myprog
Stata will read its commands from myprog.do and write its output to myprog.log in the working directory.
Note that Stata-MP is licensed for 4 cores, so please request exactly 4 cores per job in your job submission script. If you request more, you'll be wasting resources; if you request less and do not set the corresponding options appropriately in your script, Stata may use more cores than you've been assigned, resulting in your job being terminated. If you don't need Stata-MP's parallelization capabilities, you can run Stata SE with stata instead.
SAS
module load sas sas myprog.sas
SAS can write very large temporary files. You can redirect such files to your project space with the -work option when you start-up SAS.
Mathematica
module load mathematica math -script input.txt > output.txt