Examples of Jobs on Quest
On May 1st, 2019 the Quest scheduler will change from Moab to Slurm. Researchers using Quest will have to update their job submission scripts and use different commands to run under the new scheduler.
Click here for the schedule of Slurm workshops offered during the transition period. For more information, please see the project page.
To view the Slurm version of this documentation, click here.
Additional details and examples of Quest job submission scripts and commands.
This page includes additional details and examples of job submission scripts and commands.
See Research Computing's GitHub repository of example jobs for language/software specific examples and scripts you can copy and modify.
Interactive Jobs
Example: Interactive Job with More than One Node
If you submit an interactive job requiring more than one node, you will land on the Master Node for your job once the scheduler finds the resources. For more information see Submitting an Interactive Job.
[<netid>@quser11 ~]$ msub -I -l nodes=2:ppn=20 -l walltime=1:00:00 -q short qsub: waiting for job 19901348.qsched03.quest.it.northwestern.edu to start qsub: job 19901348.qsched03.quest.it.northwestern.edu ready ---------------------------------------- PBS: Begin PBS Prologue Wed May 10 13:16:50 CDT 2017 1494440210 PBS: Job ID: 19901348.qsched03.quest.it.northwestern.edu PBS: Username: <netid> PBS: Group: <netid> PBS: Executing queue: short PBS: Job name: STDIN PBS: Account: <allocationID> ---------------------------------------- The following variables are not guaranteed to be the same in prologue and the job run script ---------------------------------------- PBS: Temporary Dir($TMPDIR): /tmp/19901348.qsched03.quest.it.northwestern.edu PBS: Master Node($PBS_MSHOST): qnode5056 PBS: node file($PBS_NODEFILE): /hpc/opt/torque6/nodes/qnode5056/aux//19901348.qsched03.quest.it.northwestern.edu PBS: PATH (in prologue) : /bin:/usr/bin PBS: WORKDIR ($PBS_O_WORKDIR) is: /home/<netid> ---------------------------------------- PBS: End PBS Prologue Wed May 10 13:16:50 CDT 2017 1494440210 [<netid>@qnode5056 ~]$
To find the name of the other compute node(s) that you have been assigned as part of your job, you can use the checkjob command in a separate terminal session connected to a Quest login node. checkjob does not work from the compute nodes. Look under Allocated Nodes:.
[<netid>@quser10 ~]$ checkjob 19901348 job 19901348 AName: STDIN State: Running Creds: user:<netid> group:<netid> account:<allocationID> class:short qos:<allocationID>qos WallTime: 00:02:34 of 1:00:00 SubmitTime: Wed May 10 13:15:26 (Time Queued Total: 00:01:24 Eligible: 00:01:09) StartTime: Wed May 10 13:16:50 TemplateSets: DEFAULT NodeMatchPolicy: EXACTNODE Total Requested Tasks: 40 Req[0] TaskCount: 40 Partition: quest5 Allocated Nodes: [qnode5056:20][qnode5057:20] SystemID: Moab SystemJID: Moab.1655563 Notification Events: JobFail StartCount: 1 BypassCount: 30 Partition List: quest3,quest4,quest5,quest6 Flags: BACKFILL,ADVRES:admin1,SUSPENDABLE,INTERACTIVE Attr: BACKFILL,INTERACTIVE,checkpoint Variables: UsageRecord=9889320 StartPriority: 258 IterationJobRank: 0 Reservation '19901348' (-00:02:57 -> 00:57:03 Duration: 1:00:00)
You can then ssh to the other node(s) if needed as long as your job is active.
ssh qnode5057
Batch Jobs
Example: Job Array
Job arrays can be used to submit multiple jobs at once that use the same application script. This can be useful if you want to run the same script multiple times with different input parameters.
NOTE: While job arrays are convenient for submitting jobs, monitoring jobs submitted to Quest as part of job array can be difficult, as the normal monitoring commands will only give you information on the array of jobs, not the individual jobs. This makes it difficult for you, or Research Computing staff to help you, if something goes wrong with your job. A better approach may be to use a loop with a bash script to submit multiple jobs (see Example: Submitting Multiple Jobs below).
In the example below, the -t option defines the job array, with a name you pick and a specification of the index numbers you want to use (in this case, 1 through 25). With the option #PBS -V, the MOAB_JOBARRAYINDEX variable takes on the values of the job array index for each job (so in this case, integer values 1 through 25, one value for each job). In this example, those values are then passed on to the R script as command line arguments.
jobsubmission.sh
#!/bin/bash
#MSUB -A <allocationID>
#MSUB -q short
#MSUB -l walltime=04:00:00
#MSUB -M <email>
#MSUB -j oe
#MSUB -N <jobname>
#MSUB -l procs=1
#MSUB -t <arrayname>[1-25]
#PBS -V MOAB_JOBARRAYINDEX
module load R
# Set your working directory
cd $PBS_O_WORKDIR
Rscript myscript.R ${MOAB_JOBARRAYINDEX}
The script above would be submitted with:
msub jobsubmission.sh
And 25 jobs would be submitted to the scheduler.
To supply more than just the job array index as a command line argument to your script, consider saving the other parameters in a text or csv file and having your script use the value of MOAB_JOBARRAYINDEX to read the correct line from that file and retrieve the appropriate parameter values.
Example: Submitting Multiple Jobs
If you want to submit multiple jobs using the same submission script, but sending different parameters to your code, you can write a bash script to do this. Get the script below from the example jobs repository.
multijobs.sh
#!/bin/bash
while read P1 P2
do
JOB=`msub - << EOJ
#MSUB -A <allocationID>
#MSUB -q short
#MSUB -l walltime=00:05:00
#MSUB -M <email>
#MSUB -j oe
#MSUB -N <jobname>
#MSUB -l nodes=1:ppn=1
module load R/3.3.1
Rscript testR.R ${P1} ${P2}
EOJ
`
echo "JobID = ${JOB} for parameters ${P1} ${P2} submitted on `date`"
done < params.txt
exit
Note the two highlighted tick marks are where they should be; they surround the submission script. Make multijobs.sh executable with:
chmod u+x multijobs.sh
The script reads parameters from params.txt, which is a space delimited file:
params.txt
AA 1 BBB 2 CCCC 3
One job is created for each line in params.txt. The values are sent to testR.R:
testR.R
args <- commandArgs(trailingOnly = TRUE) cat(paste(args[1], args[2]))
To submit the jobs, run multijobs.sh:
./multijobs.sh
You will get output like:
[<netid>@quser13 ~]$ ./testmultijobs.sh JobID = 19903143 for parameters AA 1 submitted on Wed May 10 16:01:56 CDT 2017 JobID = 19903144 for parameters BBB 2 submitted on Wed May 10 16:02:02 CDT 2017 JobID = 19903145 for parameters CCCC 3 submitted on Wed May 10 16:02:02 CDT 2017
The line
echo "JobID = ${JOB} for parameters ${P1} ${P2} submitted on `date`"
from the submission script adds information to the job output file so that you can tell which job corresponds with which parameters.
Examples of Application Commands
Below are examples of importing modules and calling various software programs in a submission script. These lines come below the MSUB options, with at least a single empty line between the end of the scheduler options and the start of commands specifying the work of the job.
There may be other ways to run these programs as well. These commands are only provided as a convenient reference.
Python
module load python/anaconda
python myscript.py
There are multiple versions of Python. module load python/anaconda will load Python 2.7. For other versions of Python, use the appropriate Anaconda Python module.
R
module load R/3.3.1
Rscript mycode.R
See Using R on Quest for more information and examples.
MATLAB
module load matlab
matlab -nosplash -nodesktop -singleCompThread -r 'commands;exit'
module load matlab
matlab -nosplash -nodesktop -singleCompThread -r myscript
Where myscript is a .m file. Note that there are multiple versions of MATLAB. You should specify the version you want, as the default may change over time.
Stata
module load stata/14
stata-mp < myprog.do > myprog.log
module load stata/14
stata-mp -b do myprog
Stata will read its commands from myprog.do and write its output to myprog.log in the working directory.
Note that Stata-MP is licensed for 4 cores, so please request exactly 4 cores per job in your job submission script. If you request more, you'll be wasting resources; if you request less and do not set the corresponding options appropriately in your script, Stata may use more cores than you've been assigned, resulting in your job being terminated. If you don't need Stata-MP's parallelization capabilities, you can run Stata SE with stata instead.
SAS
module load sas
sas myprog.sas
SAS can write very large temporary files. You can redirect such files to your project space with the -work option when you start-up SAS.
Mathematica
module load mathematica
math -script input.txt > output.txt
Examples of Slurm Jobs On Quest
See Research Computing's GitHub repository of example jobs for language/software specific examples and scripts you can copy and modify.
Interactive Jobs
Example: Interactive Job with More than One Node
If you submit an interactive job requiring more than one node, you will land on the Head Node for your job once the scheduler finds the resources. For more information see Submitting an Interactive Job.
[abc123@quser22 ~]$ srun -A p12345 -p normal -N 2 --tasks-per-node=10 --mem-per-cpu=1G --time=01:00:00 --pty bash -l ---------------------------------------- srun job start: Thu Feb 28 16:31:27 CST 2019 Job ID: 557781 Username: abc123 Queue: normal Account: p12345 ---------------------------------------- The following variables are not guaranteed to be the same in prologue and the job run script ---------------------------------------- PATH (in prologue) : /hpc/slurm/usertools:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/home/abc123/bin WORKDIR is: /home/abc123 ---------------------------------------- [abc123@qnode4217 ~]$
To find the name of the other compute node(s) that you have been assigned as part of your job, you can use the squeue command in a from the same terminal session. Look under NODELIST in the squeue output:
[abc123@qnode4217 ~]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 557781 normal bash abc123 R 5:51 2 qnode[4217-4218]
You can then ssh to the other node(s) if needed as long as your job is active.
ssh qnode4218
Batch Jobs
Job Array
Job arrays can be used to submit multiple jobs at once that use the same application script. This can be useful if you want to run the same script multiple times with different input parameters.
In the example below, the --array option defines the job array, with a specification of the index numbers you want to use (in this case, 1 through 50 with 10 elements running simultaneously anytime). The SLURM_ARRAY_TASK_ID variable takes on the values of the job array index for each job (so here, integer values 1 through 50, one value for each job). In this example, those values are then passed on to the R script as command line arguments.
jobsubmission.sh
#!/bin/bash
#SBATCH -A <allocationID>
#SBATCH -p <partitionName>
#SBATCH -n 1
#SBATCH -t <hh:mm:ss>
#SBATCH --array=1-50%10
#SBATCH --error=arrayJob_%A_%a.err
#SBATCH --output=arrayJob_%A_%a.out
#SBATCH --export=NONE
#SBATCH -J <jobName>
module load R
Rscript myscript.R ${SLURM_ARRAY_TASK_ID}
In this example standard output and error files are printed separately for each element of the job array with the --output and --error options. To avoid each element overwriting these files, tag them with jobID (%A) and elementID (%a) variables (which are automatically assigned by the scheduler) so elements have their own distinct output and error files.
Submit this script with:
sbatch jobsubmission.sh
50 jobs will then be submitted to the scheduler.
To supply more than just the job array index as a command line argument to your script, consider saving the other parameters in a text or csv file and having your script use the value of SLURM_ARRAY_TASK_ID to read the correct line from that file and retrieve the appropriate parameter values.
Software Applications
Below are examples of importing modules and calling various software programs in a submission script. These lines come below the SBATCH directives, with at least a single empty line between the end of the scheduler options and the start of commands specifying the work of the job.
There may be other ways to run these programs as well. These commands are only provided as a convenient reference.
Python
module load python/anaconda python myscript.py
There are multiple versions of Python available on Quest. module load python/anaconda will load Python 2.7. For other versions of Python, use the appropriate Anaconda Python module.
R
module load R/3.3.1 Rscript mycode.R
See Using R on Quest for more information and examples.
MATLAB
module load matlab matlab -nosplash -nodesktop -singleCompThread -r 'commands;exit'
or
module load matlab matlab -nosplash -nodesktop -singleCompThread -r myscript
Where myscript is a .m file. Note that there are multiple versions of MATLAB. You should specify the version you want, as the default may change over time.
Stata
module load stata/14 stata-mp < myprog.do > myprog.log
or
module load stata/14 stata-mp -b do myprog
Stata will read its commands from myprog.do and write its output to myprog.log in the working directory.
Note that Stata-MP is licensed for 4 cores, so please request exactly 4 cores per job in your job submission script. If you request more, you'll be wasting resources; if you request less and do not set the corresponding options appropriately in your script, Stata may use more cores than you've been assigned, resulting in your job being terminated. If you don't need Stata-MP's parallelization capabilities, you can run Stata SE with stata instead.
SAS
module load sas sas myprog.sas
SAS can write very large temporary files. You can redirect such files to your project space with the -work option when you start-up SAS.
Mathematica
module load mathematica math -script input.txt > output.txt