Examples of Jobs on Quest

On May 1st, 2019 the Quest scheduler will change from Moab to Slurm. Researchers using Quest will have to update their job submission scripts and use different commands to run under the new scheduler.

Click here for the schedule of Slurm workshops offered during the transition period. For more information, please see the project page.

To view the Slurm version of this documentation, click here.

Additional details and examples of Quest job submission scripts and commands.

This page includes additional details and examples of job submission scripts and commands.

See Research Computing's GitHub repository of example jobs for language/software specific examples and scripts you can copy and modify. 

Interactive Jobs

Example: Interactive Job with More than One Node

If you submit an interactive job requiring more than one node, you will land on the Master Node for your job once the scheduler finds the resources. For more information see Submitting an Interactive Job.

[<netid>@quser11 ~]$ msub -I -l nodes=2:ppn=20 -l walltime=1:00:00 -q short
qsub: waiting for job 19901348.qsched03.quest.it.northwestern.edu to start
qsub: job 19901348.qsched03.quest.it.northwestern.edu ready

----------------------------------------
PBS: Begin PBS Prologue Wed May 10 13:16:50 CDT 2017 1494440210
PBS: Job ID:		19901348.qsched03.quest.it.northwestern.edu
PBS: Username:		<netid>
PBS: Group:		<netid>
PBS: Executing queue:     short
PBS: Job name:		STDIN
PBS: Account:		<allocationID>
----------------------------------------
   The following variables are not
   guaranteed to be the same in 
   prologue and the job run script  
----------------------------------------
PBS: Temporary Dir($TMPDIR):	/tmp/19901348.qsched03.quest.it.northwestern.edu
PBS: Master Node($PBS_MSHOST):		qnode5056
PBS: node file($PBS_NODEFILE):  /hpc/opt/torque6/nodes/qnode5056/aux//19901348.qsched03.quest.it.northwestern.edu
PBS: PATH (in prologue) : /bin:/usr/bin
PBS: WORKDIR ($PBS_O_WORKDIR) is:  /home/<netid>
----------------------------------------
PBS: End PBS Prologue Wed May 10 13:16:50 CDT 2017 1494440210
[<netid>@qnode5056 ~]$ 

To find the name of the other compute node(s) that you have been assigned as part of your job, you can use the checkjob command in a separate terminal session connected to a Quest login node. checkjob does not work from the compute nodes. Look under Allocated Nodes:.

[<netid>@quser10 ~]$ checkjob 19901348
job 19901348

AName: STDIN
State: Running 
Creds:  user:<netid>  group:<netid>  account:<allocationID>  class:short  qos:<allocationID>qos
WallTime:   00:02:34 of 1:00:00
SubmitTime: Wed May 10 13:15:26
  (Time Queued  Total: 00:01:24  Eligible: 00:01:09)

StartTime: Wed May 10 13:16:50
TemplateSets:  DEFAULT
NodeMatchPolicy: EXACTNODE
Total Requested Tasks: 40

Req[0]  TaskCount: 40  Partition: quest5

Allocated Nodes:
[qnode5056:20][qnode5057:20]


SystemID:   Moab
SystemJID:  Moab.1655563
Notification Events: JobFail

StartCount:     1
BypassCount:    30
Partition List: quest3,quest4,quest5,quest6
Flags:          BACKFILL,ADVRES:admin1,SUSPENDABLE,INTERACTIVE
Attr:           BACKFILL,INTERACTIVE,checkpoint
Variables:      UsageRecord=9889320
StartPriority:  258
IterationJobRank: 0
Reservation '19901348' (-00:02:57 -> 00:57:03  Duration: 1:00:00)

You can then ssh to the other node(s) if needed as long as your job is active.

ssh qnode5057

Batch Jobs

Example: Job Array

Job arrays can be used to submit multiple jobs at once that use the same application script. This can be useful if you want to run the same script multiple times with different input parameters.

NOTE: While job arrays are convenient for submitting jobs, monitoring jobs submitted to Quest as part of job array can be difficult, as the normal monitoring commands will only give you information on the array of jobs, not the individual jobs. This makes it difficult for you, or Research Computing staff to help you, if something goes wrong with your job. A better approach may be to use a loop with a bash script to submit multiple jobs (see Example: Submitting Multiple Jobs below).

In the example below, the -t option defines the job array, with a name you pick and a specification of the index numbers you want to use (in this case, 1 through 25). With the option #PBS -V, the MOAB_JOBARRAYINDEX variable takes on the values of the job array index for each job (so in this case, integer values 1 through 25, one value for each job). In this example, those values are then passed on to the R script as command line arguments.

jobsubmission.sh

#!/bin/bash
#MSUB -A <allocationID>
#MSUB -q short
#MSUB -l walltime=04:00:00
#MSUB -M <email>
#MSUB -j oe
#MSUB -N <jobname>
#MSUB -l procs=1
#MSUB -t <arrayname>[1-25]
#PBS -V MOAB_JOBARRAYINDEX

module load R

# Set your working directory
cd $PBS_O_WORKDIR

Rscript myscript.R ${MOAB_JOBARRAYINDEX}

The script above would be submitted with:

msub jobsubmission.sh

And 25 jobs would be submitted to the scheduler.

To supply more than just the job array index as a command line argument to your script, consider saving the other parameters in a text or csv file and having your script use the value of MOAB_JOBARRAYINDEX to read the correct line from that file and retrieve the appropriate parameter values.

Example: Submitting Multiple Jobs

If you want to submit multiple jobs using the same submission script, but sending different parameters to your code, you can write a bash script to do this.  Get the script below from the example jobs repository.

multijobs.sh
#!/bin/bash     
while read P1 P2
do
    JOB=`msub - << EOJ        
            
#MSUB -A <allocationID>
#MSUB -q short 
#MSUB -l walltime=00:05:00
#MSUB -M <email>  
#MSUB -j oe        
#MSUB -N <jobname>  
#MSUB -l nodes=1:ppn=1 
                                                                                                                                   
module load R/3.3.1      
Rscript testR.R ${P1} ${P2}   
                                                                                                                                   
EOJ  
`

echo "JobID = ${JOB} for parameters ${P1} ${P2} submitted on `date`"
done < params.txt
exit

Note the two highlighted tick marks are where they should be; they surround the submission script. Make multijobs.sh executable with:

chmod u+x multijobs.sh

The script reads parameters from params.txt, which is a space delimited file:

params.txt
AA 1
BBB 2
CCCC 3

One job is created for each line in params.txt. The values are sent to testR.R:

testR.R
args <- commandArgs(trailingOnly = TRUE)
cat(paste(args[1], args[2]))

To submit the jobs, run multijobs.sh:

./multijobs.sh

You will get output like:

[<netid>@quser13 ~]$ ./testmultijobs.sh
JobID = 
19903143 for parameters AA 1 submitted on Wed May 10 16:01:56 CDT 2017
JobID = 
19903144 for parameters BBB 2 submitted on Wed May 10 16:02:02 CDT 2017
JobID = 
19903145 for parameters CCCC 3 submitted on Wed May 10 16:02:02 CDT 2017

The line

echo "JobID = ${JOB} for parameters ${P1} ${P2} submitted on `date`"

from the submission script adds information to the job output file so that you can tell which job corresponds with which parameters.

Examples of Application Commands

Below are examples of importing modules and calling various software programs in a submission script. These lines come below the MSUB options, with at least a single empty line between the end of the scheduler options and the start of commands specifying the work of the job.

There may be other ways to run these programs as well. These commands are only provided as a convenient reference.

Python

module load python/anaconda
python myscript.py

There are multiple versions of Python. module load python/anaconda will load Python 2.7. For other versions of Python, use the appropriate Anaconda Python module.

R

module load R/3.3.1
Rscript mycode.R

See Using R on Quest for more information and examples.

MATLAB

module load matlab
matlab -nosplash -nodesktop -singleCompThread -r 'commands;exit'

or

module load matlab
matlab -nosplash -nodesktop -singleCompThread -r myscript

Where myscript is a .m file. Note that there are multiple versions of MATLAB. You should specify the version you want, as the default may change over time.

Stata

module load stata/14
stata-mp < myprog.do > myprog.log

or

module load stata/14
stata-mp -b do myprog

Stata will read its commands from myprog.do and write its output to myprog.log in the working directory.

Note that Stata-MP is licensed for 4 cores, so please request exactly 4 cores per job in your job submission script. If you request more, you'll be wasting resources; if you request less and do not set the corresponding options appropriately in your script, Stata may use more cores than you've been assigned, resulting in your job being terminated. If you don't need Stata-MP's parallelization capabilities, you can run Stata SE with stata instead.

SAS

module load sas
sas myprog.sas

SAS can write very large temporary files. You can redirect such files to your project space with the -work option when you start-up SAS.

Mathematica

module load mathematica
math -script input.txt > output.txt

Examples of Slurm Jobs On Quest

Interactive Jobs

Batch Jobs

Job Array

Software Applications

See Research Computing's GitHub repository of example jobs for language/software specific examples and scripts you can copy and modify.

Interactive Jobs

Example: Interactive Job with More than One Node

If you submit an interactive job requiring more than one node, you will land on the Head Node for your job once the scheduler finds the resources. For more information see Submitting an Interactive Job.

[abc123@quser22 ~]$ srun -A p12345 -p normal -N 2 --tasks-per-node=10 --mem-per-cpu=1G --time=01:00:00 --pty bash -l
----------------------------------------
srun job start: Thu Feb 28 16:31:27 CST 2019
Job ID: 557781
Username: abc123
Queue: normal
Account: p12345
----------------------------------------
The following variables are not
guaranteed to be the same in
prologue and the job run script
----------------------------------------
PATH (in prologue) : /hpc/slurm/usertools:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/home/abc123/bin
WORKDIR is: /home/abc123
----------------------------------------
[abc123@qnode4217 ~]$

To find the name of the other compute node(s) that you have been assigned as part of your job, you can use the squeue command in a from the same terminal session. Look under NODELIST in the squeue output:

[abc123@qnode4217 ~]$ squeue
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
557781    normal     bash  abc123   R       5:51      2 qnode[4217-4218]

You can then ssh to the other node(s) if needed as long as your job is active.

ssh qnode4218

Batch Jobs

Job Array

Job arrays can be used to submit multiple jobs at once that use the same application script. This can be useful if you want to run the same script multiple times with different input parameters.

In the example below, the --array option defines the job array, with a specification of the index numbers you want to use (in this case, 1 through 50 with 10 elements running simultaneously anytime). The SLURM_ARRAY_TASK_ID variable takes on the values of the job array index for each job (so here, integer values 1 through 50, one value for each job). In this example, those values are then passed on to the R script as command line arguments.

jobsubmission.sh
#!/bin/bash
#SBATCH -A <allocationID>
#SBATCH -p <partitionName>
#SBATCH -n 1
#SBATCH -t <hh:mm:ss>
#SBATCH --array=1-50%10
#SBATCH --error=arrayJob_%A_%a.err
#SBATCH --output=arrayJob_%A_%a.out
#SBATCH --export=NONE
#SBATCH -J <jobName>

module load R

Rscript myscript.R ${SLURM_ARRAY_TASK_ID}

In this example standard output and error files are printed separately for each element of the job array with the --output and --error options. To avoid each element overwriting these files, tag them with jobID (%A) and elementID (%a) variables (which are automatically assigned by the scheduler) so elements have their own distinct output and error files.

Submit this script with:

sbatch jobsubmission.sh

50 jobs will then be submitted to the scheduler.

To supply more than just the job array index as a command line argument to your script, consider saving the other parameters in a text or csv file and having your script use the value of SLURM_ARRAY_TASK_ID to read the correct line from that file and retrieve the appropriate parameter values.

Software Applications

Below are examples of importing modules and calling various software programs in a submission script. These lines come below the SBATCH directives, with at least a single empty line between the end of the scheduler options and the start of commands specifying the work of the job.

There may be other ways to run these programs as well. These commands are only provided as a convenient reference.

Python

module load python/anaconda
python myscript.py

There are multiple versions of Python available on Quest. module load python/anaconda will load Python 2.7. For other versions of Python, use the appropriate Anaconda Python module.

R

module load R/3.3.1
Rscript mycode.R

See Using R on Quest for more information and examples.

MATLAB

module load matlab
matlab -nosplash -nodesktop -singleCompThread -r 'commands;exit'

or

module load matlab
matlab -nosplash -nodesktop -singleCompThread -r myscript

Where myscript is a .m file. Note that there are multiple versions of MATLAB. You should specify the version you want, as the default may change over time.

Stata

module load stata/14
stata-mp < myprog.do > myprog.log

or

module load stata/14
stata-mp -b do myprog

Stata will read its commands from myprog.do and write its output to myprog.log in the working directory.

Note that Stata-MP is licensed for 4 cores, so please request exactly 4 cores per job in your job submission script. If you request more, you'll be wasting resources; if you request less and do not set the corresponding options appropriately in your script, Stata may use more cores than you've been assigned, resulting in your job being terminated. If you don't need Stata-MP's parallelization capabilities, you can run Stata SE with stata instead.

SAS

module load sas
sas myprog.sas

SAS can write very large temporary files. You can redirect such files to your project space with the -work option when you start-up SAS.

Mathematica

module load mathematica
math -script input.txt > output.txt

See Also:




Keywords:Quest, job, jobs, example, interactive, batch, submit, job array, job arrays, multiple jobs   Doc ID:70719
Owner:Research Computing .Group:Northwestern
Created:2017-02-15 16:41 CDTUpdated:2019-04-01 16:31 CDT
Sites:Northwestern
Feedback:  0   1