Examples of Jobs on Quest

Additional details and examples of Quest job submission scripts and commands.

This page includes additional details and examples of job submission scripts and commands.

See Research Computing's GitHub repository of example jobs for language/software specific examples and scripts you can copy and modify. 

Interactive Jobs

Example: Interactive Job with More than One Node

If you submit an interactive job requiring more than one node, you will land on the Master Node for your job once the scheduler finds the resources. For more information see Submitting an Interactive Job.

[<netid>@quser11 ~]$ msub -I -l nodes=2:ppn=20 -l walltime=1:00:00 -q short
qsub: waiting for job 19901348.qsched03.quest.it.northwestern.edu to start
qsub: job 19901348.qsched03.quest.it.northwestern.edu ready

----------------------------------------
PBS: Begin PBS Prologue Wed May 10 13:16:50 CDT 2017 1494440210
PBS: Job ID:		19901348.qsched03.quest.it.northwestern.edu
PBS: Username:		<netid>
PBS: Group:		<netid>
PBS: Executing queue:     short
PBS: Job name:		STDIN
PBS: Account:		<allocationID>
----------------------------------------
   The following variables are not
   guaranteed to be the same in 
   prologue and the job run script  
----------------------------------------
PBS: Temporary Dir($TMPDIR):	/tmp/19901348.qsched03.quest.it.northwestern.edu
PBS: Master Node($PBS_MSHOST):		qnode5056
PBS: node file($PBS_NODEFILE):  /hpc/opt/torque6/nodes/qnode5056/aux//19901348.qsched03.quest.it.northwestern.edu
PBS: PATH (in prologue) : /bin:/usr/bin
PBS: WORKDIR ($PBS_O_WORKDIR) is:  /home/<netid>
----------------------------------------
PBS: End PBS Prologue Wed May 10 13:16:50 CDT 2017 1494440210
[<netid>@qnode5056 ~]$ 

To find the name of the other compute node(s) that you have been assigned as part of your job, you can use the checkjob command in a separate terminal session connected to a Quest login node. checkjob does not work from the compute nodes. Look under Allocated Nodes:.

[<netid>@quser10 ~]$ checkjob 19901348
job 19901348

AName: STDIN
State: Running 
Creds:  user:<netid>  group:<netid>  account:<allocationID>  class:short  qos:<allocationID>qos
WallTime:   00:02:34 of 1:00:00
SubmitTime: Wed May 10 13:15:26
  (Time Queued  Total: 00:01:24  Eligible: 00:01:09)

StartTime: Wed May 10 13:16:50
TemplateSets:  DEFAULT
NodeMatchPolicy: EXACTNODE
Total Requested Tasks: 40

Req[0]  TaskCount: 40  Partition: quest5

Allocated Nodes:
[qnode5056:20][qnode5057:20]


SystemID:   Moab
SystemJID:  Moab.1655563
Notification Events: JobFail

StartCount:     1
BypassCount:    30
Partition List: quest3,quest4,quest5,quest6
Flags:          BACKFILL,ADVRES:admin1,SUSPENDABLE,INTERACTIVE
Attr:           BACKFILL,INTERACTIVE,checkpoint
Variables:      UsageRecord=9889320
StartPriority:  258
IterationJobRank: 0
Reservation '19901348' (-00:02:57 -> 00:57:03  Duration: 1:00:00)

You can then ssh to the other node(s) if needed as long as your job is active.

ssh qnode5057

Batch Jobs

Example: Job Array

Job arrays can be used to submit multiple jobs at once that use the same application script. This can be useful if you want to run the same script multiple times with different input parameters.

NOTE: While job arrays are convenient for submitting jobs, monitoring jobs submitted to Quest as part of job array can be difficult, as the normal monitoring commands will only give you information on the array of jobs, not the individual jobs. This makes it difficult for you, or Research Computing staff to help you, if something goes wrong with your job. A better approach may be to use a loop with a bash script to submit multiple jobs (see Example: Submitting Multiple Jobs below).

In the example below, the -t option defines the job array, with a name you pick and a specification of the index numbers you want to use (in this case, 1 through 25). With the option #PBS -V, the MOAB_JOBARRAYINDEX variable takes on the values of the job array index for each job (so in this case, integer values 1 through 25, one value for each job). In this example, those values are then passed on to the R script as command line arguments.

jobsubmission.sh

#!/bin/bash
#MSUB -A <allocationID>
#MSUB -q short
#MSUB -l walltime=04:00:00
#MSUB -M <email>
#MSUB -j oe
#MSUB -N <jobname>
#MSUB -l procs=1
#MSUB -t <arrayname>[1-25]
#PBS -V MOAB_JOBARRAYINDEX

module load R

# Set your working directory
cd $PBS_O_WORKDIR

Rscript myscript.R ${MOAB_JOBARRAYINDEX}

The script above would be submitted with:

msub jobsubmission.sh

And 25 jobs would be submitted to the scheduler.

To supply more than just the job array index as a command line argument to your script, consider saving the other parameters in a text or csv file and having your script use the value of MOAB_JOBARRAYINDEX to read the correct line from that file and retrieve the appropriate parameter values.

Example: Submitting Multiple Jobs

If you want to submit multiple jobs using the same submission script, but sending different parameters to your code, you can write a bash script to do this.  Get the script below from the example jobs repository.

multijobs.sh
#!/bin/bash     
while read P1 P2
do
    JOB=`msub - << EOJ        
            
#MSUB -A <allocationID>
#MSUB -q short 
#MSUB -l walltime=00:05:00
#MSUB -M <email>  
#MSUB -j oe        
#MSUB -N <jobname>  
#MSUB -l nodes=1:ppn=1 
                                                                                                                                   
module load R/3.3.1      
Rscript testR.R ${P1} ${P2}   
                                                                                                                                   
EOJ  
`

echo "JobID = ${JOB} for parameters ${P1} ${P2} submitted on `date`"
done < params.txt
exit

Note the two highlighted tick marks are where they should be; they surround the submission script. Make multijobs.sh executable with:

chmod u+x multijobs.sh

The script reads parameters from params.txt, which is a space delimited file:

params.txt
AA 1
BBB 2
CCCC 3

One job is created for each line in params.txt. The values are sent to testR.R:

testR.R
args <- commandArgs(trailingOnly = TRUE)
cat(paste(args[1], args[2]))

To submit the jobs, run multijobs.sh:

./multijobs.sh

You will get output like:

[<netid>@quser13 ~]$ ./testmultijobs.sh
JobID = 
19903143 for parameters AA 1 submitted on Wed May 10 16:01:56 CDT 2017
JobID = 
19903144 for parameters BBB 2 submitted on Wed May 10 16:02:02 CDT 2017
JobID = 
19903145 for parameters CCCC 3 submitted on Wed May 10 16:02:02 CDT 2017

The line

echo "JobID = ${JOB} for parameters ${P1} ${P2} submitted on `date`"

from the submission script adds information to the job output file so that you can tell which job corresponds with which parameters.

Examples of Application Commands

Below are examples of importing modules and calling various software programs in a submission script. These lines come below the MSUB options, with at least a single empty line between the end of the scheduler options and the start of commands specifying the work of the job.

There may be other ways to run these programs as well. These commands are only provided as a convenient reference.

Python

module load python/anaconda
python myscript.py

There are multiple versions of Python. module load python/anaconda will load Python 2.7. For other versions of Python, use the appropriate Anaconda Python module.

R

module load R/3.3.1
Rscript mycode.R

See Using R on Quest for more information and examples.

MATLAB

module load matlab
matlab -nosplash -nodesktop -singleCompThread -r 'commands;exit'

or

module load matlab
matlab -nosplash -nodesktop -singleCompThread -r myscript

Where myscript is a .m file. Note that there are multiple versions of MATLAB. You should specify the version you want, as the default may change over time.

Stata

module load stata/14
stata-mp < myprog.do > myprog.log

or

module load stata/14
stata-mp -b do myprog

Stata will read its commands from myprog.do and write its output to myprog.log in the working directory.

Note that Stata-MP is licensed for 4 cores, so please request exactly 4 cores per job in your job submission script. If you request more, you'll be wasting resources; if you request less and do not set the corresponding options appropriately in your script, Stata may use more cores than you've been assigned, resulting in your job being terminated. If you don't need Stata-MP's parallelization capabilities, you can run Stata SE with stata instead.

SAS

module load sas
sas myprog.sas

SAS can write very large temporary files. You can redirect such files to your project space with the -work option when you start-up SAS.

Mathematica

module load mathematica
math -script input.txt > output.txt

See Also:




Keywords:Quest, job, jobs, example, interactive, batch, submit, job array, job arrays, multiple jobs   Doc ID:70719
Owner:Research Computing .Group:Northwestern
Created:2017-02-15 15:41 CSTUpdated:2018-12-26 10:06 CST
Sites:Northwestern
Feedback:  0   1