Examples of Jobs on Quest

Additional details and examples of Quest job submission scripts and commands.

Interactive Jobs

Batch Jobs

Job Array

Software Applications

See Research Computing's GitHub repository of example jobs for language/software specific examples and scripts you can copy and modify.

Interactive Jobs

Example: Interactive Job with More than One Node

If you submit an interactive job requiring more than one node, you will land on the Head Node for your job once the scheduler finds the resources. For more information see Submitting an Interactive Job.

[abc123@quser22 ~]$ srun -A p12345 -p normal -N 2 --tasks-per-node=10 --mem-per-cpu=1G --time=01:00:00 --pty bash -l
----------------------------------------
srun job start: Thu Feb 28 16:31:27 CST 2019
Job ID: 557781
Username: abc123
Queue: normal
Account: p12345
----------------------------------------
The following variables are not
guaranteed to be the same in
prologue and the job run script
----------------------------------------
PATH (in prologue) : /hpc/slurm/usertools:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/home/abc123/bin
WORKDIR is: /home/abc123
----------------------------------------
[abc123@qnode4217 ~]$

This interactive job requests 2 nodes and 10 cores in each of those nodes. In total you will have access to 20 cores. Since the job reserved 1 GB per core, 20 GB of RAM is allocated in total (i.e. 10 GB in each node).

To find the name of the other compute node(s) that you have been assigned as part of your job, you can use the squeue command in a from the same terminal session. Look under NODELIST in the squeue output:

[abc123@qnode4217 ~]$ squeue
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
557781    normal     bash  abc123   R       5:51      2 qnode[4217-4218]

You can then ssh to the other node(s) if needed as long as your job is active.

ssh qnode4218

Batch Jobs

Job Array

Job arrays can be used to submit multiple jobs at once that use the same application script. This can be useful if you want to run the same script multiple times with different input parameters.

In the example below, the --array option defines the job array, with a specification of the index numbers you want to use (in this case, 1 through 50 with 10 elements running simultaneously anytime). The $SLURM_ARRAY_TASK_ID variable takes on the values of the job array index for each job (so here, integer values 1 through 50, one value for each job). In this example, the value of $SLURM_ARRAY_TASK_ID is used to print out a line from a CSV input file, whose fields are then passed on to a script as command line arguments.

jobsubmission.sh
#!/bin/bash
#SBATCH -A <allocationID>
#SBATCH -p <partitionName>
#SBATCH -n 1
#SBATCH -t <hh:mm:ss>
#SBATCH --mem=<total memory>G # Reserving a total amount of memory for each job within the array #SBATCH --array=1-50%10 # make sure the array number matches the number of lines in your input file! #SBATCH --error=arrayJob_%A_%a.err #SBATCH --output=arrayJob_%A_%a.out #SBATCH -J <jobName> module purge module load anaconda3/2018.12 python ./myscript.py $(sed -n ${SLURM_ARRAY_TASK_ID}p input.csv | sed 's/,/ /g')


myscript.py
import sys

first_argument = sys.argv[1]
second_argument = sys.argv[2]

print("The first argument is ", first_argument)
print("The second argument is ", second_argument)

In this example, myscript.py will receive the values in input.csv as arguments: the first field will be sys.argv[1], the second field will be sys.argv[2], etc.

Note: make sure the number you specify for the --array parameter matches the number of lines in your input file!

Also, note that in this example standard output and error files are printed separately for each element of the job array with the --output and --error options. To avoid each element overwriting these files, tag them with jobID (%A) and elementID (%a) variables (which are automatically assigned by the scheduler) so elements have their own distinct output and error files.

Submit this script with:

sbatch jobsubmission.sh

The job array will then be submitted to the scheduler.

Software Applications

Below are examples of importing modules and calling various software programs in a submission script. These lines come below the SBATCH directives, with at least a single empty line between the end of the scheduler options and the start of commands specifying the work of the job.

There may be other ways to run these programs as well. These commands are only provided as a convenient reference.

Python

module load python/anaconda
python myscript.py

There are multiple versions of Python available on Quest. module load python/anaconda will load Python 2.7. For other versions of Python, use the appropriate Anaconda Python module.

R

module load R/3.3.1
Rscript mycode.R

See Using R on Quest for more information and examples.

MATLAB

module load matlab
matlab -nosplash -nodesktop -singleCompThread -r 'commands;exit'

or

module load matlab
matlab -nosplash -nodesktop -singleCompThread -r myscript

Where myscript is a .m file. Note that there are multiple versions of MATLAB. You should specify the version you want, as the default may change over time.

Stata

module load stata/14
stata-mp < myprog.do > myprog.log

or

module load stata/14
stata-mp -b do myprog

Stata will read its commands from myprog.do and write its output to myprog.log in the working directory.

Note that Stata-MP is licensed for 4 cores, so please request exactly 4 cores per job in your job submission script. If you request more, you'll be wasting resources; if you request less and do not set the corresponding options appropriately in your script, Stata may use more cores than you've been assigned, resulting in your job being terminated. If you don't need Stata-MP's parallelization capabilities, you can run Stata SE with stata instead.

SAS

module load sas
sas myprog.sas

SAS can write very large temporary files. You can redirect such files to your project space with the -work option when you start-up SAS.

Mathematica

module load mathematica
math -script input.txt > output.txt

See Also:




Keywords:Quest, job, jobs, example, interactive, batch, submit, job array, job arrays, multiple jobs   Doc ID:70719
Owner:Research Computing .Group:Northwestern
Created:2017-02-15 15:41 CSTUpdated:2019-07-10 16:09 CST
Sites:Northwestern
Feedback:  0   1