Checking Processor and Memory Utilization for Jobs on Quest
This page provides commands for checking processor (core) and RAM memory utilization for jobs that run on Quest.
How many processors did my job run on?
While a job is running, or within seven days of when a job has completed, you can find out how many processors a job utilized by running this command on a Quest login node:
checkjob -v -v | grep Procs:
Example output:
Average Utilized Procs: 97.94
"Average Utilized Procs" shows the number of processors (cores) actually used by a job, and should be close to the number of processors reserved in the job submission script. In this example, the job used approximately 98 processors on average during it's run.
How many nodes and processors did the job reserve in the submission script? Nodes and processors are reserved with #MSUB -l nodes=X:ppn=Y, where X is the number of nodes reserved and Y is the number of processors reserved per node. Note that if a job doesn’t utilize all of the processors requested by the job submission script, those nodes and processors sit idle while the job is running. Unless you have requested additional cores to increase the RAM available to your job, requesting excess cores should be avoided because it prevents other users from running jobs on those processors, and it causes your allocation to be debited for compute hours on processors the job reserved but didn’t use.
To check how many processors a job submission script reserved, either check the job's submission script or use the command:
checkjob -v -v | grep Requested
Example output:
Total Requested Tasks: 112
Total Requested Nodes: 4
In this example, the job submission script requested 112 total processors spread across 4 nodes, reserving 28 processors on each of the 4 nodes. In this case, the job submission script reserved these nodes and processors with the line #MSUB -l nodes=4:ppn=28.
How much RAM did my job actually use?
Check with the command:
checkjob -v -v | grep Memory:
Example output:
Average Utilized Memory: 719.88 MB
"Average Utilized Memory" shows the amount of RAM each processor utilized on average. The appropriate value for this number depends on the Quest partition the job runs on. Unless instructed otherwise, the values should be less than:
- 6.4GB on quest4
- 5GB on quest5
- 4.5GB on quest6
To check which Quest partition your job ran on, use the command:
checkjob -v -v | grep Execution
Example output:
Execution Partition: quest5
In this example, the job can utilize up to 5GB of memory per processor, because this job is running on the quest5 partition.
If the job utilizes more than the recommended amount of RAM for that partition, increase the number of processors requested in the job's submission script to increase the amount of RAM available for your job. The Moab option -l mem=<n>gb is not guaranteed to reserve the requested amount of memory.
See Also:
- Managing Jobs on Quest
- Examples of Jobs on Quest
- Troubleshooting Jobs on Quest
- Submitting a Job on Quest