Resources
30 Minutes
Objectives
- Understand how to look up job statistics and profile code.
- Understand job size implications.
- Understand problems and limitations involved in using multiple CPUs.
Prerequisite
What resources?¶
Last time we submitted a job, we did not specify a number of CPUs, and therefore
we were provided the default of 2 (1 core).
As a reminder, our Slurm script example_job.sl currently looks like this.
#!/bin/bash -e
#SBATCH --job-name example_job
#SBATCH --account nesi99991
#SBATCH --time 00:15:00
#SBATCH --mem 300M
module purge
module load R/4.6.0-foss-2026
Rscript sum_matrix.r
echo "Done!"
We will now submit the same job again with more CPUs.
We ask for more CPUs by adding #SBATCH --cpus-per-task 4 to our script.
Your script should now look like this:
#!/bin/bash -e
#SBATCH --job-name example_job
#SBATCH --account nesi99991
#SBATCH --time 00:15:00
#SBATCH --mem 300M
#SBATCH --cpus-per-task 4
module purge
module load R/4.6.0-foss-2026
Rscript sum_matrix.r
echo "Done!"
And then submit using sbatch as we did before.
sbatch example_job.sl
Submitted batch job 23137702
Watch
We can prepend any command with watch in order to periodically (default 2 seconds) run a command. e.g. watch squeue --me will give us up to date information on our running jobs.
Care should be used when using watch as repeatedly running a command can have adverse effects.
Exit watch with Ctrl+C.
In squeue, the number under CPUS should be 4 while the job is pending.
Once it is running, it shows 8, because Slurm counts both logical CPUs on each of your 4 cores (see Logical CPUs).
Checking on our job with sacct.
Oh no!
JobID JobName Alloc Elapsed TotalCPU ReqMem MaxRSS State
--------------- ---------------- ----- ----------- ------------ ------- -------- ----------
23137702 example_job 8 00:00:36 00:52.908 300M OUT_OF_ME+
23137702.batch batch 8 00:00:36 00:52.908 267988K OUT_OF_ME+
23137702.extern extern 8 00:00:36 00:00:00 COMPLETED
Too little or too much¶
To understand why our job failed, we need to talk about the resources involved.
Understanding the resources you have available and how to use them most efficiently is a vital skill in high performance computing.
Below is a table of common resources and issues you may face if you do not request the correct amount.
| Not enough | Too Much | |
|---|---|---|
| CPU | The job will run more slowly than expected, and so may run out of time and get killed for exceeding its time limit. | The job will wait in the queue for longer. You will be charged for CPUs regardless of whether they are used or not. Your Fairshare score will fall more. |
| Memory | Your job will fail, probably with an 'OUT OF MEMORY' error, segmentation fault or bus error (may not happen immediately). | The job will wait in the queue for longer. You will be charged for memory regardless of whether it is used or not. Your Fairshare score will fall more. |
| Walltime | The job will run out of time and be terminated by the scheduler. | The job will wait in the queue for longer. |
Measuring a finished job¶
Since we have already run a job (successful or otherwise), this is the best source of info we currently have.
Let's look again at the sacct output for the job that ran out of memory:
sacct
JobID JobName Alloc Elapsed TotalCPU ReqMem MaxRSS State
--------------- ---------------- ----- ----------- ------------ ------- -------- ----------
23137702 example_job 8 00:00:36 00:52.908 300M OUT_OF_ME+
23137702.batch batch 8 00:00:36 00:52.908 267988K OUT_OF_ME+
23137702.extern extern 8 00:00:36 00:00:00 COMPLETED
Calculating efficiency¶
With this information, we may determine a couple of things.
Memory efficiency can be determined by comparing ReqMem (requested memory) with MaxRSS (maximum used memory), MaxRSS is given in KB, so a unit conversion is usually required.
So for the above example we see that 262 MB (267988K) of our requested 300 MB was used, meaning the memory efficiency was about 87%. The job still ran out of memory: by default Slurm only records memory use every 30 seconds, so MaxRSS can miss the peak.
CPU efficiency can be determined by comparing TotalCPU (CPU time), with the maximum possible CPU time. The maximum possible CPU time equal to Alloc (number of allocated CPUs) multiplied by Elapsed (Walltime, actual time passed).
For the above example 53 seconds of computation was done,
where the maximum possible computation time was 288 seconds (8 CPUs multiplied by 36 seconds), meaning the CPU efficiency was about 18%.
Alloc counts two CPUs per core
Each core on Mahuika has two logical CPUs, and Alloc counts both, although by default your job only uses one of them
(see Logical CPUs).
This is why a job that didn't ask for a number of CPUs shows an Alloc of 2.
So a job that keeps every CPU it asked for busy will only reach about 50% CPU efficiency by this formula.
seff, shown below, divides by cores instead, so its CPU efficiency is double the figure from this formula.
Time Efficiency is simply the Elapsed Time divided by Time Requested.
36 seconds used out of 15 minutes requested give a time efficiency of about 4%
Efficiency exercise¶
Efficiency Exercise
For the job shown below, calculate its CPU efficiency and memory efficiency.
JobID JobName Alloc Elapsed TotalCPU ReqMem MaxRSS State
--------------- ---------------- ----- ----------- ------------ ------- -------- ----------
37171050 Example-job 8 00:06:03 00:23:04 32G FAILED
37171050.batch batch 8 00:06:03 23:03.999 14082672k FAILED
37171050.extern extern 8 00:06:03 00:00.001 0 COMPLETED
What was the CPU efficiency?
What was the memory efficiency?
Using seff¶
For convenience, Mahuika has provided the command seff to calculate Slurm Efficiency.
Run it with a job ID, for example the job that ran out of memory:
seff 23137702
Job ID: 23137702
State: OUT_OF_MEMORY
Cores: 4
Job Wall-time: 4% 00:00:36 of 00:15:00 time limit
Avg CPU Utilisation: 37% 00:00:52 of 00:02:24 core-walltime
Peak Mem Utilisation: 87% 261.71 MB of 300.00 MB
Replace 23137702 with your job ID.
Knowing what we do now about job efficiency, let's submit the previous job again, this time requesting enough memory to actually cover it:
#!/bin/bash -e
#SBATCH --job-name example_job
#SBATCH --account nesi99991
#SBATCH --time 00:15:00
#SBATCH --mem 1200M
#SBATCH --cpus-per-task 4
module purge
module load R/4.6.0-foss-2026
Rscript sum_matrix.r
echo "Done!"
sbatch example_job.sl
Now that we've requested more memory, this one should complete successfully.
Once it has finished, check its MaxRSS with sacct or seff, and note it down.
You will use it to estimate how much memory to request in Scaling.
Logical CPUs (SMT)¶
Modern CPU cores have 2 threads of operation that can execute independently of one another. SMT is the technology that allows the 2 threads within one physical core to present as multiple logical cores, sometimes referred to as virtual CPUS (vCPUS).
For more details please see our documentation on SMT
Measuring a running job¶
On Mahuika, we allow users to connect directly to compute nodes from the login node. This is useful to check on a running job and see how it's doing, however, we only allow you to connect to nodes on which you have running jobs.
The most reliable way to check current system stats is with htop.
htop is an interactive process viewer that can be launched from command line.
Finding your job¶
Before we can check on our job, we need its job ID, which is in the JOBID column of squeue --me.
The job must be running, shown as RUNNING in the STATE column.
Job length
example_job.sl runs for less than a minute once it starts, so it has probably finished by now.
Submit it again with sbatch example_job.sl, and run the commands below as soon as it is running.
squeue --me
JOBID USER ACCOUNT NAME CPUS MIN_MEM PARTITI START_TIME TIME_LEFT STATE NODELIST(REASON)
26763045 username nesi99991 example_job 8 1200M milan Oct 20 16:13 14:56 RUNNING mc093
Now we can use svisit to run htop on the node where the job is running.
svisit 26763045 htop -u $USER
Replace 26763045 with your job ID.
If you cannot connect, it may be that the job has finished and you have lost permission to visit that node.
Reading htop¶
You may see something like this,
top - 21:00:19 up 3:07, 1 user, load average: 1.06, 1.05, 0.96
Tasks: 311 total, 1 running, 222 sleeping, 0 stopped, 0 zombie
%Cpu(s): 7.2 us, 3.2 sy, 0.0 ni, 89.0 id, 0.0 wa, 0.2 hi, 0.2 si, 0.0 st
KiB Mem : 16303428 total, 8454704 free, 3194668 used, 4654056 buff/cache
KiB Swap: 8220668 total, 8220668 free, 0 used. 11628168 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1693 jeff 20 0 4270580 346944 171372 S 29.8 2.1 9:31.89 gnome-shell
3140 jeff 20 0 3142044 928972 389716 S 27.5 5.7 13:30.29 Web Content
3057 jeff 20 0 3115900 521368 231288 S 18.9 3.2 10:27.71 firefox
6007 jeff 20 0 813992 112336 75592 S 4.3 0.7 0:28.25 tilix
1742 jeff 20 0 975080 164508 130624 S 2.0 1.0 3:29.83 Xwayland
1 root 20 0 230484 11924 7544 S 0.3 0.1 0:06.08 systemd
68 root 20 0 0 0 0 I 0.3 0.0 0:01.25 kworker/4:1
2913 jeff 20 0 965620 47892 37432 S 0.3 0.3 0:11.76 code
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
Overview of the most important fields:
PID: What is the numerical id of each process?USER: Who started the process?RES: What is the amount of memory currently being used by a process (in bytes)?%CPU: How much of a CPU is each process using? Values higher than 100 percent indicate that a process is running in parallel.%MEM: What percent of system memory is a process using?TIME+: How much CPU time has a process used so far? Processes using 2 CPUs accumulate time at twice the normal rate.COMMAND: What command was used to launch a process?
To exit press q.
Running this command as is will show us information on tasks running on the login node (where we should not be running resource intensive jobs anyway).
Running test jobs¶
As you may have to run several iterations before you get it right, you should choose your test job carefully. A test job should not run for more than 15 minutes. This could involve using a smaller input, coarser parameters or using a subset of the calculations. As well as being quick to run, you want your test job to be quick to start (e.g. get through queue quickly), the best way to ensure this is keep the resources requested (memory, CPUs, time) small. Some best practices for test jobs:
- Keep the test as similar as possible to actual jobs e.g. same functions etc.
- Use the same workflow, most issues are caused by small issues, typos, missing files etc. Your test job is a good chance to sort out these issues.
- Make sure outputs, logs and error files are going somewhere you can see them.
Serial Test
Often a good first test to run, is to execute your job serially e.g. using only 1 CPU. This not only saves you time by being fast to start, but serial jobs can often be easier to debug. If you confirm your job works in its most simple state you can identify problems caused by parallelisation much more easily.
You generally should ask for 20% to 30% more time and memory than you think the job will use. Testing allows you to become more precise with your resource requests. We will cover a bit more on running tests in the last lesson.
Example inputs
Mahuika has a shared directory (/opt/nesi/examples) with some small input files of commonly used types for use in testing.
If you have requests or example files to contribute to this directory, please Contact our Support Team.
Using the debug QoS¶
Before submitting a large job, first submit one as a test to make sure everything works as expected.
Often, users discover typos in their submit scripts, incorrect module names or possibly an incorrect path name after their job has queued for many hours.
Be aware that your job is not fully scanned for correctness when you submit the job.
While you may get an immediate error if your #SBATCH directives are malformed, it is not until the job starts to run that the interpreter starts to process the batch script.
Mahuika has an easy way for you to test your job submission.
One can employ the debug QOS to get a short, high priority test job.
Debug jobs have to run within 120 minutes and cannot use more than 2 nodes.
To use debug QOS, add or change the following in your batch submit script:
#SBATCH --qos=debug
#SBATCH --time=15:00
Adding these SBATCH directives will provide your job with the highest priority possible, meaning it should start to run within a few minutes, provided your resource request is not too large. Only use the debug QOS for one-off tests like this, not as a regular step in your workflow. See Debug QoS for when you may use it.
First estimates¶
As we have just discussed, the best and most reliable method of determining resource requirements is from testing, but before we run our first test there are a couple of things you can do to start yourself off in the right area.
- Read the documentation. REANNZ maintains documentation that does have some guidance on using resources for some software. However, as you noticed in the Environment and Modules lesson, we have a lot of software. So it is also advised to search the web for others that may have written up guidance for getting the most out of your specific software.
- Ask other users. If you know someone who has used the software before, they may be able to give you a rough estimate.
What Next
You can use this knowledge to set up the next job with a closer estimate of its load on the system. A good general rule is to ask the scheduler for 30% more time and memory than you expect the job to need.
Keypoints
- As your task gets larger, so does the potential for inefficiencies.
- The smaller your job (time, CPUs, memory, etc), the faster it will schedule.