From openai.gpt-5 response

Diagnosing High Load Average on Linux

A high load average means many tasks are either runnable (want CPU) or stuck in uninterruptible sleep (usually waiting on I/O) over time. It is not the same as high CPU usage. The goal is to determine whether the load is due to CPU saturation, slow I/O, memory pressure/thrashing, lock contention, or external factors (virtualization, NFS, cgroup limits).

Quick orientation

Show load and runnable count:
```
uptime cat /proc/loadavg
```
The fourth field in /proc/loadavg shows running/total tasks (e.g., 3/512).
Top-level health in seconds:
```
vmstat 1
```
Key columns:
- r: runnable tasks (CPU pressure). Persistently high r means CPU saturation.
- b: tasks in uninterruptible sleep (I/O wait). High b points to I/O or kernel waits.
- wa: CPU time waiting for I/O. High wa often accompanies high b.
- si/so: swap in/out. Non-zero indicates memory pressure/thrashing.
View per-CPU usage:
```
top # press 1 for per-CPU mpstat -P ALL 1
```
If all CPUs are near 100% user/system, it’s CPU-bound. If %steal is high, you’re losing CPU to the hypervisor.

Find the bottleneck category

CPU-bound

Identify hot processes/threads:

top # press H to show threads; add WCHAN column htop pidstat -u -t 1

See hot functions (user or kernel):
```
perf top # requires perf
```
Indicators: High r in vmstat, high user/system in mpstat/top, low wa, low b.

I/O-bound (disk, filesystem, NFS)

List tasks stuck in D-state (uninterruptible I/O) and what they wait on:
```
ps -eo pid,stat,comm,wchan:32 --sort=stat | grep ' D '
```
WCHAN hints (e.g., do_fsync, wait_on_page, nfs_wait_on_request).
Disk latency and queue:
```
iostat -x 1
```
Look at:
- await or r_await/w_await: Average request latency. Hundreds of ms+ is bad.
- avgqu-sz: Queue depth; consistently > 1 indicates a bottleneck.
- %util near 100% indicates device saturation.
Per-process I/O and delays (if delay accounting enabled):
```
iotop -oPa
```
For NFS:
```
nfsiostat nfsstat
```
Kernel messages:
```
dmesg | tail -n 200 journalctl -k
```
Look for disk errors, “blocked for more than X seconds”, I/O timeouts, controller resets.
System-wide I/O stall signals (PSI):
```
cat /proc/pressure/io
```
High “some” or “full” indicates I/O stalls affecting workloads.
Indicators: High b in vmstat, high wa in CPU stats, many D-state tasks with I/O-related WCHAN, high disk await/queue.

Memory pressure / thrashing

Reclaim and swapping:

vmstat 1 # watch si/so free -h sar -r 1 # memory sar -W 1 # swapping

Page faults per process:

pidstat -r 1 # major faults indicate disk-backed memory pressure

PSI memory stalls:
```
cat /proc/pressure/memory
```

Kernel logs:

dmesg | grep -E 'oom-killer|kswapd|memory'

Indicators: Swap in/out non-zero, reclaim activity (e.g., kswapd0 using CPU), high memory PSI; load rises even if CPU isn’t fully utilized because tasks stall on page faults and reclaim.

Lock contention / synchronization

Context switches and blocked threads:
```
pidstat -wt 1
```

Waiting on locks/semaphores:

ps -eo pid,stat,comm,wchan:32 | grep -E 'futex|rwsem|mutex' cat /proc/<pid>/stack

Observe blocking syscalls (use with care):
```
strace -p <pid>
```
Kernel lock contention:
```
perf lock record perf lock report
```
Indicators: Many runnable threads but poor progress, or D-state with futex/rwsem waits; CPU may be partly idle.

External limits and platform factors

Virtualization steal time:
```
mpstat -P ALL 1 # check %steal
```
High %steal means hypervisor is taking CPU time.
Cgroups/containers:
```
systemd-cgtop cat /sys/fs/cgroup/*/cpu.stat cat /sys/fs/cgroup/*/memory.pressure
```
Throttling or tight limits can inflate load by keeping tasks runnable but capped.
Interrupt storms or driver issues:
```
mpstat -I SUM 1 cat /proc/interrupts
```
Network/NFS/iSCSI:
```
ss -s iftop / nload
```
Network stalls can back filesystems (e.g., NFS).

A concise investigative runbook

```
uptime; cat /proc/loadavg
```
```
vmstat 1 # watch r, b, wa, si/so
```

top # press 1 for per-CPU; H for threads; note D-state processes

```
mpstat -P ALL 1
```
```
iostat -x 1
```
```
iotop -oPa
```
```
pidstat -u -d -r -wt 1
```

ps -eo pid,stat,comm,wchan:32 --sort=stat | grep ' D '

```
cat /proc/pressure/{cpu,io,memory}
```
```
dmesg | tail -n 200
```

How to interpret and act

Mostly CPU-bound:
- Reduce concurrency, optimize hot code paths, profile with perf.
- Check for runaway tasks; ensure no real-time/high-priority task starving others.
Mostly I/O-bound:
- Check disk/NFS health, throughput, latency; fix failing hardware.
- Tune queues/scheduler, mount options; reduce sync writes; batch I/O.
- Move hot data to faster storage; ensure write-back/journaling aren’t throttling.
Memory pressure:
- Add RAM; reduce working set; fix leaks; tune vm.swappiness.
- Ensure swap is fast; avoid overcommit; adjust container limits.
Lock/contention:
- Reduce thread count; fix locking; shard workloads; tune DB connection pools.
- Investigate futex/rwsem hotspots with perf/perf lock.
Virtualization or cgroups:
- Address steal time, CPU caps, or throttling; adjust limits or move workloads.

Notes

Load average is an exponential moving average over 1/5/15 minutes, so it lingers; use time-series tools (e.g., sar -q 1, netdata, collectd) to see when it rose.
Linux counts runnable tasks and tasks in uninterruptible sleep (D) toward load. That’s why you can have high load with low CPU if I/O is slow.
On newer kernels, PSI (/proc/pressure) is extremely useful to quantify whether stalls are CPU, I/O, or memory driven.
Some tools (iostat, mpstat, pidstat) come from the sysstat package; install it if missing.