I'm often experiencing slowness on a large server that is used by several users simultaneously, with many CPUs (72) and a decent amount of ram (125gb).
Granted, the server runs a lot of stuff so gets a high load, but it pretty much never happens that close to all CPUs are in use at the same time. Ram usage often goes up to about 80%. What I noticed is that iowait often becomes high (most of the time ~15%, I've seen ~20% tops), and also swap seems to be pretty much always exactly 100% utilized. Swap is 4gb.
In order to address the slowness, what would you recommend?
These are some options I've thought about:
Increase swap
Decrease swappiness to e.g. 10 (it's currently set at 60), so that at least whenever sufficient RAM is available swap isn't constantly full
It would be great to get your thoughts on these options.
I'm far from an expert in this domain, so any other suggestions are also very welcome.
Edit: adding details based on discussion in comments. See below:
vmstat
shows me a very high number for swap in, and using glances
, I see a warning (ongoing) MEMSWAP (100.0)