For some reason our system starts swapping (actively used memory) at around 60GB used memory. (see edit below, it seems I/O and disk cache usage, even of previously run processes has an effect)
Turning it off (swapoff-a) for test lead to bad_alloc's once (i guess because there were more processes also using memory at the time) but has also worked to speed up my program by a factor of mroe than 10.
This reproduces the problem (without other significant processes running EDIT: see below it only happens if an I/O intense process ran shortly before):
#include <cstdio>
#include <vector>
int main() {
size_t bytes = size_t(80) * 1024 * 1024 * 1024; // 80GB
size_t* data = new size_t[bytes / sizeof(size_t)];
for (size_t i = 0; i < bytes / sizeof(size_t); ++i) {
data[i] = i;
}
for (;;) {}
}
At around 60GB used memory, the system starts swapping and CPU usage goes below 100% (because the process is I/O bound now, I guess).
System is a Ubuntu 14.04, 64bit:
Linux ... 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Once the point is reached, this what free -m gives me:
total used free shared buff/cache available
Mem: 96671 55504 358 60 40808 40478
Swap: 47679 19366 28313
The problem persists with swappiness=1 (if something I/O intensive has just run and especially for my actual culprit (not the code above) that does both, lots of I/O and use much memory. If there has been little to no I/O recently, the program above allocates all memory and doesn't swap!
The problem goes away if there has been nothing I/O intense for a longer time. It seems the OS somehow makes my application swap because it thinks the disk cache is more valuable - even at very low swappiness. I don't understand this behavior because disk-cache memory should be just as good as free memory and most certainly don't trigger swapping on the running process.
Initially my problem happened in an application that reads large files and uses much memory. Afterwards it persist to the example code above that has no I/O at all. Finally, when I start the example code later on, no swapping occurs.
