0
I use a server at work to run models on modestly large datasets in RAM (think 10GB to 100GB). There are only a few people on this server at any time. The server has lots of RAM (over 1TB) and many processors. We have found that when RAM usage exceeds what seems to be a modest threshold - think three people loading a combined 100GB of data into RAM in applications like R or Stata - then the server becomes dramatically slower. Operations that would take seconds on my PC at home take hours or days on the server. I am not sure why this would be the case: it is as if the server doesn't want to free cached memory, and even operating on the data currently held in memory takes an insanely long amount of time (CPU load is low: <10%). Even stuff on the command line takes a while: listing files can take a few seconds, etc. I don't have permissions to edit things on the server myself. Does anyone know what could be going on here, or things I can hunt for without having root access? The system administrators don't know what is going on.
We are running Red Hat Enterprise 6.9.
Thanks so much for your help!
This feels like it's going to come down to the server not really using all of its RAM and ending up using virtual memory and swapping to disk too much (thrashing). – Spiff – 2019-02-16T02:27:32.143
Do you know how I can check this out, or what settings the system administrators should be tuning? If it helps, there is no swap space as far as I can tell (from top - swap's always at 0). That's probably a bad idea for a few reasons, but we're not getting processes killed because we have way more than enough RAM (except when we hit about 10-15% utilization, things slow down enormously). – framsey – 2019-02-16T05:03:13.363