14

I have a linux box that is used as SAN storage. It is equipped with 14 SATA HDDs and the only software it run is iscsi target and zabbix agent. This is physical box, not VM.

The RAM size installed is 128 GB, swap was arranged of 4 GB and no-one ever care for its size when RAM is quite big, but as the box run (the uptime is 694 days) swap was used more and more, and now it used for ~80%.

The htop won't show anything bad, looks like OS uses RAM as cache etc. swappiness was set to 10, then to 1, none helped. Surely I can disable swap completely but looks like this will be bad move.

So my question is simple, how can I set the box to use RAM and not use swap, and why it uses swap at all while RAM is enough?

This is SAN, so it serves blocks not files, so cache can be done on block bases, and can easily be reduced if RAM is limited, it won't impact the logic of iSCSI anyway.

Zabbix says right now I have RAM available = 123.22 GB (hey, almost all of 128 GB!), swap used is 3.24 GB of 4 GB.

Anemoia
  • 296
  • 5
  • 15
Kevin M
  • 299
  • 3
  • 10
  • 2
    Do you see significant memory usage outside of fs cache? What does `free -m` tell you? Any processes using a lot of memory (`top`, sort by memory `M`; `ps auxf`, `htop`). There's paged out memory in swap, but is the system actively swapping? What does `vmstat 60` tell you throughout the hours? – marcelm Mar 11 '20 at 17:35
  • 1
    How do you determine swap is being used? It might simply be saving memory pages to disk as part of preparing to use the corresponding ram for something else. – Thorbjørn Ravn Andersen Mar 11 '20 at 22:21
  • Could it be for limiting power usage? Keeping stuff in RAM needs energy where it's free on disk (minus the access cost). I'm not sure whether linux can actually enable/disable the power supply to the RAM though. – Richard Mar 12 '20 at 00:43
  • 1
    IIRC ZFS uses a lot of memory; and its memory usage is not readily apparent from `top` and family. Are you using ZFS? – muru Mar 12 '20 at 07:30
  • 1
    Are you troubleshooting a performance issue? If so, please share those details. If not, I suspect you're seeing a problem where none exists. RAM and swap are resources available to the operating system. The fact that both are highly utilized means the OS is getting maximum benefit from both resources. – JakeRobb Mar 12 '20 at 14:26
  • @marcelm Seems no (0) bytes swap in and out anytime. So why swap used at all? – Kevin M Mar 13 '20 at 14:13
  • @muru No ZFS at all! – Kevin M Mar 13 '20 at 14:14
  • @KevinM That would suggest that the stuff in swap is static; it was swapped out in the past, and is not actively being swapped back in, nor is anything else swapped out. If that assessment is correct, then it's memory that was allocated but isn't being used, and swap would be a good place for that. Memory is better for stuff that is actually accessed. But 3GB of this "dead memory" is a lot in my opinion. I'm used to seeing maybe tens of MB for this. – marcelm Mar 13 '20 at 15:23

1 Answers1

23

There is no problem. Even if there is still available RAM, the Linux Kernel will move memory pages which are hardly ever used into swap space.

It’s good to swap out memory pages that have been inactive for a while, keeping often-used data in cache; this is the desired situation of the Kernel.

You can have more control on this by using vfs_cache_pressure (which controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects) and swappiness (which controls how aggressive the kernel will swap memory pages: higher values will increase aggressiveness, lower values decrease the amount of swap).

You can find these settings in /etc/sysctl.conf.

Overmind
  • 2,970
  • 2
  • 15
  • 24
  • 11
    To elaborate: there's some things that are kept in RAM, but seldom or never used. Think of `cron`, or the initializing part of some other daemons. I'd rather want them in swap, freeing a few memory pages for active jobs, than using RAM itself. Swapping out doesn't immediately remove them from RAM; it just means the kernel can drop them whenever it feels something else has better use for RAM. So as long as RAM is plenty, having some pages in swap doesn't hurt; when RAM becomes scarce, having pages in swap helps. – Guntram Blohm Mar 11 '20 at 16:36
  • 4
    -1: This answer is mostly based on myths. Linux shouldn't use swap when there's more than about 3% of RAM available (or even less for systems with a lot of RAM). More on that here: https://unix.stackexchange.com/questions/533739/does-linux-perform-opportunistic-swapping-or-is-it-a-myth/ – Spc_555 Mar 11 '20 at 17:05
  • 5
    @Spc_555 I think that question's definition of "opportunistic swapping" is different from the one needed here. That question asks whether pages can be swapped out when the system is doing nothing. Apparently, they can't. But this question asks whether anonymous (non-file-backed) pages can be swapped out, to make more space for disk caches. Apparently they can. – user253751 Mar 11 '20 at 18:18
  • This is a file server which has been running for two years, so presumably there is a lot of disk access. When Linux wants a page of memory, and all memory is used, it will decide whether it's better to send a page from a program into swap space, or whether it's better to free a page from the disk cache. As a file server, it's not surprising that the disk cache is often chosen to be more important... – user253751 Mar 11 '20 at 18:19
  • 4
    @Spc_555 If that were true, then why does every major Linux distro in the history of forever behave the way this answer describes? Are they *all* misconfigured out of the box? – StackOverthrow Mar 11 '20 at 23:02
  • Saying that it’s nothing bad when the host starts swapping is like it’s nothing bad when you cook omelet and all your eggs are spoiled. Swapping dramatically decreases host performance and should be avoided (but swap still should be enabled to allow host not OOM some process). – Alexey Kamenskiy Mar 12 '20 at 07:08
  • 7
    @AlexeyKamenskiy It heavily depends on the amount of swapping. Swapping per se doesn't decrease host performance if it is a "one-time" action. If the same pages have to be constantly swapped in and out, that's what decreases performance. – glglgl Mar 12 '20 at 10:53
  • 2
    @AlexeyKamenskiy Swapping only decreases performance, when either a program is waiting on the operation because it needs RAM, a program is waiting on the operation because it needs the page read from swapor other processes are waiting on IO on the same device. When swap is written on an idle disk when there is no urgent memory pressure, it does not affect the system performance at all. – allo Mar 12 '20 at 11:54
  • In the current situation, swapping should not affect performance since the available amount of RAM is very high compared to the swapped data. Tweaking it can benefit future situations (with more RAM usage) when more swapping may start to occur. – Overmind Mar 12 '20 at 13:16
  • So to say, when I have *126 out of 128 Gb* of RAM not used and `swappiness` set to very low value I doubt `vfs_cache_pressure` can help me to free up *swap*. – Kevin M Mar 13 '20 at 12:15