I have a cluster of servers, each of them having 128GB or RAM and 6 x 2TB spinning disks dedicated for BlueStore OSDs. The servers also act like KVM hosts, so they are not dedicated to Ceph. In the past when using FileStore we noticed that if a server has low available memory (e.g. 10-20G), then the OSDs on this host start to do a lot more IOs than the others, generally slowing down the whole cluster. Now with BlueStore, I can see that each OSD daemon reserves around 3-4GB of memory for cache. I have reserved 5GB per OSD per server that won't be spent on VMs just to be safe.
My questions is does it matter now how much more free memory a host has for the performance and do I need to pack most VMs on hosts without OSDs like before? Or I don't need to think about that as long as I don't run into an OOM situation?
I am using Ceph 14.2.0.