cgroup memory.limit_in_bytes : is it VIRT or RES that is limited?

Question

Assume the VIRT/RES/SHM terminology that the "top" utility uses:

Does cgroup memory.limit_in_bytes apply to VIRT or RES memory sizes? I ask because in Fedora 23, it looked to me like RES (resident) memory was limited by cgroup. Then I installed Fedora 25, and suddenly I was getting frequent process kills that did not happen before (some dmesg output included at bottom to illustrate the problem). I have a hunch that between Fedora 23 and 25, cgroup changed behavior and now seems to limit VIRT instead of RES memory usage.

Can anyone confirm my suspicion? And is there a way to get the old behavior back? I have searched around in the cgroup kernel docs but have not found anything, perhaps because I'm searching with incorrect terminology.

dmesg output:

Tue Jun 20 01:46:12 2017: Memory cgroup out of memory: Kill process 787 (Web Content) score 204 or sacrifice child
Tue Jun 20 01:46:12 2017: Killed process 787 (Web Content) total-vm:14151032kB, anon-rss:5000940kB, file-rss:6132kB, shmem-rss:115956kB

Answer 1

It applies to the user memory including the file cache.

Check memory.limit_in_bytes here https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html

memory.limit_in_bytes sets the maximum amount of user memory (including file cache).

Being:

cache: page cache, including tmpfs (shmem), in bytes

rss: anonymous and swap cache, not including tmpfs (shmem), in bytes