Increasing ulimit open files not applied in RHEL 8.1

Question

I have a fully-updated Red Hat Enterprise Linux (RHEL) 8.1 x86_64 system where, as a local user in a local terminal (i.e. not remotely via SSH), when I run ulimit -Sn it says 1024 and ulimit -Hn yields 4096. I would like to increase those numbers because often when I compile large projects like gcc, the make step would fail with a too many open files error. I've even tried moving the compilation into a podman container, but that didn't help because the container just inherited my local user's ulimits.

Here's what I've tried on the host:

1. In /etc/security/limits.conf I've added:

*          soft nofile 32767
*          hard nofile 65535
[username] soft nofile 32767
[username] hard nofile 65535

2. I checked ls -a /etc/security/limits.d/ and there are no files there.
3. In /etc/sysctl.conf there is a line stating fs.file-max = 65535. I also ran sysctl -p.
4. I added session required pam_limits.so to the end of the file /etc/pam.d/login.
5. ls -a /etc/sysctl.d/ showed two files ./ ../ 50-libreswan.conf 99-sysctl.conf@, neither contained anything related to the number of open files.
6. cat /proc/sys/fs/file-max says 65535.
7. /etc/systemd/user.conf is empty (all lines commented out), and I tried to explicitly set DefaultLimitNOFILE=65535.

After trying the above, when I run cat /proc/self/limits or prlimit, the relevant output line still says:

NOFILE     max number of open files                1024      4096 files

The only clue I have right now is that when I use Gnome's System Activity app to look at my processes, when I have a terminal window open there are two bash processes, both under my local user. Strangely, when I run cat /proc/[process id]/limits on the two processes, one shows the large 32767 limit but the other has the original 1024 limit. I thought of checking my ~/.bashrc and ~/.bash_profile but they don't change my ulimit.

I'm stumped!

What else can I check and try in order to increase my ulimit -n? Could systemd or SELinux have something to do with it? Thank you for your help.

EDIT: Per @Gerrit's suggestion, I ran systemd-cgls which says that practically all things running under my local user are under:

-.slice
├─user.slice
│ ├─user-1005.slice
│ │ ├─session-2.scope
│ │ │ └─[many entries here]
│ │ └─user@1005.service
│ │   └─[many entries here]

Answer 1

In a systemd setup before version 240 all user processes run under default user resource limits suggested by the kernel, which on RHEL 8.1 appears to be 4096 for the hard limit and 1024 for the soft limit. In systemd 240 this changed to set the default hard limit to 512K, the default soft limit remains the same, to avoid problems with programs reserving memory for every potential file descriptor or other compatibility issues.

You can change the default HARD limit of NOFILE in /etc/systemd/system.conf in the [Manager] section with the DefaultLimitNOFILE= setting. A reboot is required to make this effective.

After that you can indeed raise the default Soft limit of NOFILE by setting it in /etc/security/limits.conf with * soft nofile nnnn. Adding pam_limits to /etc/pam.d/login is not required, at least not in Centos 7, as that is already included through /etc/pam.d/system-auth.