41

I have two CentOS 5 servers with nearly identical specs. When I login and do ulimit -u, on one machine I get unlimited, and on the other I get 77824.

When I run a cron like: 

* * * * * ulimit -u > ulimit.txt

I get the same results (unlimited, 77824).

I am trying to determine where these are set so that I can alter them. They are not set in any of my profiles (.bashrc, /etc/profile, etc.). These wouldn't affect cron anyway) nor in /etc/security/limits.conf (which is empty).

I have scoured google and even gone so far as to do grep -Ir 77824 /, but nothing has turned up so far. I don't understand how these machines could have come preset with different limits.

I am actually wondering not for these machines, but for a different (CentOS 6) machine which has a limit of 1024, which is far too small. I need to run cron jobs with a higher limit and the only way I know how to set that is in the cron job itself. That's ok, but I'd rather set it system wide so it's not as hacky.

Thanks for any help. This seems like it should be easy (NOT).

EDIT -- SOLVED

Ok, I figured this out. It seems to be an issue either with CentOS 6 or perhaps my machine configuration. On the CentOS 5 configuration, I can set in /etc/security/limits.conf:

* - nproc unlimited

and that would effectively update the accounts and cron limits. However, this does not work in my CentOS 6 box. Instead, I must do:

myname1 - nproc unlimited
myname2 - nproc unlimited
...

And things work as expected. Maybe the UID specification works to, but the wildcard (*) definitely DOES NOT here. Oddly, wildcards DO work for the nofile limit.

I still would love to know where the default values are actually coming from, because by default, this file is empty and I couldn't see why I had different defaults for the two CentOS boxes, which had identical hardware and were from the same provider.

Totor
  • 2,876
  • 3
  • 22
  • 31
nomercysir
  • 411
  • 1
  • 5
  • 4

8 Answers8

56

These "default" limits are applied by:

  • the Linux kernel at boot time (to the init or systemd process),
  • inheritance, from the parent process' limits (at fork(2) time),
  • PAM when the user session is opened (can replace kernel/inherited values),
  • systemd, especially to the processes it manages,
  • the process itself (can replace PAM & kernel/inherited values, see setrlimit(2)).

Normal users' processes cannot rise hard limits.

The Linux kernel

At boot time, Linux sets default limits to the init (or systemd) process, which are then inherited by all the other (children) processes. To see these limits: cat /proc/1/limits.

For example, the kernel default for maximum number of file descriptors (ulimit -n) was 1024/1024 (soft, hard), and has been raised to 1024/4096 in Linux 2.6.39.

The default maximum number of processes you're talking about is limited to approximately:

Total RAM in kB / 128

for x86 architectures (at least), but distributions sometimes change default kernel values, so check your kernel source code for kernel/fork.c, fork_init(). The "number of processes" limit is called RLIMIT_NPROC there.

PAM

Usually, to ensure user authentication at login, PAM is used along with some modules (see /etc/pam.d/login).

On Debian, the PAM module responsible for setting limits is here : /lib/security/pam_limits.so.

This library will read its configuration from /etc/security/limits.conf and /etc/security/limits.d/*.conf, but even if those files are empty, pam_limits.so might use hardcoded values that you can check within the source code.

For example, on Debian, the library has been patched so that by default, the maximum number of processes (nproc) is unlimited, and the maximum number of files (nofile) is 1024/1024:

  case RLIMIT_NOFILE:
      pl->limits[i].limit.rlim_cur = 1024;
      pl->limits[i].limit.rlim_max = 1024;

So, check your CentOS' PAM module source code (look for RLIMIT_NPROC).

However, please note that many processes will not go through PAM (usually, if they are not launched by a logged in user, like daemons and maybe cron jobs).

systemd

Nowadays, systemd is widely used, it replaces init and can also configure specific limits values, especially to the processes/daemons it manages and creates itself.

Some limits it uses by default can be manually configured in /etc/systemd/system.conf. There is more information available in the documentation.

Totor
  • 2,876
  • 3
  • 22
  • 31
  • True, point taken, comment removed. I guess I would say that for most users, PAM is probably enabled, so I would recommend checking your /etc/security/limits.conf and /etc/security/limits.d/* files first. In this particular instance, which I also ran into, there is a 1024 process/total user threads limit imposed by default in CentOS 6 via a limits.d file. – rogerdpack Sep 12 '14 at 17:15
  • @rogerdpack yes, PAM is certainly enabled, but, again, as I said in my answer: "please note that many processes will not go through PAM (usually, if they are not launched by a logged in user, like daemons and maybe cron jobs)". Our discussion has no added-value, therefore, if you delete all your comments, I will delete mine. Thank you. – Totor Sep 15 '14 at 08:12
  • SuSE distributions have [ulimit](http://software.opensuse.org/package/ulimit) package that provided `/etc/initscript` -- "a convenient place to adjust per process limits", configurable via `/etc/sysconfig/ulimit`. – sendmoreinfo May 07 '16 at 08:55
  • also, Linux-PAM library reads limits set by kernel (i.e. `/proc/1/limits`) since version 1.1.4 (released 2011). – sendmoreinfo May 07 '16 at 09:30
  • @sendmoreinfo and what does Linux-PAM library does with limits set by kernel apart from reading them? – Totor May 19 '16 at 17:21
  • Treats them as the lowest level of default i.e. when its config does not specify any. – sendmoreinfo May 19 '16 at 20:30
15

On RHEL6 (CentOS6) "max user processes" is set to 1024 by default.
You can change this value in file: 

/etc/security/limits.d/90-nproc.conf

See https://bugzilla.redhat.com/show_bug.cgi?id=432903 if you'd like to complain about it :)

voretaq7
  • 79,345
  • 17
  • 128
  • 213
Tomas
  • 281
  • 2
  • 7
  • I doubt this 1024 value for nproc is correct and the author said that its limits.d dir was empty, so the default value is obviously not defined there. – Totor Mar 11 '13 at 15:33
  • Totor can't argue with you technically but Tom I found it helpful so thanks! – Partly Cloudy Oct 11 '13 at 17:23
6

Info on this is terrible on the internet, heres a limits.conf file i made for debian linux, showing all possible options and their maximum "safe" limits, tweak accordingly.

These are the highest values you can set, some things are hashed out, activating those causes you to error out and be unable to login to your console, modify the commented out options at your own risk, but you shouldnt need to (default is unlimited on most)

I hope this is usefull to someone, as i could not find this info anywhere, theres 4 hours of research on this file.

==== FILE START =====
# /etc/security/limits.conf
# 
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#- a user name
#- a group name, with @group syntax
#- the wildcard     *, for default entry
#- the wildcard %, can be also used with %group syntax,
#         for maxlogin limit
#- NOTE: group and wildcard limits are not applied to     root.
#  To apply a limit to the     root user, <domain> must be
#  the literal username     root.
#
#<type> can have the two values:
#- "soft" for enforcing the soft limits
#- "hard" for enforcing hard limits
#
#<item> can be one of the following:
#- core - limits the core file size (KB)
#- data - max data size (KB)
#- fsize - maximum filesize (KB)
#- memlock - max locked-in-memory address space (KB)
#- nofile - max number of open files
#- rss - max resident set size (KB)
#- stack - max stack size (KB)
#- cpu - max CPU time (MIN)
#- nproc - max number of processes
#- as - address space limit (KB)
#- maxlogins - max number of logins for this user
#- maxsyslogins - max number of logins on the system
#- priority - the priority to run user process with
#- locks - max number of file locks the user can hold
#- sigpending - max number of pending signals
#- msgqueue - max memory used by POSIX message queues (bytes)
#- nice - max nice priority allowed to raise to values: [-20, 19]
#- rtprio - max realtime priority
#- chroot - change     root to directory (Debian-specific)
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#root            hard    core            100000
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#ftp             -       chroot          /ftp
#@student        -       maxlogins       4

# -- Defaults:
#(core) core file size                (blocks, -c) 0 (ulimit -Hc or -Sc)
#(data) data seg size                  (bytes, -d) unlimited
#(priority) scheduling priority               (-e) 0
#(fsize) file size                    (blocks, -f) unlimited
#(sigpending) pending signals                 (-i) 378197
#(memlock) max locked memory          (kbytes, -l) 64
# max memory size                     (kbytes, -m) unlimited
#(nofile) open files                          (-n) 65536
# pipe size                        (512 bytes, -p) 8
#(msgqueue) POSIX message queues       (bytes, -q) 819200
#(rtprio) real-time priority                  (-r) 0
#(stack) stack size                   (kbytes, -s) 8192
#(cpu) cpu time                      (seconds, -t) unlimited
#(nproc) max user processes                   (-u) 378197
# virtual memory                      (kbytes, -v) unlimited
#(locks) file locks                           (-x) unlimited

# --     root Limits:
root               -    core            -1
root               -    data            -1
root               -    fsize           -1
root               -    memlock         -1
root               -    nofile          999999
root               -    stack           -1
root               -    cpu             -1
root               -    nproc           -1
root               -    priority        0
root               -    locks           -1
root               -    sigpending      -1
root               -    msgqueue        -1
root               -    rtprio          -1
root               -    maxlogins       -1
root               -    maxsyslogins    -1
#root               -    rss             -1
#root               -    as              -1
#root               -    nice            0
#root               -    chroot          -1

#All Users:
# -- Hard Limits
*               hard    core            -1
*               hard    data            -1
*               hard    fsize           -1
*               hard    memlock         -1
*               hard    nofile          999999
*               hard    stack           -1
*               hard    cpu             -1
*               hard    nproc           -1
*               hard    priority        0
*               hard    locks           -1
*               hard    sigpending      -1
*               hard    msgqueue        -1
*               hard    rtprio          -1
*               hard    maxlogins       -1
*               hard    maxsyslogins    -1
#*               hard    rss             -1
#*               hard    as              -1
#*               hard    nice            0
#*               hard    chroot          -1

# -- Soft Limits
*               soft    core            -1
*               soft    data            -1
*               soft    fsize           -1
*               soft    memlock         -1
*               soft    nofile          999999
*               soft    stack           -1
*               soft    cpu             -1
*               soft    nproc           -1
*               soft    priority        0
*               soft    locks           -1
*               soft    sigpending      -1
*               soft    msgqueue        -1
*               soft    maxlogins       -1
*               soft    maxsyslogins    -1
*               soft    rtprio          -1
#*               soft    rss             -1
#*               soft    as              -1
#*               soft    nice            0
#*               soft    chroot          -1

#randomuser:
# -- Soft Limits
randomuser           soft    core            -1
randomuser           soft    data            -1
randomuser           soft    fsize           -1
randomuser           soft    memlock         -1
randomuser           soft    nofile          999999
randomuser           soft    stack           -1
randomuser           soft    cpu             -1
randomuser           soft    nproc           -1
randomuser           soft    priority        0
randomuser           soft    locks           -1
randomuser           soft    sigpending      -1
randomuser           soft    msgqueue        -1
randomuser           soft    maxlogins       -1
randomuser           soft    maxsyslogins    -1
randomuser           soft    rtprio          -1
#randomuser           soft    rss             -1
#randomuser           soft    as              -1
#randomuser           soft    nice            0
#randomuser           soft    chroot          -1

# End of file
XionicFire
  • 61
  • 1
  • 1
3

When you checked the limits, were you using the root user to do so?

From the limits.conf manpage:

NOTE: group and wildcard limits are not applied to the root user. To set a limit for the root user, this field must contain the literal username root.

Using explicit usernames would resolve the issue in this case.

Christopher Cashell
  • 8,999
  • 2
  • 31
  • 43
  • Be careful, this is probably a [Debian specific](http://patch-tracker.debian.org/patch/series/view/pam/1.1.1-6.1+squeeze1/027_pam_limits_better_init_allow_explicit_root) "feature". – Totor Mar 06 '13 at 15:54
  • Also, the `limits.conf` file is empty (as the `limits.d` directory). – Totor Mar 11 '13 at 02:02
2

kernel/fork.c

max_threads = mempages / (8 * THREAD_SIZE / PAGE_SIZE);

On 64 bit Thread size is 8192

 grep -i total /proc/meminfo 
 MemTotal:        8069352 kB

Now i get the total in kb in division by 4

 echo $((8069352/4))
 2017338

Now i got the number of pages

 echo $((8 * 8192 / 4096)
 16

The final result is

echo $((2017338/16))
126083

In this way you got the thread-max parameter and the default user process limit is half

init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;

ulimit from root

ulimit -u
62932
echo $((62932*2))
125864 #we are near
c4f4t0r
  • 5,149
  • 3
  • 28
  • 41
2

There is one more possibility that the configuration for "noproc" is not working while configuring in /etc/security/limits.conf.

There is one more file which overrides your configuration /etc/security/limits.d/90-nproc.conf.

*          soft    nproc     1024
root       soft    nproc     unlimited

Here * config will override whatever you set in previous config file. So ideally you configure your setting in this file.

Suyash Jain
  • 241
  • 2
  • 9
1

It appears to be /etc/security/limits.conf

http://ss64.com/bash/limits.conf.html

jamesbtate
  • 567
  • 2
  • 6
  • 14
  • 1
    I mentioned that in my post already. It has no effect, nor are those values (unlimited, 77824) set there for the respective machines (that file is empty). – nomercysir Feb 05 '12 at 04:36
  • oh i saw you checked the .bashrc etc. but didn't see you mentioned this one too. – jamesbtate Feb 05 '12 at 05:00
0

I solved this after I struggled with this problem for more than an hour! I deleted the config and recreated the file like this:

vi /etc/security/limits.conf
root    -   nproc   500000
root    -   nofile  500000
myuser  -   nproc   130000
myuser  -   nofile  130000

After logout and login, limit was working:

# ulimit -n
500000
Arpatma
  • 1
  • 1