15

/var/log/secure:

su: pam_keyinit(su-l:session): Unable to change UID to 500 temporarily
su: pam_keyinit(su-l:session): Unable to change UID to 500 temporarily
su: pam_unix(su-l:session): session opened for user adtech by root(uid=0)
su: pam_unix(su-l:session): session closed for user adtech

I guess this is caused by per-user limit, but there is no different when comparing with another user.

Here're ulimit -n for adtech:

[adtech@hmaster87 root]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 192025
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 655360
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

and this one for quanta:

[quanta@hmaster87 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 192025
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 655360
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

the number of processes running by adtech:

[root@hmaster87 ~]# ps -U adtech | wc -l
25

Any other thing to check?

UPDATE Sat Jul 21 09:21:26 ICT 2012:

# getent passwd adtech
adtech:x:500:502::/home/adtech:/bin/bash

As I said in the below comment, my co-worker had found out the process which maybe the culprit:

adtech 12901 1 0 08:58 ? 00:00:00 /home/adtech/nexus/bin/../bin/jsw/linux-x86-64/wrapper /home/adtech/nexus/bin/../bin/jsw/conf/wrapper.conf wrapper.syslog.ident=nexus wrapper.pidfile=/home/adtech/nexus/bin/../bin/jsw/linux-x86-64/nexus.pid wrapper.daemonize=TRUE

adtech 12903 12901 1 08:58 ? 00:00:24 java -Dsun.net.inetaddr.ttl=3600 -DbundleBasedir=. -Djava.io.tmpdir=./tmp -DjettyContext=nexus.properties -DjettyContextIncludeKeys=bundleBasedir -DjettyPlexusCompatibility=true -Djava.library.path=bin/jsw/lib -classpath bin/jsw/lib/wrapper-3.2.3.jar:./lib/plexus-classworlds-2.4.jar:./conf/ -Dwrapper.key=ejxHaBJASiFkAB8w -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=12901 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.codehaus.plexus.classworlds.launcher.Launcher ./conf/jetty.xml

By killing this process the problem go away but we still don't know which limit was exceeded.

UPDATE Sat Dec 15 00:56:13 ICT 2012:

The @favadi's answer is right, but I update here in case someone google's this thread.

The log file said that:

jvm 1    | Server daemon died!
jvm 1    | java.lang.OutOfMemoryError: unable to create new native thread
jvm 1    |      at java.lang.Thread.start0(Native Method)
jvm 1    |      at java.lang.Thread.start(Thread.java:640)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.privilegedStopInner(WrapperManager.java:3152)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3797)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
jvm 1    |      at java.lang.Thread.run(Thread.java:662)
quanta
  • 50,327
  • 19
  • 152
  • 213
  • Apologies if this is too obvious, but is there a userID 500 on your system? Does it relate to a userName that would be using? Good luck. – shellter Jul 20 '12 at 17:00
  • Sure, `adtech` user has UID 500. See my updated. My co-worker had found out the process which is culprit. By killing this process the problem go away but we still don't know which limit was exceeded: open files does not, number of processes does not, maybe memory or anything else. Any thoughts? – quanta Jul 21 '12 at 02:28
  • Try attaching strace -f -p to that process and looking for obviously failing syscalls and what they are trying to do... – rackandboneman Dec 10 '12 at 11:58

4 Answers4

12

It could be possible that the max user processes (-u) 1024 is too low.

Remember that processes and threads are counting together. You can use ps -eLF | grep adtech | wc -l to show your current value.

030
  • 5,731
  • 12
  • 61
  • 107
favadi
  • 537
  • 2
  • 7
  • 15
  • 7
    More precisely, it should be `ps -eLF -U adtech | wc -l`. – quanta Dec 14 '12 at 18:03
  • 2
    If you're wondering where that's set, look in /etc/security/limits.d/90-nproc.conf (assuming you're on a RH system). – mricon Dec 14 '12 at 18:04
  • @mricon checking `/etc/security/limits.d/90-nproc.conf` returns `/etc/security/limits.d/90-nproc.conf: No such file or directory` on CentOS7 – 030 Jul 07 '15 at 21:16
  • @Utrecht well, you could have done a "ls" in /etc/security/limits.d/ and noticed that on EL-7 it's called "20-nproc.conf", which would have probably been faster than asking it here. – mricon Jul 08 '15 at 19:00
  • 2
    @quanta, to be more precise, it should be `ps -LF -U adtech | wc -l`. When using the `-e` option you also get other users processes. – Lambert Sep 24 '15 at 07:58
2

Look in the jvm log for evidence it is hitting resource limits. Stack size might be the problem, depending how many java threads the killed process was running.

Searching on your error message finds bug reports for pam_keyinit: check with your vendor's repository whether an updated version is available.

ramruma
  • 2,730
  • 1
  • 14
  • 8
  • +1. I forgot the lesson: take a look at the log when getting the problem. Updated my question. – quanta Dec 14 '12 at 18:09
0

The error was reported by pam_keyinit. Since I am not familiar with this module, I searched for documentation and found this manpage. Based on the description, I wonder if perhaps the process that you killed prevented the necessary access to some files that pam_keyinit needs to modify? Hopefully this gives you some direction.

dsh
  • 303
  • 1
  • 6
0

This issue can happen if the user's process-run-limit is reached. The process limit can be increased by editing: /etc/security/limits.conf file with a user having root permission. The entry to check will be similar to:

*          hard     nproc         100

No need to restart any services.

j0k
  • 401
  • 9
  • 16
Santhosh Kamath
  • 1