40

How do I switch on PAM debugging in Debian Squeeze at the admin level?

I have checked every resource I was able to find. Google, manpages, whatever. The only thing I haven't tried yet (I simply not dare to, did I mention that I hate PAM?) is digging into the PAM's library source.

I tried to google for a solution, nothing. What I found so far:

http://www.bitbull.ch/wiki/index.php/Pam_debugging_funktion (/etc/pam_debug) and http://nixdoc.net/man-pages/HP-UX/man4/pam.conf.4.html (debug option on PAM entries in /etc/pam.d/).

Nope, does not work. No PAM output, nothing, absolute silence.

While searching for a solution I even followed links to Pam, that are gas stations here in Germany. Well, yes, perhaps in all those billion of hits might hiding a clue, but shoot me I'd be dead before I discover.

Rest is FYI:

What problem did I have?

After upgrading to Debian Squeeze something got weird (well, hey, it once was, uh, what was right over the Etch .. ah, yes, Woody). So it's probably not Debian's fault, just a long lived screwed up setup. I immediately had the impression it has to do something with PAM, but I really did not know what's going on. I was completely in the dark, left alone, helpless as a baby, YKWIM. Some ssh logins worked, some not. It was kind of funny. No clues in ssh -v, no clues in /var/log/*, nothing. Just "auth succeeded" or "auth fail", sometimes the same user logging in parallely succeeded with one session and failed with the other, at the same time. And nothing you really can get hold of.

After digging trainloads of other options I was able to find out. There is nullok and nullok_secure, a Debian special. Something screwed with /etc/securetty and depending on the tty (which is somewhat random) a login was rejected or not. REALLY NICE, phew!

The fix was easy and everything's now fine again.

However this left me with the question, how to debug such a mess in future. It's not the first time PAM drives me nuts. So I would like to see a final solution. Final as in "solved", not final as in "armageddon". Thanks.

Ah, BTW, this again strengthened my belief in that it's good to hate PAM since it came up. Did I mention that I do?

0xC0000022L
  • 1,456
  • 2
  • 20
  • 41
Tino
  • 1,103
  • 1
  • 12
  • 16
  • Here is how to create this trouble yourself on Debian: `passwd -d user` and then try to ssh into the box as this `user`. The output "failed password" in syslog has nothing to do with PAM debugging at all, so PAM stays silent. – Tino Jul 21 '11 at 15:40
  • I forgot to mention that you must set `PermitEmptyPasswords yes` in `/etc/ssh/sshd_config` of course, then PAM outputs something like `pam_unix(sshd:auth): authentication failure`, but still nothing to the debug channel nor any hint which PAM module caused the failure. – Tino Jul 21 '11 at 15:49
  • 1
    Does debian have a `/var/log/auth.log` file? I recently discovered that Ubuntu has it, and logs all the pam related stuff there. None of the answers here helped me, but looking in `/var/log/auth.log` helped me fix my problem. – LordOfThePigs Jul 10 '14 at 09:41
  • `/var/log/auth.log` is `syslog`. The problem is not logging but debugging. If for example the PAM stack fails early, you will not see anything, because the modules which output to `syslog` are not invoked at all. Or something fails and something not, but both log exactly the same lines. It is correct that, I guess, 95% of all cases can be solved by looking into the usual logs, but 5% cannot, as there simply is no trace of what really happens behind the scenes. – Tino Jul 12 '14 at 13:10
  • 7
    +1 for hating PAM. ;) – Zayne S Halsall Nov 13 '15 at 12:43

7 Answers7

29

A couple of things for you to try:

Did you enable logging of debug messages in syslog?

cp /etc/syslog.conf /etc/syslog.conf.original
vi /etc/syslog.conf

Add the following line:

*.debug     /var/log/debug.log

Exit with :wq!.

touch /var/log/debug.log
service syslog restart

You can enable debugging for all modules like so:

touch /etc/pam_debug

OR you can enable debugging only for the modules you're interested in by adding "debug" to the end of the relevant lines in /etc/pam.d/system-auth or the other /etc/pam.d/* files:

login   auth    required    pam_unix.so debug

Then debugging messages should start appearing in /var/log/debug.log. Hope this helps you out!

Olli
  • 768
  • 6
  • 16
sn3ak3rp1mp
  • 306
  • 3
  • 3
  • Good answer but I think I had syslog debugging on. I will check it out. – Tino Jul 21 '11 at 15:18
  • I checked it, sorry, your answer is not the solution. PAM still stays silent. Perhaps this is a `nullok` special that just this module is lacking debugging. Let me say it with this words: Lacking debugging on such a crucial central piece of code is a worse nightmare than just being haunted by Freddy Kruger. – Tino Jul 21 '11 at 15:33
  • OK, well, I think you answer accutally is correct! It's not the answer's fault that `PAM` is mute. So for the time being I accept it as "solution" until `PAM` capitulates. Thanks. – Tino Jul 21 '11 at 15:58
  • 1
    I still don't see the debug output, but anyway, on Ubuntu 16.04, to view syslog debug, do: echo '*.debug /var/log/debug.log' > /etc/rsyslog.d/90-debug.conf; systemctl restart rsyslog.service – Noam Feb 01 '17 at 12:17
  • 1
    Note that you need proper permissions and file ownership on debug.log - set it to the same as syslog. (It's simple and easy to forget.) – mgarey Jun 29 '18 at 16:36
  • I wonder where this info about `/etc/pam_debug` comes from? Didn't work for me. I mean, on which OS's is it supposed to work, or under which conditions? – x-yuri Dec 24 '21 at 13:46
11

At least on CentOS 6.4, /etc/pam_debug is NOT used.

If the pam_warn.so module is installed, you can get some logging output this way:

auth required pam_warn.so

success required pam_warn.so

etc. This module ensures that it will not interfere with the authentication process at any point, but it logs meaningful stuff via syslog.

Update

After examining the code and doing some compiling, I found that (1) it's possible to enable this debug mode through the source, and (2) a RHEL patch makes the feature nearly unusable (at least the pam_unix module) and (3) it's probably better to patch the code anyway.

To get this to work for RHEL, you can get the Linux-PAM ... src.rpm (for any 1.1 version) and change the spec file as follows:

  • Find the line beginning with

    %configure \

and after it, add --enable-debug \

  • Remove the line or comment-out the line (above the previous one) that begins with %patch2

Then build the rpm and install (with force, to overwrite existing packages). Now create the file /var/run/pam-debug.log:

install -m 622 /dev/null /var/run/pam-debug.log

If the file does not exist, debug output will be sent to stderr.

  • This sending out to stderr is, in my opinion, stupid, and is what causes the patch conflict. You can change that behavior by going into the file libpam/include/security/_pam_macros.h and replacing the 4 lines of

    logfile = stderr;

with

return;

On build, you'll get warnings about unreachable statements, but they can be ignored. You can make this change in a sed script (and put it in the %prep section of the RPM after the patches)...

sed -i 's/logfile = stderr;$/return;/' libpam/include/security/_pam_macros.h

IF you do this little patch, you can restore the %patch2, as it should work again properly.

Otheus
  • 432
  • 3
  • 12
Otheus
  • 307
  • 3
  • 5
  • Thanks. Good hint. I'll try if I ever have problems again. So hopefully never .. ;) – Tino Feb 20 '14 at 04:48
  • This worked for me, but note that if you're running SELinux that you'll need to set the appropriate contexts on /var/run/pam-debug.log (system_u:object_r:var_log_t caught most of the messages). Otherwise a lot of the debugging output will be written to standard error (or silently discarded if you patched out the RedHat's standard error behavior). – jgibson Sep 07 '17 at 12:37
7

Debian and Ubuntu (and maybe other distros) have a special log file into which all pam output is logged:

/var/log/auth.log

I've been struggling with a pam related problem for a day and a half, finally found out about this log file, and saved myself from insanity.

Here's a sample of the contents of this file when things don't go as planned.

Jul 10 09:31:14 vagrant-ubuntu-trusty-64 pamtester: pam_userdb(vsftpd:auth): user_lookup: could not open database `/etc/vsftpd_users.db': No such file or directory
Jul 10 09:36:20 vagrant-ubuntu-trusty-64 sudo:  vagrant : TTY=pts/1 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/cat /var/log/auth.log

Here's how it looks when it works:

Jul 10 09:47:00 vagrant-ubuntu-trusty-64 sshd[5222]: pam_unix(sshd:session): session closed for user vagrant
Jul 10 09:50:58 vagrant-ubuntu-trusty-64 sshd[5584]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jul 10 09:50:58 vagrant-ubuntu-trusty-64 sshd[5584]: Accepted publickey for vagrant from 10.0.2.2 port 54652 ssh2: RSA dd:3b:b8:2e:85:04:06:e9:ab:ff:a8:0a:c0:04:6e:d6
Jul 10 09:50:58 vagrant-ubuntu-trusty-64 sshd[5584]: pam_unix(sshd:session): session opened for user vagrant by (uid=0)
Jul 10 09:51:13 vagrant-ubuntu-trusty-64 sudo:  vagrant : TTY=pts/1 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/cat /var/log/auth.log
Jul 10 09:51:13 vagrant-ubuntu-trusty-64 sudo: pam_unix(sudo:session): session opened for user root by vagrant(uid=0)
Jul 10 09:51:13 vagrant-ubuntu-trusty-64 sudo: pam_unix(sudo:session): session closed for user root
Jul 10 09:51:41 vagrant-ubuntu-trusty-64 pamtester: pam_userdb(vsftpd:auth): user 'foo' granted access
Jul 10 09:51:44 vagrant-ubuntu-trusty-64 sudo:  vagrant : TTY=pts/1 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/cat /var/log/auth.log
Jul 10 09:51:44 vagrant-ubuntu-trusty-64 sudo: pam_unix(sudo:session): session opened for user root by vagrant(uid=0)

Note that none of the other possibilities for enabling pam debug logging worked for me.

LordOfThePigs
  • 697
  • 1
  • 7
  • 7
  • 1
    Please note that all lines like `pam_*` actually are PAM realated. The other lines are output by the tools anyway, regardless if they use PAM or not. This means: If PAM denies, for any reason, it is really hard to find the real cause, if it is in PAM. The non-PAM lines are not helpful (as the problem sits in PAM) and the PAM lines are often not helpful, too, as they are often too silent. With the presence of many PAM modules you really have a hard time guessing which module might be the culprit, let alone to find out how to enable debugging, as this is different for each single one. – Tino Jul 12 '14 at 13:20
7

I just happened to spend several hours trying to find out how to enable debug logs in PAM on CentOS 6.4. Although this question is for Debian, I will still write down how to do it on CentOS in the hope that other people don't have to put in the time that I already have.

As it ultimately turned out, enabling debug logs in the pam CentOS package is straightforward. The difficulty stems from the fact that it involves recompilation of the package. So, first find the SRPM (e.g. pam-1.1.1-13.el6.src.rpm) from here. Folks who don't know about compiling packages from SRPMs, can refer to the steps on setting up a RPM build environment.

Here is the main step. Open the spec file and add --enable-debug to the %build section in the configure invocation. Recompile! Reinstall the newly created package. Finally, create the file where debug logs will get written.

$ sudo touch /var/run/pam-debug.log

If you don't create the file then a lot logs will fly by at the terminal, which might not be very useful.

wojtow
  • 163
  • 5
pdp
  • 778
  • 1
  • 6
  • 16
0

Basically this information is present in other answers, but mostly hidden in lines of text.

One option is to pass debug to a module, that makes it more verbose but not much.

If you need more information, you need to build pam with --enable-debug. Either by building from source (autoreconf && ./configure --enable-debug ... && make install), or from a source package (deb, rpm, ...), which is OS-specific. As a result, if it has enough permissions, it creates /var/run/pam-debug.log and writes to it. Otherwise, it outputs to stderr. You might want to create /var/log/pam-debug.log in advance for it to have enough rights.

About /etc/pam_debug, it didn't work for me.

x-yuri
  • 1,845
  • 1
  • 22
  • 27
0

Asket... I really loved your post :) I was fighting with stuff like this for the last 15 hours... (I might have had a 30 min break though)

Somehow i got it working by doing all the stuff you did, which means i have a /etc/pam_debug and debug on pam entries. BUT as in my case i was struggling with pam_mysql, I had to put another verbose=1 after debug on my pam entries:

mailsystem:~# cat /etc/pam.d/smtp

auth required pam_mysql.so debug sqllog=1 verbose=1 user=mail passwd=imverysecret host=127.0.0.1 db=mail table=mailbox usercolumn=username passwdcolumn=password crypt=1 logtable=log_auth logmsgcolumn=msg logusercolumn=user logpidcolumn=pid loghostcolumn=host logtimecolumn=time

account sufficient pam_mysql.so debug sqllog=1 verbose=1 user=mail passwd=imverysecret host=127.0.0.1 db=mail table=mailbox usercolumn=username passwdcolumn=password crypt=1 logtable=log_auth logmsgcolumn=msg logusercolumn=user logpidcolumn=pid loghostcolumn=host logtimecolumn=times

That "sqllog" is just to write logs in the DB.

So maybe this helps you just a little bit.

We all hate PAM. Good luck!

bahamat
  • 6,193
  • 23
  • 28
Alex
  • 11
  • 1
    Thanks for the hint, but unfortunately this does not help: `pam_unix(sshd:auth): unrecognized option [verbose=1]` – Tino Jul 21 '11 at 15:51
  • The "verbose"-flag is for pam_mysql (for which it is quite useful), it does not apply to pam_unix. – M. Schmidt May 10 '20 at 10:15
0

Something screwed with /etc/securetty and depending on the tty (which is somewhat random) a login was rejected or not. REALLY NICE, phew!

Could you expand on that a little?

Per securetty's manpage:

the file contains the device names of terminal lines (one per line, without leading /dev/) on which root is allowed to login.

The behaviour you describe sounds quite a lot like securetty is working normally (assuming you are indeed logging on as root).

Some ssh logins worked, some not.

Here too, there may be non-PAM restrictions in place, so it may help to provide some insight into what your /etc/ssh/sshd_config looks like.

Notably, from your description:

  • if you were trying to log in as root, and failing, then that could be because of this line being present in sshd_config: PermitRootLogin no
  • if some users/groups work, and others not, one reason could be the use in sshd_config of AllowGroups or AllowUsers. A sample line might look like: AllowGroups users admins

It is of course entirely possible PAM is part of the issue, but your main 'symptoms' sound to me like they could be explained by other means.

iwaseatenbyagrue
  • 3,588
  • 12
  • 22