I have five CentOS 6 linux systems at work, and encountered a rather strange issue that only seems to happen with my userid across all the linux systems I have... This is an example of the problem from entries I excepted from the last
command...
mpenning pts/19 Fri Nov 16 10:32 - 10:35 (00:03)
mpenning pts/17 Fri Nov 16 10:21 - 10:42 (00:21)
bill pts/15 sol-bill.local Fri Nov 16 10:19 - 10:36 (00:16)
mpenning pts/1 192.0.2.91 Fri Nov 16 10:17 - 10:49 (12+00:31)
kkim14 pts/14 192.0.2.225 Thu Nov 15 18:02 - 15:17 (4+21:15)
gduarte pts/10 192.0.2.135 Thu Nov 15 12:33 - 08:10 (11+19:36)
gduarte pts/9 192.0.2.135 Thu Nov 15 12:31 - 08:10 (11+19:38)
kkim14 pts/0 :0.0 Thu Nov 15 12:27 - 15:17 (5+02:49)
gduarte pts/6 192.0.2.135 Thu Nov 15 11:44 - 08:10 (11+20:25)
kkim14 pts/13 192.0.2.225 Thu Nov 15 09:56 - 15:17 (5+05:20)
kkim14 pts/12 192.0.2.225 Thu Nov 15 08:28 - 15:17 (5+06:49)
kkim14 pts/11 192.0.2.225 Thu Nov 15 08:26 - 15:17 (5+06:50)
dspencer pts/8 192.0.2.130 Wed Nov 14 18:24 still logged in
mpenning pts/18 alpha-console-1. Mon Nov 12 14:41 - 14:46 (00:04)
You can see two of my pts login entries above that do not have a source IP address associated with them. My CentOS machines have as many as six other users that share the systems. Approximately 10% of my logins see this issue, but no other usernames exhibit this behavior. There is no entry in /var/log/secure
for the entries without a source IP address.
Questions
Given the kind of scripts I keep on these systems (which control much of our network infrastructure), I'm a little spooked by this and would like to understand what would cause my logins to occasionally miss source addresses.
- Why does
last -i
show0.0.0.0
for pts line entries (also see this answer) - Is there anything (other than malicious activity) that would reasonably explain the behavior?
- Other than bash history timestamping, are there other things I can do to track the issue down?
Informational
Since this started happening, I enabled bash
history time-stamping (i.e. HISTTIMEFORMAT="%y-%m-%d %T "
in .bash_profile
) and also added a few other bash history hacks; however, that does not give clues to what happened during the previous occurrences.
All the systems run CentOS 6.3...
[mpenning@typo ~]$ uname -a
Linux typo.local 2.6.32-279.9.1.el6.x86_64 #1 SMP Tue Sep 25 21:43:11 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
[mpenning@typo ~]$
EDIT
If I use last -i mpenning
, I see entries like this...
mpenning pts/19 0.0.0.0 Fri Nov 16 10:32 - 10:35 (00:03)
mpenning pts/17 0.0.0.0 Fri Nov 16 10:21 - 10:42 (00:21)
Note for those trying to answer: I have not logged in with the screen
command or the GUI. All my logins are from SSH; to receive the bounty award, you must cite authoritative references to explain the last -i
0.0.0.0
entries sourced only via SSH.
EDIT 2 (for ewwhite's questions)
/etc/resolv.conf
(note that I used .local
addrs in last
output above to hide my company's info)
[mpenning@sasmars network]$ cat /etc/resolv.conf
nameserver 192.0.2.40
nameserver 192.0.2.60
domain mycompany.com
search mycompany.com
[mpenning@sasmars network]$
/etc/hosts
info (note that this customized hosts file only exists on one of the machines that has these problems)
[mpenning@sasmars network]$ cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.0.2.44 sasmars.mycompany.com sasmars
::1 localhost6.localdomain6 localhost6
## Temporary kludge until I add reverse hostname mappings...
## Firewalls
192.0.2.254 a2-inet-fw1
192.0.2.253 a2-inet-fw2
192.0.2.254 a2-wan-fw1
192.0.2.253 a2-wan-fw2
192.0.2.201 a2-fab-fw1
192.0.2.202 a2-fab-fw2
192.0.2.203 t1-eds-fw1
192.0.2.42 sasvpn
192.0.2.246 sasasa1
192.0.2.10 sasoutfw1
## Wireless
192.0.2.6 saswcs1
192.0.2.2 l2wlc3
192.0.2.4 l2wlc4
192.0.2.12 f2wlc5
192.0.2.16 f2wlc6
192.0.2.14 f2wlc1
192.0.2.8 f2wlc2
[mpenning@sasmars network]$
sftp
Output from /var/log/secure
*
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: called (pam_tacplus v1.3.7)
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: user [mpenning] obtained
Dec 26 10:36:37 sasmars sshd[26016]: tacacs_get_password: called
Dec 26 10:36:37 sasmars sshd[26016]: tacacs_get_password: obtained password
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: password obtained
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: tty [ssh] obtained
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: rhost [192.0.2.91] obtained
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: trying srv 0
Dec 26 10:36:38 sasmars sshd[26016]: Accepted password for mpenning from 192.0.2.91 port 55118 ssh2
Dec 26 10:36:38 sasmars sshd[26016]: pam_sm_setcred: called (pam_tacplus v1.3.7)
Dec 26 10:36:38 sasmars sshd[26016]: pam_unix(sshd:session): session opened for user mpenning by (uid=0)
Dec 26 10:36:38 sasmars sshd[26018]: pam_sm_setcred: called (pam_tacplus v1.3.7)
Dec 26 10:36:38 sasmars sshd[26018]: subsystem request for sftp
Dec 26 10:37:20 sasmars sshd[26016]: pam_unix(sshd:session): session closed for user mpenning
Dec 26 10:37:20 sasmars sshd[26016]: pam_sm_setcred: called (pam_tacplus v1.3.7)
FINAL RESOLUTION
See my answer below