5

Running an Ubuntu Trusty 14.04.1 LTS server at Rackspace but recently after running updates for bash, then with the reboots for the xen host vulnerability, I have the following weird issue.

Some random auth.log entries are popping up out of date sync, for example:

Oct 14 12:12:10 myserver sshd[2097]: pam_unix(sshd:session): session closed for user
Oct 13 12:58:30 myserver sshd[2522]: Failed password for foo from 21.21.21.21 port 1490 ssh2
Oct 14 12:21:28 myserver sshd[3389]: Accepted password for bar from 22.22.22.22 port 61173 ssh2

I have a seperate cron.log running too, (I changed the configs for rsyslog) but have random out of date sync entries there too, for example:

Oct 14 07:11:01 myserver CRON[32099]: (root) CMD
Oct 13 03:16:01 myserver CRON[32226]: (root) CMD
Oct 14 07:12:01 myserver CRON[32226]: (root) CMD

EDIT TO ADD:

I've noticed that it looks like some log entries are delayed being added to the *.log by rsyslog. Some entries show the following pattern:

Oct 12 09:19:07 myserver sshd[4792]: pam_unix(sshd:auth): authentication failure;
Oct 12 09:19:09 myserver sshd[4792]: Failed password for x
Oct 14 12:21:32 myserver su[3484]: pam_unix(su:session): session open
Oct 12 09:19:12 myserver sshd[4792]: Failed password for x from 20.20.20.20 port 2158 ssh2

The above example shows the first two entries came in on time, but the last one (12 seconds) was actually delayed by two days!

I have checked the date on the server is correct, restarted SSH processes, restarted rsyslog.

Everything else on server is running fine - Apache, sync, varnish etc etc

Any ideas on what I'm missing here?

Ray A
  • 153
  • 5
  • Please verify that there are no duplicate of process. Because maybe you did restart rsyslog but actually that restarted process specified in a PID file and a bad one copy kept running. – Glueon Oct 14 '14 at 12:41
  • pgrep rsyslog or pgrep syslog produces only one PID – Ray A Oct 14 '14 at 12:42
  • 2
    Seems to be a similar error to this one previous posted: http://serverfault.com/questions/608690/weird-syslog-order – Ray A Oct 14 '14 at 13:10
  • @RayA It does indeed look like the same problem. But I am hesitating to vote to close, because I feel the newer question is better worded than the older. – kasperd Dec 28 '14 at 13:47

1 Answers1

6

This is a tricksy one, and it's actually a bug in rsyslog, specifically RepeatedMsgReduction On, and a change in behaviour with the version released with Trusty (compared to earlier versions)

See http://bugzilla.adiscon.com/show_bug.cgi?id=527 for the gory details.

In short, turn off RepeatedMsgReduction on Trusty. It's not helpful, and does dumb things.

Craig Miskell
  • 4,086
  • 1
  • 15
  • 16
  • Hi Craig, Thanks for the update. I did indeed upgrade my Ubuntu to Trusty prior to seeing the errors in rsyslog! I've made the necessary adjustments to the config and restarted so let's hope this sorts this all out. I'll mark the the question answered if it corrects the issue. – Ray A Dec 29 '14 at 13:38
  • After 48 hours of logging all seems back to normal - thanks! – Ray A Dec 31 '14 at 13:19