13

We have a CentOS OS that became unresponsive this morning to external network traffic. It is a virtual machine. I was able to reboot the VM. After logging back in, I found the following in the /var/log/messages file, repeating over and over, up to the point of the reboot:

Jan 21 06:53:01 PBX kernel: audit: backlog limit exceeded
Jan 21 06:53:01 PBX kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Jan 21 06:54:01 PBX kernel: printk: 8 messages suppressed.
Jan 21 06:54:01 PBX kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Jan 21 06:54:01 PBX kernel: audit: audit_lost=1130 audit_rate_limit=0 audit_backlog_limit=320

I read on another forum that the following command could identify the source of the backlog traffic:

[root@PBX log]# aureport --start today --event --summary -i

Event Summary Report
======================
total  type
======================
486  USER_ACCT
486  CRED_ACQ
486  USER_START
485  LOGIN
477  CRED_DISP
477  USER_END
6  USER_LOGIN
3  USER_AUTH
2  CONFIG_CHANGE
2  CRED_REFR
1  DAEMON_START

Can anyone advise me as to what next steps I should take in preventing this issue from happening again? I'm not particularly familiar with the purpose of the backlog or what the output of the event summary report means.

YWCA Hello
  • 203
  • 1
  • 2
  • 9
  • Can you rule out a storage problem? Logs are not written if the storage is inaccessible, but the kernel remains running - at least for a while. – the-wabbit Jan 23 '12 at 01:22
  • Storage is local and hasn't showed any signs of trouble. I think it's more likely that useful info is not being logged. – YWCA Hello Jan 26 '12 at 05:24

2 Answers2

5

You can increase the backlog by modifying -b 320 in /etc/audit/audit.rules to something larger and see if it has any effect, but these amounts you show us still very few audit results, so I doubt the audit error has anything much to do with the system freezing in itself. Its probably just a sympthom of something else happening.

Check /var/log/audit/audit.log to see what events have been logged to see if they can be of any use to your debugging.

Mattias Ahnberg
  • 4,039
  • 18
  • 19
  • `audit.log` essentially went quiet about 2 hours before we noticed the problem (this happened in the early morning). Then, messages started up again with the reboot. I'm hoping this isn't another linux freeze scenario where no real answers are found from the logs :/ – YWCA Hello Jan 21 '12 at 19:00
  • Note that on RHEL7-based system, you need to change the file /etc/audit/rules.d/audit.rules because the /etc/audit/audit.rules gets rewritten at restart of auditd. – Bruno Mairlot Jun 10 '15 at 08:07
4

There is multiple solution:

  1. To lengthen the backlog, add or edit /etc/audit/audit.rules by adding or editing "-b 320" to "-b 8192".
  2. change the priority by editing priority_boost from 3 to 4 or 5 in /etc/audit/auditd.conf.

To find out about what problem cause this issue, run aureport --start today or aureport --start today --event --summary -i

Esmaeil MIRZAEE
  • 141
  • 2