4

I'm finding instances of syslog-ng writing out garbage followed by a blank kernel.emerg line in one of our production environments. Example of one:

Dec 21 00:14:56 someserver [syslog-ng.err] Error processing log message: <Q▒b
+\c 21 00:14:56 someserver [syslog-ng.err] Error processing log message: <;E0
Dec 21 00:14:56 someserver [syslog-ng.err] Error processing log message: <▒"▒l
Dec 21 00:14:57 someserver [syslog-ng.err] Error processing log message: <▒▒▒▒e▒F
Dec 21 00:14:57 someserver [syslog-ng.err] Error processing log message: <▒▒
Dec 21 00:14:58 someserver [kernel.emerg]

The kernel.emerg line is of particular concern to me. According to man 3 syslog:

       LOG_KERN
              kernel messages (these can’t be generated from user processes)

This seems to suggest that the kernel facility cannot be spoofed to. I can see where the system calls themselves might prevent such spoofing, but am I correct in thinking that there is nothing to stop a process from writing directly to /dev/log and spoofing a facility of kernel? I want to say that the only thing that could truly stop such spoofing would be the syslog daemon differentiating between whether it obtained a message from /proc/kmsg vs. other sources.

  • Distro is RHEL5.5. Kernel version is 2.6.18-194.8.1.el5. These are not factors I have control over yet; consider me scolded.
  • The syslog daemon is a company-built syslog-ng 3.1.4 package (32-bit and running on a 64-bit kernel, but I wouldn't expact that to be related).
  • There are some mailing list posts I found via Google suggesting that changes to kmsg in kernel 3.5 can create output errors like this, but that's definitely not the case here.

The messages are not coming in from the network. This is the only source defined:

source s_syslog {
# message generated by Syslog-NG
internal();
# standard Linux log source (this is the default place for the syslog()
# function to send logs to)
unix-stream("/dev/log");
# messages from the kernel
file("/proc/kmsg" program_override("kernel: "));
};
Andrew B
  • 31,858
  • 12
  • 90
  • 128
  • Do you mean "what's to keep something from writing to /dev/log and simply prepending the literal string 'kernel: '"? – Bandrami Dec 21 '13 at 07:19
  • I'm trying to track down the origin of the blank `kernel.emerg` messages. I've tried to monitor the syslog daemon with `strace` on the `/proc/kmsg` file handle but haven't had any luck so far. (i.e. `strace -pPID -f -e read=8`) I wanted to make sure I understand how the protections work so that I can narrow my investigation further. – Andrew B Dec 21 '13 at 10:08
  • Definitely not coming from the `/proc/kmsg` source. as `program_override()` would have been injecting a prefix of "kernel: " to those lines. There are some clarifications in the documentation for `file()` that suggest a lack of facility for a message means that `kernel` will be assumed. No such statement is made under `unix-stream()`, but I would not be surprised if the same logic is being used there. I'll scan the source code on Monday. That likely solves my problem, but doesn't answer the question definitively. I'll leave this open so someone can snag the credit. – Andrew B Dec 21 '13 at 10:45

1 Answers1

1

Per the syslog-ng Administrators' Guide ( http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/html-single/index.html#kernel-messages ), the kernel facility (at least as defined by default) reads directly from /proc/kmsg, which cannot be written to from userland.

(This is from memory, but I'm pretty sure RHEL 5.5 doesn't need klogd or even ksymoops for symbol-to-address output; check your docs on that, though..)

If you're worried (see my comment above) about some rogue process prepending the literal string "kernel: " to their message, you can always add a filter that would remove the string "kernel: " at the start of any message received to be sure. Alternately, define /proc/kmsg as a separate source with a separate destination.

EDIT: On looking more at that section:

"Note

If the message does not have a proper syslog header, syslog-ng treats messages received from files as sent by the kern facility. Use the default-facility and default-priority options in the source definition to assign a different facility if needed."

Bandrami
  • 893
  • 4
  • 8
  • As near as I can tell based on that quoted section, there is no special treatment for the handling of `/proc/kmsg` vs. the handling of any other source being monitored by `file()`. – Andrew B Dec 21 '13 at 10:23
  • Check the note right above where I linked in the manual: "Note If the message does not have a proper syslog header, syslog-ng treats messages received from files as sent by the kern facility. Use the default-facility and default-priority options in the source definition to assign a different facility if needed." So, if syslog-ng can't figure out what it is, it thinks it's from the kernel. – Bandrami Dec 21 '13 at 11:10
  • Right - likewise note the comment I had posted before that. It was not coming from a `file()` source, which narrows it down to `unix-stream()` or `internal()` -- which have no such note. The most likely case is that they behave similarly and the documentation could stand for some improvement, but this has not been confirmed yet. Paragraph #1 is still incorrect. – Andrew B Dec 21 '13 at 11:22