5

I know it sounds odd but I need a slower or cached filesystem.

I have a lot of firewalls that are syslog'ing their data to a pair of Linux VMs which write these files to their 'local' (actually FC SAN attached) ext3-formatted disks and also forward the messages to our Splunk servers.

The problem is that the syslog server is writing these syslog messages as hundreds, sometimes thousands, of tiny ~4k writes per second back to our FC SAN - which can handle this workload right now but our FW traffic's going to be growing by at least a factor of 5000% (really) in coming months and that'll be a pain for the SAN, I want to fix the root cause before it's a problem.

So I need some help figuring out a way of getting these writes cached or held-off in some way from the 'physical' disks so that the VMs fire off larger, but less frequent, writes - there's no way of avoiding these writes but there's no need for it to do so many tiny ones.

I've looked at the various ext3 options, setting noatime and nodiratime but that's not made much of a dent in the problem. Obviously I'm investigating other file systems but thought I'd throw this out in case others have the same problem in the future.

Oh and I can't just forward these messages to Splunk, our firewall team insist they're in their original format for diag purposes.

Chopper3
  • 100,240
  • 9
  • 106
  • 238

2 Answers2

1

Perhaps commit ext3 mount option would help you? For example, commit=60 would flush all the data and the meta-data only once per minute.

Obligatory warning: this might lead to data loss of up to one minute of data (if you pass the commit=60 value).

Janne Pikkarainen
  • 31,454
  • 4
  • 56
  • 78
  • Isn't that just for journal commit, not data? – Chopper3 Nov 23 '11 at 10:08
  • That's what I originally thought, but mount man page says about commit `Sync all data and metadata every nrsec seconds. The default value is 5 seconds. Zero means default.` ... perhaps together with `commit` and `data=journal` you could achieve what you want? With data=journal `All data is committed into the journal prior to being written into the main file system.` – Janne Pikkarainen Nov 23 '11 at 10:43
  • It might be the data=journal bit I need, thanks so far. – Chopper3 Nov 23 '11 at 10:55
1

Filesystem: Disable write barriers if your devices are using them, and disable atime updates across the board.

But chances are you can also tune your syslog at the expense of greater loss of data in the case of a failure event (power, etc).

Example directives from syslog-ng (which may not be what you are using):

  • flush_lines() Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch.

  • flush_timeout() Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.

The destination is in this case is a disk file.

Yolo Perdiem
  • 606
  • 1
  • 5
  • 14
  • I'd already mentioned I've disabled atime and we're not using syslog-ng either - I'll have a look at the write barriers though. – Chopper3 Nov 25 '11 at 21:05
  • Which syslog server *are* you using? If it's vanilla syslogd, you can prefix the destination filename with a dash `-` to prevent every line from being forced to disk. – MikeyB Nov 25 '11 at 21:31
  • If the problem is significant enough, would you consider changing to a syslog daemon which does support what you need to do? – Yolo Perdiem Nov 25 '11 at 22:09
  • Not sure how much choice I've got in that area, our security guys tend to get what they want – Chopper3 Nov 26 '11 at 08:35