14

I have noticed recently that sometimes tail -f <logfile> will stop updating to the screen.

Doing a Ctrl>-C and restarting the tail works fine, though. And I checked to make sure the logfile isn't being rotated midstream (which can make tail lose its mind).

What would cause this? I'm running RHEL 5.2 x64.

warren
  • 17,829
  • 23
  • 82
  • 134
  • Does this happen on all logfiles, or just certain ones? What filesystem is the logfile on? What application is writing the logfile? Have you timed the failure to see if it happens either (1) a certain period of time after you started the tail command; or (2) at certain fixed intervals (eg always at :00 :10 :20 :30 :40 and :50 after the hour)? – daveadams Feb 08 '11 at 18:42
  • @daveadams - so far as I've noticed, it's only been on two (one on each of two servers): in this case, the dataminer logfile for BSAE (reporting tool) on an HPSA Core server (data center automation) and the server.log for BSAE on the reporting host – warren Feb 08 '11 at 19:05
  • 1
    as an aside, tail -F will continue to work even if the file is removes and later replaced by another file entirely. – Sirex Feb 16 '11 at 16:30

4 Answers4

10

Try using:

tail --follow=name <logfile>

And see if that works better. You don't have to worry about it being rotated out from under you.

Any pattern to it stopping? Certain length of time? Certain time of day?

MikeyB
  • 38,725
  • 10
  • 102
  • 186
  • the file is not being rotated-out from underneath `tail` - it's just periodically (sometime between 2 and 20 hours) stopping to follow anymore.. wish there was more of a pattern :-\ – warren Feb 08 '11 at 18:29
  • If you hit other keys on the keyboard (other than ctrl-c) does it get better? When it stops, you could try attaching to it using strace/ltrace/etc. to see if it's doing anything. – MikeyB Feb 08 '11 at 18:31
  • good thought on the strace - neither enter nor other keys get anything to happen either; I sometimes like to leave a tail-f running in a `screen` session for extended debugging, and this is worrisome – warren Feb 08 '11 at 18:35
  • 1
    Ah… possibly not your issue, but ensure you don't accidentally leave a screen window open in copy mode with tail -f running. Copy mode will make command output block eventually (when the buffer fills up). – MikeyB Feb 09 '11 at 01:37
  • Excellent answer. Of course my files were getting rotated out at the top of each hour! – alex Jul 11 '19 at 08:09
  • What does `--follow=name` do? – Geoffrey Jan 20 '22 at 07:42
  • as per the man page: "Use --follow=name […]. That causes tail to track the named file in a way that accommodates renaming, removal and creation." – MikeyB Jan 21 '22 at 21:50
6

Try wrapping your tail command with strace if you have it:

strace -Tt -o /tmp/tail.trace tail -f /var/log/messages

Then just for crazy recursive kicks you can tail the strace output (doesnt' matter if that breaks because its going out to a file):

 tail -f /tmp/tail.trace

Mine looks like:

8:39:00 write(1, "ng SMAC\n", 8)       = 8 <0.000026>
18:39:00 read(3, "", 0)                 = 0 <0.000019>
18:39:00 fstat64(3, {st_mode=S_IFREG|0640, st_size=92990, ...}) = 0 <0.000019>
18:39:00 fstatfs64(3, 84, {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=4807069, f_bfree=1924458, f_bavail=1680271, f_files=1221600, f_ffree=820806, f_fsid={-1331083162, -1313908385}, f_namelen=255, f_frsize=4096}) = 0 <0.000021>
18:39:00 inotify_init()                 = 4 <0.000033>
18:39:00 inotify_add_watch(4, "/var/log/messages", IN_MODIFY|IN_ATTRIB|IN_DELETE_SELF|IN_MOVE_SELF) = 1 <0.000041>
18:39:00 fstat64(3, {st_mode=S_IFREG|0640, st_size=92990, ...}) = 0 <0.000019>
18:39:00 read(4,

The -t switches on the time and -T switches on time spent in calls.

Hit return 4 or 5 times to make a bit of vertical space, then wait for it to stop tailing. Hopefully there will be some clues in the output.

gm3dmo
  • 9,632
  • 1
  • 40
  • 35
4

Given that both problematic logfiles are written by different components of the same application, I wonder if it's not some part of the logging code for that application that's causing the problem. I propose two tests to get a better idea of what's going on:

  • Note the inode of the logfile (ls -i logfile) prior to starting the tail, and once the tail fails, check it again. If the inode has changed, then the logger is rewriting the entire logfile, which would break tail's connection.

  • Note the last line before tail stops working, and then visit the file and find the first log entry after that line. Do this 3-5 times if possible. If it's a problem with the program doing the logging, the part of the program that wrote the log entry immediately after you see tail break is most likely responsible. If that log entry is always the same, or if it comes from the same component of the program, you may have enough data to submit a problem report to the application vendor.

Good luck.

daveadams
  • 1,269
  • 6
  • 12
3

Having the same problem here.

Problem was, that the file i'm watching was mounted from a different machine. The change notify was not propagated through the mount.

Solution was to use tail on the original machine.

Stephan Hoyer
  • 31
  • 1