Special file that keeps only the last n lines

Is there a way to create special files under Linux that would keep only, say the 100 last written lines? I have a process filling a log file, and I'd like to regularly parse its 100 last lines.

I know I could use some kind of logrotate, but is there a way to create a special file that would fill up till it reaches 100 lines, then, adding a line removes the oldest one, so that the file only keeps 100 lines? (a kind of line-based FIFO)

Thanks a lot

Daladim

Posted 2016-02-15T22:45:24.017

Reputation: 1

2tail does that, so you could write a cron job to fill the file on whatever refresh interval you want. just overwrite with tail <file> > <targetfile>. if your refresh rate is high enough, it will act like a FIFO buffer. – Frank Thomas – 2016-02-15T22:51:22.707

While it would be cool to have a custom virtual file system that you can cat to get derived content (like a ring buffer), these aren't very popular. They are called log-structured file systems. It would be perfect for that scenario. https://en.wikipedia.org/wiki/Log-structured_file_system . For now, I suggest you just write a script that does a tail -n 100 to itself.

– David Betz – 2016-02-16T00:31:34.840

Thanks, I was not aware of such filesystems! The ideal would have been special files that act like such ring buffers (that would avoid needing to install and mount a whole FS, and I thought mainstream Linux distros already supported this). I'll definitely end up using tail, but I'm happy I have learnt something :) – Daladim – 2016-02-16T21:49:47.473

In general there is a way to create files that are just what you want them to be: custom FUSE (Filesystem in Userspace). It is not as easy as you probably hoped for, but it's possible. – Kamil Maciorowski – 2016-02-22T09:48:42.303

Answers

Logs are text(like) files and being that kind, appending new disk blocks to them when it's demanded by new lines is a quick action well supported by any file system.

However, constantly dropping out the first line when a new is coming would mean reorganizing at least some if not all the blocks of the file CONSTANTLY. That would mean big overhead dedicated to logging while one of the most important characteristics we want from logging is that it be lightweight .

File systems are not prepared for this (at least I haven't heard about this type), that's why logrotate/tail/database-backed logging are used where the last records are of importance.

Gombai Sándor

Posted 2016-02-15T22:45:24.017

Reputation: 3 325

Thanks for this low-level hint I had not thought about :) – Daladim – 2016-02-16T21:53:14.733

Presuming that you have a script that processes only the last 100 lines each time it is run, these lines are best captured with the tail command, as it does pretty much what you want to do. The key here is the -n switch which dictates how many lines it should capture, starting from the end.

You can incoprorate tail -n 100 somefile.log in your script directly, or you can periodically run tail -n 100 somefile.log > onlylast100lines.log to create a file with only the last 100 lines. The latter approach will rewrite the target file every time, so no need to delete it between each run.

Jarmund

Posted 2016-02-15T22:45:24.017

Reputation: 5 155

I was aware of tail (and I'll finally end up using it), but out of curiosity, I wanted to know whether there were Linux features I did not know that would do he job. Thanks anyway. – Daladim – 2016-02-16T21:51:14.707

Elaborating on Jarmunds method, you could make a bash script like this:

while true; do 
    tail -n 100 somefile.log > onlylast100lines.log
    sleep 5
done

this would parse the last 100 lines of somefile.log into onlylast100lines.log every 5 seconds. The file would be overwritten each time, so it would be always only the last 100 lines. This could be added to your login script or to whatever runlevel you want this to be executed.

Mikkel Bue Tellus

Posted 2016-02-15T22:45:24.017

Reputation: 21