grep and sed with pipe from tail -f appears to be caching

Question

I am trying to get a working system to monitor special logs. I usually just want a very specific pattern which I extract using grep and a pipe from tail -f. I have noticed that grep does not output everything but instead keeps some lines buffered. I guess that makes sense if you have a pipe that will output everything and then terminate and close the stream.
But with tail -f that doesn't work out for me.

The same problem appears with sed.

Here is an example command I want to use:

clear && tail -F -n1000 /var/log/fail2ban.log | grep 'WARNING.*Ban' | sed s/'fail2ban.actions: WARNING '//g | grep -E --color 'ssh-iptables-perma|$'

To provide an example:

The last line of the command above is this:

2015-05-04 11:17:24,551 [ssh-iptables] Ban x.x.x.x

And using this command:

clear && tail -F -n1000 /var/log/fail2ban.log | grep 'WARNING.*Ban' | sed s/'fail2ban.actions: WARNING '//g

The last line ist this:

2015-05-04 19:45:17,615 [ssh-iptables] Ban y.y.y.y

Removing further pipes gets me further to the most recent entries.

How can I possibly avoid this caching in the pipes?

Answer 1

Add grep's option --line-buffered and sed's option --unbuffered.

Answer 2

Use grep's --line-buffered option.

By default, utilities use line-buffering if their standard output is a terminal, but use bigger buffers (probably 4 or 8 kB) when their output is connected to a file descriptor or a pipe.

You're lucky that tail -F uses line-buffering by default, and grep has a command line option to enable that. I'm not aware of a generic method to enable line buffering for an arbitrary command.