Removing lines in text document that end the same

Question

We have a script that downloads a list of domains for our squid box to block, but we keep getting warnings like the following:

2015/03/02 17:08:47| WARNING: You should probably remove '.artnau.com' from the ACL named 'chat_domains'
2015/03/02 17:08:47| WARNING: '.artnau.com' is a subdomain of '.css.artnau.com'
2015/03/02 17:08:47| WARNING: because of this '.css.artnau.com' is ignored to keep splay tree searching predictable
2015/03/02 17:08:47| WARNING: You should probably remove '.artnau.com' from the ACL named 'chat_domains'
2015/03/02 17:08:47| WARNING: '.chatserve.com' is a subdomain of '.eagles.chatserve.com'
2015/03/02 17:08:47| WARNING: because of this '.eagles.chatserve.com' is ignored to keep splay tree searching predictable

Is there any way to go through the file, and remove the subdomains from existing domains in the list?

So grab the first line and check to see if there are any other lines in the text that end with that text and remove it?

Answer 1

Here is how you would do it in Perl:

$ cat a.txt
.artnau.com
.bar.foo.example.org
.chatserve.com
.css.artnau.com
.eagles.chatserve.com
.example.com
.foo.example.org
$ cat a.txt | perl -ne 'BEGIN { my %h; } $h{$_} = ""; END { foreach (keys %h) { $orig = $_; $_ =~ s/^\..*?\./\./; print $orig if not exists $h{$_} } }' | sort > b.txt
$ cat b.txt 
.artnau.com
.chatserve.com
.example.com
.foo.example.org

The Perl one-liner loops through a.txt and adding every line to a hash named %h. After every line of the file is added, it goes through every key in the hash, removes the first part of the domain (the first period until the second period), and if the resulting string is not in the hash, prints it out. The output is then piped through sort (you can probably guess what that does) and saved to b.txt.