Counting by number of occurrences in first column

Question

I have an input file like this:

ATTACK-RESPONSES id check returned root
BACKDOOR ACKcmdC trojan scan
BACKDOOR hack-a-tack attempt 
BACKDOOR WinCrash 1.0 Server Active
ICMP Destination Unreachable Port Unreachable
ICMP Destination Unreachable Port Unreachable
ICMP Destination Unreachable Port Unreachable
SNMP trap tcp

Output:

1 ATTACK-RESPONSES id check returned root
3 BACKDOOR
3 ICMP Destination Unreachable Port Unreachable 
1 SNMP trap tcp 

I want to find and match the longest common substring in each line of the text and return the number of repetitions for each of them – so from input I'd need to get Output.

How could I do this?

Answer 1

This is rather difficult with a single pass, and even more difficult if you don't assume that the start needs to be the same.

You could write a perl script that matches regular expressions against previous lines, sort of like this:

my @words_on_line = split(/ /, $current_line);
my $i = 0; my $substring = ''; my $expression = '';
do {
  $expression = join(' ', $words_on_line[0..$i++]);
  if ($previous_line =~ m/^$expression/) {
    $substring = $expression;
  }
} until ($substring ne $expression);

Then, you'd also have to check the next line and potentiall reduce the substring match, e.g. when you have

FOO a b c
FOO a b
FOO d

The first match (from 2 to 1) would give you FOO a b, but comparing below, you'd only get FOO.

Which boils down to: you need to buffer your lines until you get a no-match line. So instead of printing, you'd do something like

unless ($substring) {
  push @buffer, $current_line;
  foreach (@buffer) {
    unless (m/$substring/) {
      $buffer_substring = $substring;
    }
  }
} else {
  print scalar @buffer, " $buffer_substring\n";
}

And then you'd just combine this.

If it's not "first common starting from beginning of line", you'd have to check every possible of sequence of words against every possible sequence of words in other lines, which is utterly complicated and which I will not reproduce here.