Counting duplicates in a sorted sequence using command line tools

Question

I have a command (cmd1) that greps through a log file to filter out a set of numbers. The numbers are in random order, so I use sort -gr to get a reverse sorted list of numbers. There may be duplicates within this sorted list. I need to find the count for each unique number in that list.

For example, if the output of cmd1 is

100 100 100 99 99 26 25 24 24

I need another command that I can pipe the above output to, so that I get :

100 3 99 2 26 1 25 1 24 2

related: http://stackoverflow.com/questions/1092405/counting-duplicates-in-a-sorted-sequence-using-command-line-tools — David Cary, Jun 24 '12 at 17:24

score 15 · Accepted Answer · answered Jul 07 '09 at 13:45

15

If you can handle the output being in a slightly different format, you could do:

cmd1 | tr " " "\n" | uniq -c

You'd get back:

answered Jul 07 '09 at 13:45

Evan Anderson

141,071
19
191
328

1

+1 for simplicity. You could do one final pipe to "awk '{print $2 " "$1}'" to get the columns switched. – Geoff Fritz Jul 07 '09 at 14:12
1

and after doing that awk, you could do "| tr '\n' ' '" to get it all on one line – Amandasaurus Jul 07 '09 at 14:44
Or use printf to get it all on one line. – Dennis Williamson Jul 07 '09 at 19:02

score 1 · Answer 2 · answered Jul 07 '09 at 14:36

1

Also add in the -u switch. Thus you would have:

cmd1 | sort -gru

From the sort manpage:

-u, --unique
without -c, output only the first of an equal run

answered Jul 07 '09 at 14:36

Kevin M

2,302
1
16
21

But he wants the count. – Dennis Williamson Jul 07 '09 at 19:03
specifics, specifics :)
cmd1 | sort -gru | wc -l – Kevin M Jul 07 '09 at 21:38

score 0 · Answer 3 · answered Jul 07 '09 at 14:10

(I'm assuming your input is one number per line, as that's what sort would output.)

You could try awk:

<your_command> | awk '{numbers[$1]++} END {for (number in numbers) print number " " numbers[number]}'

This would give you an un-sorted list (the order in which arrays are walked through in awk are undefined, so far as I know), so you'd have to sort to your liking again.

score 0 · Answer 4 · answered Jul 07 '09 at 15:25

0

$ echo '100 100 100 99 99 26 25 24 24' | perl -e 'while (<>) { chomp; my %nums; foreach (split(/ /)) { $nums{$_} += 1; }; foreach (sort {$b <=> $a} keys %nums) { print "$_ $nums{$_} " }; print "\n"; }'
100 3 99 2 26 1 25 1 24 2

answered Jul 07 '09 at 15:25

towo

1,887
14
12

This would also work with multiple lines of input, sorting each line individually. – towo Jul 07 '09 at 15:26

Counting duplicates in a sorted sequence using command line tools

4 Answers4