How do I delete BOTH lines of a duplicate pair in a text file in bash?

1

I want to delete both lines of a duplicate pair in a text file.

There is no way to do this using sort -u file or awk '!a[$0]++' file which both delete duplicated lines.

Somehow I would have to capture the deleted lines and run sed to use this list and then delete the already deleted (is there some way to get this output?). The pattern of the duplicated lines is not predictable.

I am looking for a shorter way than writing a complete bash program. It seems like a useful tool and there should be an easy way to do it.

Easy to delete one of the duplicates. I need to delete both and can't find a way to do it. Encountered during parsing.

user322035

Posted 2019-12-09T21:40:04.727

Reputation: 21

1

Please note that https://superuser.com is not a free script/code writing service. If you tell us what you have tried so far (include the scripts/code you are already using) and where you are stuck then we can try to help with specific problems. You should also read How do I ask a good question?.

– DavidPostill – 2019-12-09T21:41:19.767

Is it necessary to maintain the order of the lines? – dirdi – 2019-12-09T21:42:03.833

Yes I need to maintain order – user322035 – 2019-12-09T21:48:29.990

How large is the file; does it fit in memory? – user1686 – 2019-12-09T21:58:50.077

Not a large file. 10-200 lines of ascii text – user322035 – 2019-12-09T21:59:36.210

I voted to reopen. If the file is not very large then something like awk '{a[$0]++; 1} END {for (line in a) if (a[line] == 1) print line}' may be a good start. – Kamil Maciorowski – 2019-12-10T10:03:45.760

Thanks... redid script so I don't need to do this! Found a better way. – user322035 – 2019-12-10T15:04:10.450

Answers

0

You can use the -u command-line argument to the uniq utility, which does precisely what you want:

−u   Suppress the writing of lines
     that are repeated in the
     input.

You still need to sort the input, of course:

sort file | uniq -u

rici

Posted 2019-12-09T21:40:04.727

Reputation: 3 493

Asked: 2019-12-09T21:40:04.727

Viewed: 67 times

Active: 2019-12-10T19:27:03.823