Linux commands: uniq
A quick guide to the `uniq` command, used to work with duplicate records/lines in text
uniq
is a command useful to sort lines of text.
You can get those lines from a file, or using pipes from the output of another command:
uniq dogs.txt
ls | uniq
You need to consider this key thing: uniq
will only detect adjacent duplicate lines.
This implies that you will most likely use it along with sort
:
sort dogs.txt | uniq
The sort
command has its own way to remove duplicates with the -u
(unique) option. But uniq
has more power.
By default it removes duplicate lines:
You can tell it to only display duplicate lines, for example, with the -d
option:
sort dogs.txt | uniq -d
You can use the -u
option to only display non-duplicate lines:
You can count the occurrences of each line with the -c
option:
Use the special combination:
sort dogs.txt | uniq -c | sort -nr
to then sort those lines by most frequent:
The uniq
command works on Linux, macOS, WSL, and anywhere you have a UNIX environment
→ I wrote 17 books to help you become a better developer, download them all at $0 cost by joining my newsletter
→ JOIN MY CODING BOOTCAMP, an amazing cohort course that will be a huge step up in your coding career - covering React, Next.js - next edition February 2025