Learn how to use the uniq command in Linux with this easy, practical Linux Crash Course tutorial! In this video, I break down what uniq does, how it works, and the must-know tips that help you avoid common mistakes when filtering duplicate lines in files or log output.

Whether you’re a Linux beginner or a seasoned sysadmin, this tutorial gives you clear, real-world examples of how to use uniq to clean up data, analyze logs, and streamline your command-line workflow.

Basic Examples

First, we’re going to start with some very basic examples so you can see exactly how the uniq command works. In the next section, I’ll give you some helpful real-world examples you can actually use, but before that we should understand the basic usage of this command to see how it works.

Like I mentioned during the intro, the uniq command is primarily used to remove duplicated lines. To see it in action, consider the following command:

echo -e "Cinnamon\nCOSMIC\nCOSMIC\nGNOME\nPlasma\nMATE\nMATE\nxfce"

With this command, I’m simply printing a list of some of my favorite desktop environments. If you’re curious, the -e option of echo enables it to parse backslash escapes, which is how I was able to use \n to add a new line after each item.

As you can see, there’s some duplicated entries here. And sure, I just added those on purpose, but again – we’ll see more useful examples later. Anyway, watch what happens when I pipe the output into the uniq command:

echo -e "Cinnamon\nCOSMIC\nCOSMIC\nGNOME\nPlasma\nMATE\nMATE\nxfce" | uniq

As you can see, all the duplicate lines are gone. And even though this example was a bit contrived, the most common usage of the uniq command is using it with other commands. When we use echo to print the list, the list contains duplicates – but then once the output is passed over to uniq, the duplicates are gone.

Before we go any further though, there’s a common “gotcha” with the uniq command that may confuse some people at first. Consider this example:

echo -e "Cinnamon\nCOSMIC\nMATE\nGNOME\nCOSMIC\nPlasma\nMATE\nxfce" | uniq

In this case, it doesn’t appear as though the uniq command is working at all. The duplicated entries are still present, even after piping the echo command into uniq. So why is it that the first example worked, but the second one didn’t?

The reason is due to an important distinction – the uniq command removes sequential duplicates. In the example I just gave you, all I did was change the order. However, this situation is pretty simple to fix – we can sort the entries first and then pipe the output into uniq:

echo -e "Cinnamon\nCOSMIC\nMATE\nGNOME\nCOSMIC\nPlasma\nMATE\nxfce" | sort | uniq

With this example, after the echo command runs we’re passing it into the sort command, which will reorder the list. We can see that right away if we run the command again without uniq:

echo -e "Cinnamon\nCOSMIC\nMATE\nGNOME\nCOSMIC\nPlasma\nMATE\nxfce" | sort

As you can see, even though I used a random order for the list, the sort command reordered each line alphabetically. At this point, all of the duplicated lines are sequential, which means the uniq command is able to then remove the duplicated lines.

However, most of the time you’ll use the uniq command against files, instead of something like an echo statement. After all, I could fix my earlier examples a lot easier by simply not typing duplicates. Files, on the other hand, will often have duplicated lines – especially log files. We’ll look at log files later, but for now I’ve prepared a couple of text files that I’ll use for a few more quick examples.

First, I have a file saved that contains a list of some of my favorite Linux distributions:

cat linux_distros_list_1.list

Of course, there’s some duplicated lines. But what we can do is pipe the output of the cat command directly into uniq:

cat linux_distros_list_1.list | uniq

As you can expect, it’s pretty much the same thing – we have a list with duplicates and we’re cleaning up the output.

The second example is the same as the first, but I’ve rearranged it a bit:

cat linux_distros_list_2.list

Now, the duplicates are no longer sequential. And, as you already know from one of my earlier examples, uniq won’t do anything unless the duplicated lines are sequential:

cat linux_distros_list_2.list | uniq

But similar to our earlier example, I can also include the sort command into the chain in order to group everything together, which will make the uniq command able to do its job:

cat linux_distros_list_2.list | sort | uniq

Again, there’s really not all that much new with my last few examples, but since most often the uniq command is used with files, I wanted you to see that.

We’re going to see more examples throughout the article and also some common command-line options we can use. But I think it would be more fun to do so while looking through some real-world examples, so that way you can get a better idea of how this command can actually help you while you manage Linux systems.

Real-world Example

So, what I’m going to do is log in to the actual server that’s responsible for this channel’s web site. Since you won’t have access to my server, you won’t be able to follow along with me exactly – but you can use any file you may have access to while we explore the uniq command. If you need a bit of inspiration for which file to use as an example, consider log files – in fact, using the uniq command with system logs is a very common use-case.

On my end, I’m using OpenLiteSpeed as the webserver for my site, so what I’ll do is navigate to the directory where these log files can be found:

cd /usr/local/lsws/logs

Inside this directory, we have my access log, which shows all the connection attempts made to my server. Unfortunately, I can’t actually show you the contents of this file for privacy reasons, but what I can do is tell you how many lines are currently inside this file:

cat access.log | wc -l

In this case, I’m printing the contents of the access log – but instead of printing the contents on screen, I’m piping the output into the wc command. The wc command gives us a count of how many words are in a file – and if we use the -l option, it’ll return the total number of lines.

Now, watch what happens when we add the uniq command to the chain:

cat access.log | uniq | wc -l

In this case, the total number of lines is quite a bit lower, it’s down by almost half. And that’s without even sorting the output, which means this file has a large number of sequential duplicates. But already we can see a really good real-world use-case for the uniq command. If you’re working with log files, maybe because you’re troubleshooting something, you may end up in a situation where you have the same error message repeated multiple times. In order to make it easier to go through the file, removing duplicates will give you fewer lines to go through. Even better, you can create a new copy of the log files with the duplicated lines removed:

cat access.log | uniq > ~/access.log-deduplicated

Now that I’ve made a copy of my access log, if I wanted to I could go through the copied file as I troubleshoot – which will not only have fewer lines, but my copy will persist even if the source log file was rotated – and it’s good to have a backup.

Now let’s see another example, and this will be a great opportunity to start showing you some of the options you can use with the uniq command. We’re going to start with the -c option, which will give you a count for how many times a particular item shows up within a text file.

For this example, I made a backup copy of my access log – and I went in and changed the IP addresses that it contains with randomly generated IP’s, so that way I can show you the content of this file without actually exposing any private information. Anyway, what we’re going to do in particular is pull all the IP addresses from the file, and see a count of how many times each IP address has accessed the server.

What I’ll do is type out the command, and then I’ll explain exactly what it does:

cut -d ' ' -f1 access.log.changed | sort | uniq -c

Here, we have another command string. And again, this is common with the uniq command – since it’s most often used with other commands. Here, we’re using the cut command, which is something I’ve already covered earlier in the Linux Crash Course series. In this case, we’re telling the cut command to use space as a delimiter between fields, and we’re specifically targeting the first field – which is the IP address. The rest of each line will be truncated after that.

Then, I’m piping the output of the cut command into the sort command, which is going to end up printing the IP addresses in order. Remember, the uniq command only works with sequential duplicate lines, so we’re using sort to facilitate that requirement. After that, we’re piping the output of that into the uniq command, and we’re using the -c option this time which will give us a count for how many times each item shows up. Putting it all together, we’re stripping the access log down to just the IP addresses it contains, then we’re sorting the output, and then finally having the uniq command tally up the totals for us.

In the real-world, this might be useful in a situation in which you think you may have a user that’s hammering your web server repeatedly. Regular visitors to my site will only visit it a few times a day on average, but anyone that’s trying to do anything malicious will probably attempt over and over. If I wanted to add a firewall rule to block an IP address that’s attempting to brute force my server, the output of this command should make it fairly obvious which IP’s are trying to do that.

In fact, to make this even better, let’s adjust the command a bit:

cut -d ' ' -f1 access.log.changed | sort | uniq -c | sort -nr | head

In this case, I decided to add a few more commands into the chain. After the uniq -c command (which is where the previous example ended) I’m piping the output into the sort command again – but this time I included the -n option, which enables numerical sorting, and the -r option reverses the order – which means the highest counts will be near the top, rather than at the bottom. Finally, I’m piping the output again – this time into the head command which will show the first ten lines by default. The end result is that the command will give me the top ten IP’s with the most attempts to access my server.

Additional Options with the uniq command

Before I end the article, I wanted to go over some additional options you can use with the uniq command. To keep the remaining examples simple, I’m going to return to using my distro list as the example file.

As a refresher, my distro list contains the following items:

cat linux_distros_1.list

Now what I want to do is show you the -d option with uniq. What this will do is show us only the lines that were duplicated:

uniq -d distro-list-file

As you can see, it gives me three lines, which are specifically the lines that appear more than once in the file. It’s a simple example, but if you wanted to see only the duplicated lines, that’s how you do it.

If you want to do the exact opposite of that, try the -u option:

uniq -u linux_distros_1.list

As you can see, this time we see the non-duplicated lines instead.

Another option you may want to know about is -i. The -i option isn’t something you’ll use all that often – but you never know, it may come in handy some day. Basically what it does is it makes the uniq command case insensitive, so capitalization won’t matter anymore when it looks for duplicates. Normally, repeated messages will have the same case, so you might not use that much, but it may be a good idea to include it in your notes if a need for this ever comes up.

Anyway, that’s about it when it comes to the uniq command. It’s very simple to use by itself, but its true power comes from using it alongside other commands. In this article, we discussed several examples of it in action – and there’s a few more options available if you check uniq’s man page. But for the most part, the examples we’ve gone over in this article should be the majority of what you’ll ever need.