Counting IP Addresses in a Log File

October 11, 2019
linux

I've been using grep to search through files on linux / mac for years, but one flag I didn't use much until recently is the -o flag. This tells grep to only output the matched pattern (instead of lines that mach the pattern).

This feature turns out to be pretty handy, lets say you want to find all the IP addresses in a file. You just need to come up with a regular expression to match an IP, I'll use this: "[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+" it's not perfect, but it will work.

grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+" httpd.log

What if I want to see just unique IPs

We can use the uniq command to remove duplicate ip addresses, but uniq needs a sorted input. We can do that with the sort command, like so:

grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+" httpd.log | sort | uniq

Show me the number of times each IP shows up in the log

Now we can use the -c flag for uniq to display counts:

grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+" httpd.log | sort | uniq -c

This will output something like:

    7 10.0.0.30
    1 10.0.0.80
    3 10.0.0.70

The counts are not in order, so we can pass our results through sort again, this time with the -n flag to use a numeric sort.

grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+" httpd.log | sort | uniq -c | sort -n

The above will put them in order from least to greatest, you can pipe the result to tail if you only want to see the top N IP addresses!

Pretty handy right?


Like this? Follow me ↯


You might also like:

This entry was:

Comments

Man -- this works, even on my Windows server. (I think I have some form of Unix tools installed.) With Windows, though, the output includes the filename. Not too problematic. But I wish I were a better dark-arts-regex wizard, like you. I often scan my SSH server logs for hacking attempts (there are many!) and manually block the IP addresses at the firewall. Unfortunately, the *reason* associated with the IP address in on the NEXT line (I_LOGON_AUTH_FAILED), therefore the regex doesn't quite work for me. (It's all XML.) But this is a great use of grep! Thanks for the post.
Cool, sort of similar to what I use to get the same results parsed into a separate file. cat access.log | cut -d" " -f9 | sort | uniq -c | sort -rn > output.log

Post a Comment




  






Foundeo Inc.