List top 404 URLs in Apache access_log

Question

I'm looking for a command to list the 10 or 20 top 404 error URLs out of my apache access_log. Can anyone help me with this?

thanks

score 8 · Answer 1 · edited Jan 29 '18 at 09:20

8

Assuming the normal access_log format this should do it:

cat access_log | awk '{ if($9 == 404) { print $7 } }' | sort | uniq -c | sort -nr | head -10

edited Jan 29 '18 at 09:20

Sven

97,248
13
177
225

answered Feb 03 '11 at 16:15

carson

1,620
11
15

I hadn't thought of the field math, good call. I edited to fix the error code. – Scott Pack Feb 03 '11 at 16:16
@packs That was fast, I just noticed the issue when I double checked it again and you got to it before I could. :) – carson Feb 03 '11 at 16:19
Ninja edit! – Scott Pack Feb 03 '11 at 16:23
3

`sort -r` sorts alphabetically in reverse, so it will put "1", "10" and "100" after "2". You should use numeric reverse sort: `sort -nr`. – Christian Davén Oct 30 '12 at 09:26
Per Christian Davén, sort -nr should be fixed in the answer, I had all my urls with a count of 9 up top rather then the ones with 1000's – R. S. Feb 12 '13 at 01:45
1

There is no need to use cat and pipe the output to awk. awk can read files itself ;) `awk '{ if($9 == 404) { print $7 } }' access_log | sort | uniq -c | sort -nr | head -10` – Lukas Jul 26 '13 at 13:36

score 3 · Answer 2 · answered Feb 03 '11 at 16:16

I've done this using standard unix utilities, awk, sort, etc, and it works fairly well. Since the format of your logs could be different, you may have to change some things to work in your environment, but the basic command would be this:

cat access_log | awk '/" 404 / {print $7}' | sort | uniq -c | sort -n | tail -n10

If you are not familiar with awk, what we are doing is:

for each line
   if it contains the string '" 404', then
     print the 7th field (where fields are space delimited)

If you are using a custom apache log format, then you will need to change the $7 to match the field for the URL in the GET request. You can also change the number of lines from the tail command to display more, or rewer, results.

List top 404 URLs in Apache access_log

2 Answers2