quote-enclosed fields in input to awk

Question

I'm trying to process a webserver logfile in awk. Like

123.222.333.444  - - [24/Feb/2015:13:09:19 +0100] "GET / HTTP/1.1" 200 15852 "https://www.google.dk/" "Mozilla/5.0 (Windows NT 6.1) ...."

• Field 1,2,3 are just separated by whitespace.
• Field 4, the date is enclosed in []
• Field 5 the url, is enclosed in ""
• Field 6,7 are just separated by whitespace. (Integers)
• Field 8,9 the referer and useragent is enclosed in ""

How can I parse these fields?

Answer 1

I would choose one "main" delimeter, and use split() on some of the fields:

awk -F '"' '
{
  http_cmd = $2
  url = $4
  ua = $6
  split($1, a, /[][]/); date = a[2]
  split(a[1], b, " "); ip = b[1]; f2 = b[2]; f3 = b[3]
  split($3, c, " "); code = c[1]; size = c[2]
  OFS = "\n"
  print ip, f2, f3, date, http_cmd, code, size, url, ua
}' << END
123.222.333.444  - - [24/Feb/2015:13:09:19 +0100] "GET / HTTP/1.1" 200 15852 "https://www.google.dk/" "Mozilla/5.0 (Windows NT 6.1) ...."
END

123.222.333.444
-
-
24/Feb/2015:13:09:19 +0100
GET / HTTP/1.1
200
15852
https://www.google.dk/
Mozilla/5.0 (Windows NT 6.1) ....

Answer 2

Another solution is to parse the log:

function consume(pat) {
    sub(/^[ \t]+/, "")                      # strip leading whitespace
    if (!match($0, pat)) return
    s = substr($0, p, RLENGTH)              # extract part matching regex pattern
    $0 = substr($0, RSTART + RLENGTH + 1)   # strip matched part
    return s
}
BEGIN {
    # regular expressions to match ...
    nonblank = "[^ \t]+"                    # a sequence of non-whitespace characters
    quoted_string = "\"[^\"]+\""            # a ""-quoted string
    bracketed_string = "\\[[^]]+\\]"        # a []-quoted string
}
{ print
  array["ip"] = consume(nonblank)
  array["identity"] = consume(nonblank)
  array["userid"] = consume(nonblank)
  array["time"] = consume(bracketed_string)
  array["request"] = consume(quoted_string)
  array["status"] = consume(nonblank)
  array["size"] = consume(nonblank)
  array["referer"] = consume(quoted_string)
  array["agent"] = consume(quoted_string)
  for (key in array) printf "  %10s: %s\n", key, array[key]
}

Note that embedded quotes will throw off the parser. There are other improvements to be made, but I hope you get the idea.

I think, though, that if you need more power, it would be better to check out dedicated log file parsers, or a more powerful scripting language. For instance, Perl has a module Text::Balanced that can extract quoted and bracketed strings, and generally does The Right Thing with embedded quotes and other difficulties. (BTW, a script in Perl using Text::Balanced would look very similar to the above!)