0

I was curious on how I would go about calculating a running percentage for specific URLs... For example, when varnishlog is executed you get a realtime output for information of incoming http requests.

Below an excerpt with some information cut out.

  46 ReqStart     c 
   46 RxRequest    c GET
   46 RxURL        c /cat?getParam
   46 RxProtocol   c HTTP/1.1
   46 RxHeader     c User-Agent: **************
   46 RxHeader     c Accept-Encoding: gzip
   46 RxHeader     c Host: myServer.org
   46 RxHeader     c Connection: Keep-Alive
   46 RxHeader     c Cookie: ********
   46 RxHeader     c Cookie2: ************
   46 VCL_call     c recv lookup
   46 VCL_call     c hash
   46 Hash         c /cat?getParam
   46 Hash         c myServer.org
   46 VCL_return   c hash
   46 VCL_call     c miss fetch
   46 Backend      c 90 default default
   ...
   46 ObjHeader    c Server: Apache-Coyote/1.1
   46 ObjHeader    c Content-Type: text/xml
   46 ObjHeader    c Content-Encoding: gzip
   46 ObjHeader    c Vary: Accept-Encoding
   46 ObjHeader    c Date: Thu, 05 Sep 2013 12:56:04 GMT
   46 Gzip         c u F - 1596 8098 80 80 12697
   46 VCL_call     c deliver deliver
   46 TxProtocol   c HTTP/1.1
   46 TxStatus     c 200
   46 TxResponse   c OK
   ...
   46 ReqEnd       c ...

What I am interested is RxURL. Specifically, I have requests starting with:

  • /cat?
  • /template?

How would you go about running a command that can count:

  • total number of requests
  • number of requests starting with /cat
  • number of requests starting with /temp

and output these numbers at the end?

Currently, I am executing varnishlog | grep "Hash" > requests.txt ,

And then utilizing an excel spreadsheet to clean up, and make my counts and calculate percentage.

I was curious though if AWK or a script could be utilized to do this directly on the server.

Update

varnishtop -i txurl is even more suited as it displays the URLS directly.

Menelaos
  • 155
  • 1
  • 1
  • 9

1 Answers1

0

You can use Varnishncsa[1] to produce an "Apache like" log (and even better, a filtered one if you want) and take advantage of some battle tested scripts[2][3]

Obtaining the counts you want realtime (for the next 100 request, by example) is trivial:

  • varnishncsa | head -n 100 | awk '{print $7}' | wc -l
  • varnishncsa | head -n 100 | awk '{print $7}' | grep /cat | wc -l
  • varnishncsa | head -n 100 | awk '{print $7}' | grep /temp | wc -l

[1] https://www.varnish-cache.org/docs/trunk/reference/varnishncsa.html

[2] Do you have any useful awk and grep scripts for parsing apache logs?

[3] http://www.the-art-of-web.com/system/logs/#.Uin5AEBdXsh

Community
  • 1
NITEMAN
  • 314
  • 1
  • 5