Monitoring HTTP traffic using tcpdump

Question

To monitor HTTP traffic between a server and a web server, I'm currently using tcpdump. This works fine, but I'd like to get rid of some superfluous data in the output (I know about tcpflow and wireshark, but they're not readily available in my environment).

From the tcpdump man page:

To print all IPv4 HTTP packets to and from port 80, i.e. print only packets that contain data, not, for example, SYN and FIN packets and ACK-only packets.

tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

This command

sudo tcpdump -A 'src example.com and tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

provides the following output:

19:44:03.529413 IP 192.0.32.10.http > 10.0.1.6.52369: Flags [P.], seq 918827135:918827862, ack 351213824, win 4316, options [nop,nop,TS val 4093273405 ecr 869959372], length 727

E.....@....... ....P..6.0.........D...... __..e=3...__HTTP/1.1 200 OK Server: Apache/2.2.3 (Red Hat) Content-Type: text/html; charset=UTF-8 Date: Sat, 14 Nov 2009 18:35:22 GMT Age: 7149
Content-Length: 438

<HTML> <HEAD> <TITLE>Example Web Page</TITLE> </HEAD> <body>
<p>You have reached this web page ...</p> </BODY> </HTML>

This is nearly perfect, except for the highlighted part. What is this, end -- more importantly -- how do I get rid of it? Maybe it's just a little tweak to the expression at the end of the command?

score 39 · Accepted Answer · answered Nov 14 '09 at 21:29

39

tcpdump prints complete packets. "Garbage" you see are actually TCP package headers.

you can certainly massage the output with i.e. a perl script, but why not use tshark, the textual version of wireshark instead?

tshark 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

it takes the same arguments as tcpdump (same library) but since its an analyzer it can do deep packet inspection so you can refine your filters even more, i.e.

tshark 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -R'http.request.method == "GET" || http.request.method == "HEAD"'

answered Nov 14 '09 at 21:29

Aleksandar Ivanisevic

3,327
19
24

1

Thanks -- after trying out all the suggestions, tshark seems like the best tool for the job. I'm currently using "tshark -d tcp.port==8070,http -R 'http.request or http.response'". Now if only I could get tshark to "follow the tcp stream" just like wireshark can (This gets asked a lot, but I still haven't found the answer). "-V" displays info about the TCP and IP packets and so on, which I'm not interested in. But I guess I can remove that using a script. – otto.poellath Nov 18 '09 at 15:39
4

You can also search for "GET" in a capture filter by matching the ASCII values for each character: `tcp port 80 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420`. I added a page to the Wireshark web site a while back that helps you create string matching capture filters: http://www.wireshark.org/tools/string-cf.html – Gerald Combs Sep 13 '11 at 20:03

score 18 · Answer 2 · answered Nov 14 '09 at 21:56

18

take a look at ngrep - it mighe be of some use for you.

as reference for others httpry [ server seems to be down now but i hope it's temporary ] and tshark are also useful for passive protocol analysis - first one just for http, second - for much more.

answered Nov 14 '09 at 21:56

pQd

29,561
5
64
106

2

+1 for ngrep. Very useful tool. – tylerl Nov 15 '09 at 06:39
ngrep and httpry look definitely useful, but ngrep is unaware of HTTP and AFAIK, httpry only works with HTTP headers and cannot display the payload data. – otto.poellath Nov 18 '09 at 15:41
@sapporo - sure - ngrep is protocol agnostic, but you can add pcap filter "port 80" and get.. in [most cases] http traffic only. – pQd Nov 18 '09 at 16:31
Imo, ngrep is 100000x better than tcpdump – Daniel W. Apr 07 '16 at 14:53

score 5 · Answer 3 · answered Sep 13 '11 at 18:22

5

Try httpry or justniffer

Justniffer works well on tcp packets reordering retrasmissions and ip fragmentation

answered Sep 13 '11 at 18:22

Mole24

51
1
1

1

Thanks! justniffer is what I was looking for (I think it's the only tool mentioned that is very simple *and* measures request duration). – gkop Dec 08 '11 at 21:24

score 1 · Answer 4 · answered Nov 15 '09 at 03:24

I would suggest using a dumbed down tcpdump command line that stores everything in a pcap file for post process. Depending on what exactly you are looking at diagnosing tcpflow works great for putting communications back together in a coherent way for analysis.

Some other good information including some usages for httpry can be found at: http://taosecurity.blogspot.com/2008/06/logging-web-traffic-with-httpry.html

score 0 · Answer 5 · answered Jul 03 '10 at 07:46

0

There are several tools available on the market designed specially for monitoring the HTTP Traffic. Fiddler2 (http://www.fiddler2.org) and HTTP Debugger Pro are examples of such tools.

answered Jul 03 '10 at 07:46

4

Thanks for your suggestions. Unfortunately, both tools seem to work on Windows only. I hadn't mentioned it, but I'm loooking for something that works on Linux. – otto.poellath Jul 21 '10 at 09:30

score 0 · Answer 6 · answered Nov 14 '09 at 21:31

0

Does the webserver you're using not produce logs? Surely that'd be a much better way to monitor HTTP traffic, there's a plethora of tools to analyse the data and any competent webserver should produce reliable logs.

answered Nov 14 '09 at 21:31

JamesHannah

1,731
2
11
22

4

Clearly. I imagine he already thought of that, though. Sometimes it's useful to monitor the actual data being sent between point A and point B. – tylerl Nov 15 '09 at 06:37

score 0 · Answer 7 · edited Jan 22 '20 at 14:00

0

Your "issue" while using the TCPDUMP is that it is showing you the content of the header of the package.

If you still want to use TCPDUMP just remove the -A. Doing this should remove that from the output.

sudo tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

edited Jan 22 '20 at 14:00

kenlukas

2,886
2
14
25

answered Jan 22 '20 at 11:41

BANJOSA

350
1
3
15

Monitoring HTTP traffic using tcpdump

7 Answers7