monitor traffic at the scale of millisecond

Question

I have several linux servers running Ubuntu 14.04, and turned them into a Spark cluster. At this point, I want to measure the network flow between the servers, at the scale of milliseconds, in order to perform some inference and analysis task, for research purposes. I have tried the following without success:

1) I repeatedly call "iptables -nvx -L" from a script and grab its output, which contains the information I wanted (number of packets and bytes). However, calling it takes longer than 1ms, as I implemented in both Python (using subprocess module) and Bash scripts.

Python:

args = ['iptables',' -nvx', '-L', 'Log']

raw_traffic = sp.Popen('iptables -nvx -L Log', shell = True, stdout = sp.PIPE).stdout

(The above code takes 1.2ms to run)

Bash:

start_time=$(date +%s%N)

std_output=$(iptables -nvx -L Log)

echo "(($(date +%s%N)-$start_time)/1000000)" | bc -l

with the output:

$ sudo sh foo.sh

1.17057700000000000000

2) I tried to ask iptables to log every packet from the IP addresses of the servers. It certainly serves the purpose, as it measures everything in finest detail. However, it slows down the traffic and creates a large overhead. The links between the servers are 10G so at full speed around 1M lines of logs are generated per second, per IP address, and this is not feasible. The information I need at this moment is the per-millisecond packets and bytes between the servers.

Are there any workarounds of monitoring at 1ms scale? I suppose it should be doable with iptables, with some hacks perhaps. Any suggestions are more than appreciated. Thanks in advance!

Answer 1

You can actually do this, but you'll need to use the NF_LOG iptables target and write a program that specifically covers your needs.

NF_LOG will send packets to a receiving application which (given how quickly you manage to do what you are doing) would count the packets are a per-millisecond level.

You can avoid writing out every packet doing this and truncating the data out to ensure you can attempt to get the 1 million pps you need.

The issue however, is no such receiving program exists -- you'd need to write one.

Having done something with NF_LOG before and your statement that LOG slows down the traffic I suspect its the writing of the packets to disk that probably is sowing things down.

I still suspect a million packets per second is still a high bar to achieve.

Another alternative (which isn't realtime) is to use tcpdump to capture the packets and as a post step later use a program to read the pcap file and make the appropriate millisecond calculations.

Answer 2

Regardless of how you go about processing this info, you may need to move to a lower level language like C for performance. You might even need to go multi-threaded.

I haven't tried it but the netfilter library libnetfilter_log looks promising. iptables is essentially the netfilter command line interface and netfilter is the packet filter in the Linux kernel. So this would be your direct library to the packet filter. Sounds efficient.

http://www.netfilter.org/projects/libnetfilter_log/index.html

What is libnetfilter_log?

libnetfilter_log is a userspace library providing interface to packets that have been logged by the kernel packet filter. It is is part of a system that deprecates the old syslog/dmesg based packet logging. This library has been previously known as libnfnetlink_log.

Main Features

receiving to-be-logged packets from the kernel nfnetlink_log subsystem

Here's some example code (example copied from netfilter site / not so bad right?)

00002 #include <stdio.h>
00003 #include <stdlib.h>
00004 #include <unistd.h>
00005 #include <netinet/in.h>
00006 
00007 #include <libnetfilter_log/libnetfilter_log.h>
00008 
00009 static int print_pkt(struct nflog_data *ldata)
00010 {
00011         struct nfulnl_msg_packet_hdr *ph = nflog_get_msg_packet_hdr(ldata);
00012         u_int32_t mark = nflog_get_nfmark(ldata);
00013         u_int32_t indev = nflog_get_indev(ldata);
00014         u_int32_t outdev = nflog_get_outdev(ldata);
00015         char *prefix = nflog_get_prefix(ldata);
00016         char *payload;
00017         int payload_len = nflog_get_payload(ldata, &payload);
00018         
00019         if (ph) {
00020                 printf("hw_protocol=0x%04x hook=%u ", 
00021                         ntohs(ph->hw_protocol), ph->hook);
00022         }
00023 
00024         printf("mark=%u ", mark);
00025 
00026         if (indev > 0)
00027                 printf("indev=%u ", indev);
00028 
00029         if (outdev > 0)
00030                 printf("outdev=%u ", outdev);
00031 
00032 
00033         if (prefix) {
00034                 printf("prefix=\"%s\" ", prefix);
00035         }
00036         if (payload_len >= 0)
00037                 printf("payload_len=%d ", payload_len);
00038 
00039         fputc('\n', stdout);
00040         return 0;
00041 }
00042 
00043 static int cb(struct nflog_g_handle *gh, struct nfgenmsg *nfmsg,
00044                 struct nflog_data *nfa, void *data)
00045 {
00046         print_pkt(nfa);
00047 }
00048 
00049 
00050 int main(int argc, char **argv)
00051 {
00052         struct nflog_handle *h;
00053         struct nflog_g_handle *qh;
00054         struct nflog_g_handle *qh100;
00055         int rv, fd;
00056         char buf[4096];
00057 
00058         h = nflog_open();
00059         if (!h) {
00060                 fprintf(stderr, "error during nflog_open()\n");
00061                 exit(1);
00062         }
00063 
00064         printf("unbinding existing nf_log handler for AF_INET (if any)\n");
00065         if (nflog_unbind_pf(h, AF_INET) < 0) {
00066                 fprintf(stderr, "error nflog_unbind_pf()\n");
00067                 exit(1);
00068         }
00069 
00070         printf("binding nfnetlink_log to AF_INET\n");
00071         if (nflog_bind_pf(h, AF_INET) < 0) {
00072                 fprintf(stderr, "error during nflog_bind_pf()\n");
00073                 exit(1);
00074         }
00075         printf("binding this socket to group 0\n");
00076         qh = nflog_bind_group(h, 0);
00077         if (!qh) {
00078                 fprintf(stderr, "no handle for grup 0\n");
00079                 exit(1);
00080         }
00081 
00082         printf("binding this socket to group 100\n");
00083         qh100 = nflog_bind_group(h, 100);
00084         if (!qh100) {
00085                 fprintf(stderr, "no handle for group 100\n");
00086                 exit(1);
00087         }
00088 
00089         printf("setting copy_packet mode\n");
00090         if (nflog_set_mode(qh, NFULNL_COPY_PACKET, 0xffff) < 0) {
00091                 fprintf(stderr, "can't set packet copy mode\n");
00092                 exit(1);
00093         }
00094 
00095         fd = nflog_fd(h);
00096 
00097         printf("registering callback for group 0\n");
00098         nflog_callback_register(qh, &cb, NULL);
00099 
00100         printf("going into main loop\n");
00101         while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
00102                 struct nlmsghdr *nlh;
00103                 printf("pkt received (len=%u)\n", rv);
00104 
00105                 /* handle messages in just-received packet */
00106                 nflog_handle_packet(h, buf, rv);
00107         }
00108 
00109         printf("unbinding from group 100\n");
00110         nflog_unbind_group(qh100);
00111         printf("unbinding from group 0\n");
00112         nflog_unbind_group(qh);
00113 
00114 #ifdef INSANE
00115         /* norally, applications SHOULD NOT issue this command,
00116          * since it detaches other programs/sockets from AF_INET, too ! */
00117         printf("unbinding from AF_INET\n");
00118         nflog_unbind_pf(h, AF_INET);
00119 #endif
00120 
00121         printf("closing handle\n");
00122         nflog_close(h);
00123 
00124         exit(0);
00125 }