2

I have two machines, A and B, running RHEL Linux. Each machine has a 1Gig network card connected to what I call the "LAN" network. Each machine has a 10Gig network card that is connected to what I call the "BIGDATA" network. These networks are not connected to each other, except through these machines.

I have fiber taps on the 10Gig interface of A on the BIGDATA network. The taps lead off to a capture server, where I want to record all the data going in and out. Initially all TCP, but perhaps some UDP would be nice in the future.

For development and testing purposes, I want to run both sides of a TCP conversation on A and yet capture the conversation with those taps. The developers who will be using this setup won't have accounts on B (at least that's my goal).

In my mind, it should be possible for a superuser to set up some sort of forwarding/relay on B, that would allow a client on A to connect to a server on A, but over the tapped fiber. I.e traffic goes from A to B on the LAN network, then back to A on the BIGDATA network. Return traffic takes the same path back.

My attempt so far has been to use ssh tunneling. Like A:~ $ssh B -L 8051:<A's-BIGDATA-IP>:3434 -N. That actually worked great for capturing traffic from the client to the server. But unfortunately the server's responses are finding the shortcut of just staying local, and never appear on the fiber. Looking at wireshark, the packets know their destination IP, and don't feel obligated to take the tunnel back.

I found this answer, but I believe all the solutions require both interfaces to be on the same network. Maybe the seeds of my solution are in there, but I can't find them.

I think I could write this from scratch in C, making a program to run on B to take TCP connections on the LAN interface, initiate another TCP connection on the BIGDATA interface, back to A, and forward the replies as well. It would only copy the TCP payload, which would be sufficient for my purposes.

Is there a cool tool that does this forwarding/relay already?

Is there a way to make ssh tunneling work the way I want for the return traffic?

Community
  • 1
RaveTheTadpole
  • 186
  • 1
  • 7

3 Answers3

1

Have a look at EtherPuppet, possibly in combination with Scapy; EtherPuppet is described as

a small program for Linux that will create a voodoo doll for an Ethernet interface.

...

everything seen by the real interface will be seen by the virtual one. Everything that will be sent to the virtual interface will be emitted by the real one.

Scapy is described as

a powerful interactive packet manipulation program. It is able to forge or decode packets of a wide number of protocols, send them on the wire, capture them, match requests and replies, and much more.

Community
  • 1
Andrew
  • 7,772
  • 3
  • 34
  • 43
  • Thanks for the idea. I hadn't heard of EtherPuppet before. From my reading, it would allow me to write to a fake interface on A, and the packets would actually be emitted from the BIGDATA interface on B. That might work. Unfortunately, in my particular circumstances, that BIGDATA interface on B is also being used for some other important traffic, and I don't want to interfere with it. It looks like Scapy is like the Pcapy+Impacket combo that I've been using to process the captured data. – RaveTheTadpole Aug 18 '13 at 16:58
1

Source and Destination NAT

You can forward the traffic out and back with a combination of DNAT and SNAT using the same idea as the answer you found as long as you don't mind mangling the packets you're capturing. The captures will have a destination or src IP of B but it sounds like you have already done that with your forwarding client anyway. The advantage to NAT would be speed I guess, everything stays in kernel space.

This example is a client on A connecting to a server on A:BIGDATA:3434. NAT rules forward the connection from A out to B, then B forwards back to A, while setting the source as B to set the return path. Probably have a look at the netfilter guide if you are unfamiliar with NAT tables.

  • Make sure IP forwarding is enabled on B with either:

    sysctl -w net.ipv4.ip_forward=1
echo 1 > /proc/sys/net/ipv4/ip_forward

  • traffic goes from A to B on the LAN network

    # On A, forward local traffic for local service out LAN to B.
iptables -t nat -A OUTPUT -p tcp -d A:BIGDATA --dport 3434 -j DNAT --to-destination B:LAN

  • then back to A on the BIGDATA network. 

    # On B, forward the traffic to back to A over fibre
iptables -t nat -A PREROUTING -p tcp -d B:LAN --dport 3434 -j DNAT --to-destination A:BIGDATA

  • Return traffic takes the same path back

    # On B, set the source address for this traffic to B fibre
iptables -t nat -A POSTROUTING -d A:BIGDATA -p tcp --dport 3434 -j SNAT --to-source B:BIGDATA

NAT connection tracking takes care of the last mapping back to original source IP.

Community
  • 1
Matt
  • 1,537
  • 8
  • 11
  • Thanks for the clear steps with full command lines! When my home-brewed solution fails me (undoubtedly it will), I will switch to this. This is the network-smartypants answer I knew must exist! – RaveTheTadpole Aug 19 '13 at 22:26
0

While it's not the answer I was hoping to find, I ended up rolling my own.

I wrote a program to run on B that would accept connections on the LAN interface and initiate connections out the BIGDATA interface. The program copies the TCP payload into the new outgoing messages. It does the same thing with replies going back the other way. This doesn't preserve the header information (e.g. source IP) -- in my case I think that is a good thing, so that the return traffic doesn't shortcut.

It's working, though it was a pain for something that I consider so basic. I'd post the source, but it's pretty long-winded and probably not very well written.

I'm going to leave answers unaccepted, in hopes that someone will eventually stumble across it with the perfect solution!

RaveTheTadpole
  • 186
  • 1
  • 7