1

What I'm trying to achieve

At work we use a specific operating system, which we (the developers) run within virtualbox. The virtualized OS is running Samba server and rsh that we use to access the OS. Also, the virtualized OS needs access to several servers on our network.

What I would like to achieve is to have a way to quickly start virtual machines as a part of Docker setup. The idea is to have a docker container that would provide access to the virtual machine, a container that runs tests on the virtual machine and in the future some other containers that would provide eg. analysis of the system.

What is the problem

I have a docker container that sets up a VirtualBox machine with bridged network. I can access the virtual machine using rsh from the container that started it without any problems. However, I cannot access the virtual machine using rsh from other containers (well, I found a solution that I will describe bellow, but I don't think it's a good one). Interestingly, ping works fine.

Right now the setup looks like this:

Virtual Container

  • the Docker container that holds the Virtual machine
  • IP address: 172.18.0.5
  • Gateway: 172.18.0.1
  • Netmask: 255.255.0.0

VirtualBox machine

  • running inside Virtual container
  • IP address: 172.18.1.5
  • Gateway: 172.18.0.1
  • Netmask: 255.255.0.0

Worker container

  • a container that I would like to connect to the VirtualBox machine
  • IP address: 172.18.0.6
  • Gateway: 172.18.0.1
  • Netmask: 255.255.0.0

So far I found only one solution on how to connect using rsh to the VirtualBox machine, and that is to add a route:

ip route add 172.18.1.5 via 172.18.0.5

And then rsh to 172.18.1.5. However, this has one serious problem. Since I need to know the address assigned to the virtual machine, I must set the virtualbox address to an address that can be easily guessed from the address of the container it is running in. Changing the route within container also means the container needs to be privileged, and I would like to keep the number of privileged containers to a minimum.

What I would like much more is to be able to rsh to the Virtual container (172.18.0.5) directly. That way I would not have to guess the IP of the virtual machine, but I could use the hostname of Virtual thanks to docker-compose.

I thought I could use NAT to achieve that. What I did was to:

  • change gateway of the virtualbox machine to the IP address of Virtual
  • expose all ports of Virtual
  • add following two rules to the Virtual container:
iptables -t nat -A PREROUTING --destination 172.18.0.5 -j DNAT --to-destination 172.18.1.5
iptables -t nat -A POSTROUTING --source 172.18.1.5 -j SNAT --to-source 172.18.0.5

However, it doesn't seem to work at all. Note: I do not care that whether all ports are forwarder or not. Obviously it would be better or forward only rsh and Samba, but rsh is problematic due to the server selecting the port for communication similar to passive FTP.

Any ideas what I'm doing wrong or how else can I achieve my goal?

Update

The contents of the iptables nat table:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       all  --  anywhere             4f04152f8562         to:172.18.1.5

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER_OUTPUT  all  --  anywhere             127.0.0.11          

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER_POSTROUTING  all  --  anywhere             127.0.0.11          
SNAT       all  --  anywhere             anywhere             to:172.18.0.5

Chain DOCKER_OUTPUT (1 references)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             127.0.0.11           tcp dpt:domain to:127.0.0.11:43467
DNAT       udp  --  anywhere             127.0.0.11           udp dpt:domain to:127.0.0.11:39285

Chain DOCKER_POSTROUTING (1 references)
target     prot opt source               destination         
SNAT       tcp  --  127.0.0.11           anywhere             tcp spt:43467 to::53
SNAT       udp  --  127.0.0.11           anywhere             udp spt:39285 to::53A

Also, I've installed wireshark into the container and I think the problem is that there is a cycle caused by the iptables rules.

I can see that packets from 172.18.0.6 to 172.18.0.5 are sent to 172.18.1.5, so that is correct. Also the replies from 172.18.1.5 have the source address translated to 172.18.0.5 and then they are sent to 172.18.0.6, which is correct, too. However, then there is a connection from 172.18.1.5 to 172.18.0.5. I think there lies the problem, since the connection should be to 172.18.0.6. However, I don't know how to make sure the virtual knows the correct address to connect to.

enter image description here

Update 2

Now that I'm thinking about it - isn't DNAT supposed to leave the source address unchanged? According to the wireshark it is changed to the address of Virtual. So it's no wonder the virtual machine creates connection to Virtual instead of Worker.

stativ
  • 111
  • 5

1 Answers1

0

So the solution is incredibly simple:

Move the Virtual into another subnet and set its gateway to the IP of the Container

That's it. NAT rules as mentioned in the question are correct.

Because the Worker and the Virtual were on the same subnet, the Virtual connected to the Worker directly when creating the connection to Worker that RSH uses, thus bypassing NAT.

Unfortunately I did one more mistake while debugging this issue using wireshark. Being so hopeless I changed the iptables rules to:

-t nat -A PREROUTING -j DNAT --to-destination 172.18.1.5
-t nat -A POSTROUTING -j SNAT --to-source 172.18.0.5

With such rules it seemed to partially work, because suddenly the Container became the source which made the initial handshake work. However, it broke the RSH connection from Virtual since the information that the original connection was coming from Worker was already lost.

stativ
  • 111
  • 5