4

We're moving a bunch of services, say from 1.2.3.4 to 5.6.7.8.

In order to test that the new services are correctly configured, we'd like to redirect (to the new host) all traffic destined for the original host that originates from our testing workstations.

Of course, such redirection could be implemented on routers within the network itself—but for stability reasons we've decided to implement it on each workstation directly (which are all OS X 10.10 Yosemite, so use pre-v4.7 OpenBSD pf).

I've added to /etc/pf.anchors/com.apple:

rdr-anchor "910.TestServiceMove/*"
anchor "910.TestServiceMove/*"
load anchor "910.TestServiceMove" from "/etc/pf.anchors/910.TestServiceMove"

And created /etc/pf.anchors/910.TestServiceMove:

rdr pass log on lo0 from any to 1.2.3.4 -> 5.6.7.8
pass out log route-to lo0 from any to 1.2.3.4 keep state

When the rules are loaded, both appear to work correctly:

$ sudo tcpdump -v -n -e -ttt -i pflog0
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes
00:00:00.000000 rule 0.910.TestServiceMove.0/0(match): pass out on en1: (tos 0x0, ttl 64, id 40691, offset 0, flags [DF], proto TCP (6), length 64)
    9.9.9.9.58029 > 1.2.3.4.22: Flags [S], cksum 0x291a (correct), seq 3399416413, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 2063366865 ecr 0,sackOK,eol], length 0
00:00:00.000047 rule 0/0(match): rdr in on lo0: (tos 0x0, ttl 64, id 40691, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 896a (->b4da)!)
    9.9.9.9.58029 > 5.6.7.8.22: Flags [S], cksum 0xb284 (correct), seq 3399416413, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 2063366865 ecr 0,sackOK,eol], length 0

But the TCP handshake does not complete (SYN-ACKs are ignored, and SYN is repeatedly sent until the connection times out):

$ sudo tcpdump -v -n -e -ttt host 5.6.7.8
tcpdump: data link type PKTAP
tcpdump: listening on pktap, link-type PKTAP (Packet Tap), capture size 65535 bytes
00:00:00.000000 e8:80:2e:e7:67:bc > 84:80:2d:35:e5:43, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 63, id 40691, offset 0, flags [DF], proto TCP (6), length 64)
    9.9.9.9.58029 > 5.6.7.8.22: Flags [S], cksum 0xb284 (correct), seq 3399416413, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 2063366865 ecr 0,sackOK,eol], length 0
00:00:00.015524 84:80:2d:35:e5:43 > e8:80:2e:e7:67:bc, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    5.6.7.8.22 > 9.9.9.9.58029: Flags [S.], cksum 0x7ce4 (correct), seq 1901846890, ack 3399416414, win 14480, options [mss 1460,sackOK,TS val 523934721 ecr 2063366865,nop,wscale 7], length 0
00:00:00.986946 e8:80:2e:e7:67:bc > 84:80:2d:35:e5:43, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 63, id 25319, offset 0, flags [DF], proto TCP (6), length 64)
    9.9.9.9.58029 > 5.6.7.8.22: Flags [S], cksum 0xae9c (correct), seq 3399416413, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 2063367865 ecr 0,sackOK,eol], length 0
00:00:00.014938 84:80:2d:35:e5:43 > e8:80:2e:e7:67:bc, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    5.6.7.8.22 > 9.9.9.9.58029: Flags [S.], cksum 0x78fa (correct), seq 1901846890, ack 3399416414, win 14480, options [mss 1460,sackOK,TS val 523935723 ecr 2063366865,nop,wscale 7], length 0
00:00:00.397794 84:80:2d:35:e5:43 > e8:80:2e:e7:67:bc, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    5.6.7.8.22 > 9.9.9.9.58029: Flags [S.], cksum 0x776c (correct), seq 1901846890, ack 3399416414, win 14480, options [mss 1460,sackOK,TS val 523936121 ecr 2063366865,nop,wscale 7], length 0
00:00:00.588237 e8:80:2e:e7:67:bc > 84:80:2d:35:e5:43, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 63, id 50201, offset 0, flags [DF], proto TCP (6), length 64)
    9.9.9.9.58029 > 5.6.7.8.22: Flags [S], cksum 0xaab4 (correct), seq 3399416413, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 2063368865 ecr 0,sackOK,eol], length 0

I guess that the TCP stack is discarding SYN-ACKs that have originated from a host other than that to which the SYN was sent. But shouldn't redirection rules rewrite traffic in both directions—indeed, shouldn't keep state be ensuring that the connection is tracked for this purpose?

eggyal
  • 392
  • 4
  • 16
  • @Downvoter: Care to comment? Have I not shown research effort? Is the question unclear? I'd happily improve the question if you could help me understand what's wrong with it! – eggyal Mar 22 '15 at 12:20
  • 1
    Why not simply change the DNS records or edit your host file? That's easier and does the same. – megamorf Mar 22 '15 at 12:37
  • @megamorf: It only "does the same" where name lookups are involved. Some of the applications that we are testing unfortunately do not use hostnames, but rather use the IP address directly. – eggyal Mar 22 '15 at 12:49
  • @megamorf: In addition, for the applications that *do* use name lookups, we'd need to ensure that wildcard labels in over 2000 domains (most of which are outside our control) all resolve to the new host: sure, we *could* create an internally split DNS to accomplish that (though could not use `/etc/hosts` because of the wildcards), but would need to limit its viewing to the testing workstations. With so many domains and so much configuration involved—plus the applications that use hard IP addresses—we have determined that packet redirection is the most appropriate option. – eggyal Mar 22 '15 at 12:50
  • It may be a dumb question, but I can not make any sense of it ... Why are you routing it through loopback? Wouldn't `rdr pass log from any to 1.2.3.4 -> 5.6.7.8` work as well and without the `pass`? – Fox Mar 25 '15 at 23:00
  • @Fox: If I've understood pf correctly, rdr is only applied to traffic ingressing through an interface—so one must first route to loopback in order to achieve that. – eggyal Mar 25 '15 at 23:02
  • I see this weird wording in FreeBSD man page (sorry, too lazy to go looking for my macbook) about the order of rules. But it does not mention the third rule does not get executed. And even the example section talks about `rdr`ing outgoing packets on the interface itself. tbh i am not sure what are the exact impact of your rules, but i'd try reducing it to just rdr ... – Fox Mar 25 '15 at 23:10
  • Ah I found the Mac man online. It contains the same things as the one I was referring to. – Fox Mar 25 '15 at 23:12
  • @Fox: it is only because rdr alone did not work that I hunted around and discovered the necessity of routing through loopback first. Am on my mobile right now, otherwise would link you to the specific documentation that I found. – eggyal Mar 25 '15 at 23:12
  • I see. I'd try reverting back to the rdr only version and trying again. While at that, try dumping on actual ethernet interface and try inspecting the pf state table using `pfctl -s states`... btw. i do not get the downvotes either. it's a nice problem :) – Fox Mar 25 '15 at 23:19
  • @Fox: Okay, thanks. I will try tomorrow morning (UK time) and report back. – eggyal Mar 25 '15 at 23:21
  • I have found what you have found and I take back my suggestion regarding the rules ... You are right. `rdr` works only on inbound traffic. But, there are another suggestion, that might help a little ... do you have `set block-policy return`? That could help you figure out where it gets dropped(rejected)... – Fox Mar 25 '15 at 23:42
  • @Matt: Could you clarify your comment please? Just because the "new server doesn't answer for the old address" surely doesn't stop pf on the client rewriting the received packets so that they appear to the application to be from the old address? Is that not, after all, how redirection works? If not, what possible purpose can `rdr` serve for connection-oriented protocols like TCP? Indeed, the documentation that I've read appears to categorically contradict what you have said... – eggyal Mar 26 '15 at 19:03
  • @eggyal: have you found any solution to this problem since? – jamborta Jun 21 '15 at 12:28
  • I'm interested in a solution too because i've the same needs – Not Important Feb 16 '16 at 14:39

3 Answers3

1

If the services you are accessing are all accessed by name then I'd just create a host entry with in the hosts file on your dev machines.

For example, if your machine was called myserver and resolved to 1.2.3.4 Then you can now create a host file entry /etc/hosts with

myserver 5.6.7.8

Anything that now tries to reach myserver will end up hitting the new machine.

Alternatively, just create a new isolated network with the new server at the old address and plug your dev machine into it to test.

The problem with a straight redirection or route rule is that the new server doesn't answer under the name of the old address. So you'd have to use NAT.

hookenz
  • 14,132
  • 22
  • 86
  • 142
  • I've already commented that `hosts` is not a suitable solution for two reasons: (1) some applications do not use name lookups; and (2) others rely on wildcard DNS labels. An isolated network with the new machine at the old address would be a plausible solution, but my question was how to get the redirection working. You say that I need to use NAT because "the new server doesn't answer under the name of the old address" but my reading of the pf manual is that rdr rules should handle exactly this situation: else what is the point in them? – eggyal Mar 26 '15 at 18:51
  • I might have misunderstood the OSX rdr app... anyway, I'd just use a separate network. The amount of time you've spent working this out you'd have tested it by now. – hookenz Mar 26 '15 at 18:56
0

Wouldn't it just be easier to throw the machines behind a simple loadbalancer or NAT? That's generally more of an industry-standard practice, anyway, and would allow you to further scale the application over time.

RVT
  • 397
  • 1
  • 8
  • Which machines? The test workstations? They're distributed across 28 offices globally; where do you propose this "simple loadbalancer" be placed? In any event, I'm interested in getting packet redirection working. There's nothing, conceptually, about it that shouldn't work—and the pf documentation, to my reading, makes it abundantly clear that this configuration is supported. So all I really need is some help getting it to work, rather than countless suggestions of alternatives (that happen not to be very useful in our situation anyway: hence why we chose this approach in the first place!). – eggyal Mar 26 '15 at 22:45
  • Then, I'd say your question wasn't terribly clear, and should be rephrased. My apologies, but I read you're moving from one IP, to one IP. So, in that general case, it's generally *easier* to use use a third-party NAT (which, in my opinion, is a bit easier that pf rewrite rules - though generally requires a bastion-type host) – RVT Mar 26 '15 at 23:03
  • You may well be right that my question is unclear—I'm certainly not getting from it the response I anticipated. We *are* "moving from one IP to one IP", but at this time we are simply trying to *test* that the services on the new IP are correctly configured. I will have a think about how best to rephrase my question to make this clear. – eggyal Mar 26 '15 at 23:09
  • Use a fake network. You'd be done by now. – hookenz Mar 26 '15 at 23:40
  • LVS was my first thought. That could be done in a Linux-VM and does not require additional hardware. – Nils Mar 27 '15 at 14:35
-1

Can you just offer the "old" address from the "new" server ala sub-interface, or is the new address in a totally different subnet?

If you can't attach the new IP to the new server, then you haven't really moved the address. To route traffic across the old address to the new server, you need to NAT the address in the old subnet. Traffic would land on the "old" address and then be source NAT'd to a new address on the same server/appliance and sent to the "new" IP. Return traffic would then pass back to the SNAT address, which would un-NAT the traffic and return it to the original requester. This is really not a good long term solution, especially if the "new" address is across a WAN link. You've put off an inevitable failure and added latency to the mix as well.

A better long term solution would be to consider: 1. use DNS instead of IP 2. virtualize either the IP address (NAT'ing device like a firewall or ipmasq box), 3. or the server itself (load-balancer)

Rick Buford
  • 166
  • 5
  • We're not trying to move the service right away. The question is only how to *test* that the new server is configured appropriately, so it is undesirable at this time either to move the IP address or to indiscriminately redirect traffic from the old address (both those solutions would, obviously, affect the incumbent service). Afraid I'm going to downvote, as I don't see how these suggestions are useful. Sorry. – eggyal Mar 22 '15 at 15:03
  • you didn't answer my question – Rick Buford Mar 25 '15 at 22:09
  • It's on a totally different subnet. – eggyal Mar 25 '15 at 22:11
  • Then I have, actually, answered your question. To properly route the traffic to an IP, you will have to NAT the traffic from the "old" subnet to the "new" host. – Rick Buford Mar 25 '15 at 22:14
  • Configuration at the "old" (i.e. currently live, public-facing) service is not an option, not least because it risks affecting ongoing live traffic. We must configure the test platform to test the new service without affecting the live service. Your proposal does not assist with that, whereas the approach outlined in my question would achieve exactly what we want—if only I could understand how to fix the configuration. The pf documentation is quite clear that this is possible, I just need some help getting it to work. – eggyal Mar 25 '15 at 22:20