This is just a thought, I will try and test later.
What if you had multiple separate anti-affinity rules from those VMs to other VMs that are on other hosts. (Yes I know this is not how anti-affinity is supposed to work). I don't know if a single VM can be part of multiple anti-affinity rules or not which is what this whole thing hinges on.
VM1, VM2 being the ones you want to keep together, you set up anti-affinity from each VM individually to VM3, and do the same to VM4. Also setup anti-affinity between VM3 and VM4.
If it works it should move around like a nice shell game of VMs moving around smoothly.
Only other way I can think of is setting preferred hosts for the two VMs you want to keep together until you get your external switch issue solved.
EDIT
Looks like it can be part of multiple anti-affinity groups, just do a += when you assign the affinity class name. So you should be good to go if you want to do do this