I have a multihomed setup with two upstream providers. And I want to confine some connections to use only one provider — local outgoing connections that is. So, I've made a custom routing table, which has a directly-connected network route and a default route:
$ ip route show table 2
default via 5.1.0.1 dev upstream_b
5.1.0.0/16 dev upstream_b scope link src 5.1.0.2
Next, I'm using a selector in iptables to add fwmark to some connections and ip-rule to override routing table selection:
# for output traffic
# restore packet mark from connection mark
iptables --table mangle --append OUTPUT -j CONNMARK --restore-mark
# for !0 packet marks bail out
iptables --table mangle --append OUTPUT -m mark ! --mark 0 -j ACCEPT
# set packet mark
iptables --table mangle --append OUTPUT -m cgroup --cgroup 0x00010002 -j MARK --set-mark 0x2
# save packet mark to connection mark
iptables --table mangle --append OUTPUT -j CONNMARK --save-mark
# for input traffic restoring mark from connmark
iptables --table mangle --append PREROUTING -j CONNMARK --restore-mark
# adding policy forwarding
ip rule add from all fwmark 2 table 2
So, for some process [in the specified net_cls cgroup] its outgoing connections will hit connection mark rule and then should be routed via upstream_b interface.
But that is not working (for a bash shell, which was put into the net_cls cgroup):
$ curl --resolve ifconfig.co:80:188.113.88.193 -v http://ifconfig.co
* Added ifconfig.co:80:188.113.88.193 to DNS cache
* Rebuilt URL to: http://ifconfig.co/
* Hostname ifconfig.co was found in DNS cache
* Trying 188.113.88.193...
* Immediate connect fail for 188.113.88.193: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server
The previous result took place while main routing table has no default route (i.e. no route to ifconfig.co).
But this is not the whole story. Since I have the multihomed setup, there usually is a ECMP route to balance traffic via both providers. And for the mentioned test I got an interesting result. The tcp-syn packages went to the correct interface but with an incorrect source address (from the other interface).
So, joining these observations, firstly, a routing decision is made based on main routing table (thus, network is unreachable). Next, while a package is processed further, it hits the marking rules and then a rerouting happens (based on the custom routing table). But the source address is already set in the first routing decision, and it is not corrected here.
Is my understanding of the problem origins correct?
I need to enforce a non-default routing table for some local processes and the method above is not working. Could the enforcement be accomplished? For example, is it possible to trigger the source address recalculation?
EDIT. Considering the following diagram (the kernel version here is not that modern, but still useful; the diagram is from http://open-source.arkoon.net/kernel/kernel_net.png): 
It seems that the first routing decision and the package source address selection are indeed performed before any iptables rule could kick in (see right bottom corner).
