rp_filter not working, still get martian errors and dropped traffic

Question

I am using the below commands to setup test environment for traffic flow. I am using a bash shell for traffic source but would eventually be VM or container.

I don't understand where the traffic is being dropped and was hoping someone could help.

echo 1 > /proc/sys/net/ipv4/conf/all/log_martians

brctl addbr br0
brctl addbr br1

ip netns add nstest
ip link add veth-a type veth peer name veth-b
ip link add veth-c type veth peer name veth-d

ip link set veth-b netns nstest
ip netns exec nstest ip addr add 172.20.0.2/24 dev veth-b
ip netns exec nstest ip route add default via 172.20.0.1
ip netns exec nstest ip link set dev veth-b up

brctl addif br0 veth-a
brctl addif br0 veth-d
brctl addif br1 veth-c

ip addr add 172.20.0.1/24 dev br1

ip link set dev br0 up
ip link set dev br1 up
ip link set dev veth-a up
ip link set dev veth-c up
ip link set dev veth-d up

ip route flush cache

find /proc/sys -name rp_filter -exec sh -c "echo 0 > {}" \;
find /proc/sys -name rp_filter -print -exec sh -c "cat {}" \;

iptables -t nat -I POSTROUTING -s 172.20.0.0/16 ! -d 172.20.0.0/16 -j MASQUERADE

ip netns exec nstest bash
 $ ping -c 1 172.20.0.1/24
 $ ping -c 1 8.8.8.8

Two bridges are connected with veth interfaces and the GW is placed on BR1 to make traffic traverse the two bridges.

172.20.0.2[veth-b]----[veth-a][br0][veth-d]-----[veth-c][br1][172.20.0.1]

From bash shell i can ping GW 172.20.0.1 OK but if i try to ping public address e.g. 8.8.8.8 i get no reply.

conntrack shows the traffic

icmp     1 28 src=172.20.0.2 dst=8.8.8.8 type=8 code=0 id=6085 [UNREPLIED] src=8.8.8.8 dst=10.0.2.15 type=0 code=0 id=6085 mark=0 use=1

The tcpdump is bit weird. The MAC e2:1c:84:b1:a3:5f is assigned to veth-d interface in the nstest network namespace.

12:37:53.547319   P e2:1c:84:b1:a3:5f ethertype IPv4 (0x0800), length 100:       172.20.0.2 > 8.8.8.8: ICMP echo request, id 6017, seq 1, length 64
12:37:53.547435 Out e2:1c:84:b1:a3:5f ethertype IPv4 (0x0800), length 100: 10.0.2.15 > 8.8.8.8: ICMP echo request, id 6017, seq 1, length 64
12:37:53.547437  In e2:1c:84:b1:a3:5f ethertype IPv4 (0x0800), length 100: 10.0.2.15 > 8.8.8.8: ICMP echo request, id 6017, seq 1, length 64
12:37:53.547437  In e2:1c:84:b1:a3:5f ethertype IPv4 (0x0800), length 100: 10.0.2.15 > 8.8.8.8: ICMP echo request, id 0, seq 1, length 64

And this is a iptables TRACE

TRACE: raw:PREROUTING:policy:2 IN=br0 OUT= PHYSIN=veth-a MAC=02:e4:cc:1e:06:cc:e2:1c:84:b1:a3:5f:08:00 SRC=172.20.0.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 
TRACE: mangle:PREROUTING:policy:1 IN=br0 OUT= PHYSIN=veth-a MAC=02:e4:cc:1e:06:cc:e2:1c:84:b1:a3:5f:08:00 SRC=172.20.0.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 
TRACE: nat:PREROUTING:policy:2 IN=br0 OUT= PHYSIN=veth-a MAC=02:e4:cc:1e:06:cc:e2:1c:84:b1:a3:5f:08:00 SRC=172.20.0.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 
TRACE: mangle:FORWARD:policy:1 IN=br0 OUT=br0 PHYSIN=veth-a PHYSOUT=veth-d MAC=02:e4:cc:1e:06:cc:e2:1c:84:b1:a3:5f:08:00 SRC=172.20.0.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 
TRACE: filter:FORWARD:rule:1 IN=br0 OUT=br0 PHYSIN=veth-a PHYSOUT=veth-d MAC=02:e4:cc:1e:06:cc:e2:1c:84:b1:a3:5f:08:00 SRC=172.20.0.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 
TRACE: filter:DOCKER-ISOLATION:return:7 IN=br0 OUT=br0 PHYSIN=veth-a PHYSOUT=veth-d MAC=02:e4:cc:1e:06:cc:e2:1c:84:b1:a3:5f:08:00 SRC=172.20.0.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 
TRACE: filter:FORWARD:policy:16 IN=br0 OUT=br0 PHYSIN=veth-a PHYSOUT=veth-d MAC=02:e4:cc:1e:06:cc:e2:1c:84:b1:a3:5f:08:00 SRC=172.20.0.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 
TRACE: mangle:POSTROUTING:policy:2 IN= OUT=br0 PHYSIN=veth-a PHYSOUT=veth-d SRC=172.20.0.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 
TRACE: nat:POSTROUTING:rule:1 IN= OUT=br0 PHYSIN=veth-a PHYSOUT=veth-d SRC=172.20.0.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 

TRACE: raw:PREROUTING:policy:2 IN=br1 OUT= PHYSIN=veth-c MAC=02:e4:cc:1e:06:cc:e2:1c:84:b1:a3:5f:08:00 SRC=10.0.2.15 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 
TRACE: mangle:PREROUTING:policy:1 IN=br1 OUT= PHYSIN=veth-c MAC=02:e4:cc:1e:06:cc:e2:1c:84:b1:a3:5f:08:00 SRC=10.0.2.15 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 
TRACE: nat:PREROUTING:policy:2 IN=br1 OUT= PHYSIN=veth-c MAC=02:e4:cc:1e:06:cc:e2:1c:84:b1:a3:5f:08:00 SRC=10.0.2.15 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=33737 DF PROTO=ICMP TYPE=8 CODE=0 ID=4182 SEQ=1 

IPv4: martian source 8.8.8.8 from 10.0.2.15, on dev br1
ll header: 00000000: 02 e4 cc 1e 06 cc e2 1c 84 b1 a3 5f 08 00        ..........._..

I don't know if the martian thing is relevant since i disabled rp_filter.

Thank you fLo

Update with more info

1) Yes, /proc/sys/net/ipv4/ip_forward has been enabled.

2) enp0s3 which is the main host adapter in default network space has address of 10.0.2.15/24

3) Here is output of ip route from default namespace

default via 10.0.2.2 dev enp0s3 
10.0.2.0/24 dev enp0s3  proto kernel  scope link  src 10.0.2.15 
172.20.0.0/24 dev br1  proto kernel  scope link  src 172.20.0.1 

and from the testns namespace

default via 172.20.0.1 dev veth-b 
172.20.0.0/24 dev veth-b  proto kernel  scope link  src 172.20.0.2 

4) Here is ip a show from host if useful.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:3b:e4:70 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe3b:e470/64 scope link 
       valid_lft forever preferred_lft forever
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 4a:0a:90:fc:18:53 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::480a:90ff:fefc:1853/64 scope link 
       valid_lft forever preferred_lft forever
4: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:e4:cc:1e:06:cc brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.1/24 scope global br1
       valid_lft forever preferred_lft forever
    inet6 fe80::e4:ccff:fe1e:6cc/64 scope link 
       valid_lft forever preferred_lft forever
5: veth-a@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 4a:0a:90:fc:18:53 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::480a:90ff:fefc:1853/64 scope link 
       valid_lft forever preferred_lft forever
6: veth-d@veth-c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether de:5d:d5:85:08:ee brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc5d:d5ff:fe85:8ee/64 scope link 
       valid_lft forever preferred_lft forever
7: veth-c@veth-d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1 state UP group default qlen 1000
    link/ether 02:e4:cc:1e:06:cc brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e4:ccff:fe1e:6cc/64 scope link 
       valid_lft forever preferred_lft forever