SOLVED: check the end of the post
Network diagram: https://i.imgur.com/5mc2woO.jpg
This is called asymmetrical routing. I do not want it. It happens because despite the request arriving at eth0 as intended, Centos recognizes that the client source IP also belongs to one of the servers local subnets, and proceeds to reply to the client using that local subnet which belongs to eth1. The reply communication is made locally (switch), bypassing PFsense, which then proceeds to terminate the connection after 30 seconds because it never saw the server replying.
In PFsense there is an option to bypass all traffic belonging to the same subnet, and in Centos there are options to mitigate part of this issue. net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.all.arp_announce = 1
With all these options activated in both Pfsense and centos, the timeouts were increased from 30 to ~800 seconds, but they still happen like clockwork.
I have read I should be able to flag every packet arriving @ eth0 and make a policy to always always always reply the flagged packets using eth0.
PS: the reason the server has both subnets (192.168.0.x and 10.10.10.x) is because both client and server are on separate vlans and the server serves two purposes. 1- SMB share which should always be accessed via eth0 @ vlan 1680 for fire-walling purposes. 2- FTP for the clients disk clones/backups, very resource heavy traffic which should always be accessed via eth1 @ vlan 1010 in order to bypass the pfsense router as to not overload it (I have many client PCs doing backups at the same time).
I would greatly appreciate some input on the best course of action.