I (somehow) made it work using an old router which was collecting dust laying around even though theoretically it shouldn't work. In case someone else might stumble upon the same question I'm going to post my working configuration here even though it's currently hardware specific one.
The solution uses VLAN tagging capabilities of both TP-Link TD-W8901G and ZTE ZXHN H108L routers but I'm still puzzled why it works even though none of these routers support VLAN trunking. Both of these routers have almost the same web interface which led me to believe that the underlaying firmware is also almost the same.
Wiring of the working network setup is the following:
Second floor
------------
TP-Link TD-W8901G (router, acting as an ADSL modem and now also as a switch)
# RJ11 - connected to the ADSL splitter
# LAN1 - connected to the ASUS RT-AC58U WAN port
# LAN2 - n/c (IPTV used to be connected here via powerline)
# LAN3 - connected to the ASUS RT-AC58U LAN4 port
# LAN4 - connected to the TP-Link TL-PA4010 powerline adapter (on the same floor)
ASUS RT-AC58U (router)
# WAN - connected to the TP-Link TD-W8901G LAN1 port
# LAN1 - connected to the ASUS RT-AC1200G+ (which acts as an access point)
# LAN2 - n/c
# LAN3 - connected to the Raspberry Pi (for smart home use)
# LAN4 - connected to the TP-Link TD-W8901G LAN3 port
TP-Link TL-PA4010 (powerline adapter)
# LAN - connected to the TP-Link TD-W8901G LAN4 port
First floor
-----------
TP-Link TL-PA4010 (powerline adapter)
# LAN - connected to the ZTE ZXHN H108L LAN4 port
ZTE ZXHN H108L (router which acts as a switch only)
# LAN1 - connected to the IPTV STB
# LAN2 - connected to the Android smart box
# LAN3 - connected to the Samsung Smart TV
# LAN4 - connected to the TP-Link TL-PA4010
VLAN configuration on the TP-Link TD-W8901G router/modem/switch:
VLAN rules / VLAN port assignment
VLAN configuration on the ZTE ZXHN H108L router/switch:
VLAN rules / VLAN port assignment
Basically what I did was create VLAN 7 which would carry the IPTV stream and VLAN 8 which would carry the local network. Since TP-Link TD-W8901G router already has support for VLAN tagging I decided to "re-use" it also as a switch so that's why I brought local network back from the ASUS RT-AC58U to its LAN3 port and since IPTV was already in this TP-Link I decided to output both VLANs on its LAN4 port.
Then on the ZTE ZXHN H108L I split incoming VLANs on LAN4 port back into two separate networks and assign each VLAN its port on the router/switch (LAN1 for IPTV, the other 2 LAN ports for local network).
After everything was working great (IPTV on the STB and local network/internet on both Android smart box and Samsung Smart TV) I verified that I get IPTV signal only on LAN1 of the ZTE ZXHN H108L router/switch as expected. So the setup works but I'm still wondering why?
1ad 1) Nope.
ad 2) Might actually be a thing. Go for it. – vautee – 2018-02-14T09:43:35.027
It's possible on some routers - depends if the firmware supports it - you'll have to give it a try. – djsmiley2k TMW – 2018-02-14T13:23:13.653
I actually made it working with the 2). Check my answer. – choxnox – 2018-02-16T08:03:52.793