Lan Wifi conection to TV box with huge latency

Question

I am facing an odd problem. I use an android TV box at home, connected to my network via WI-FI. Ethernet isnt an option. I use Kodi, and Yatse as its remote on my android phone. I am seeing an odd issue where if I perform any action in Yatse, it takes about 3-5 seconds to manifest on the TV box. Just scrolling through a list is a chore. The truely weird thing is that if I somehow time a second action exactly when the first action manifests, both the action goes through. This almost sounds like the http connection is taking forever to form, but any message reusing it is as fast as expected.

Another way I verified this behavior is using the app lanmote lite which is basically a more generic remote control app. If I go to the mouse section of the app and start making any motion, at first there is no response, and then after a while, usually it gets to a point when it is responding immediately.. as long as I keep moving my finger.

This behavior is kinda new actually.. but ive had issues with wifi on this tv stick since the begining. The more consistent issue is that ssh behavior with ssh droid is very bad, not anywhere near realtime typing. I have a different Android TV also which sshdroid which works perfectly.

I need to know how to debug this, if this is a problem with the hardware and unfixable. Or is it a problem with the location? (possibly the walls or being close to the TV is making it drop packets. Or some configuration problem on the stick or just performance issue.

Sorry for the long question, not sure if this is the right network to put it but most other places didnt seem a good fit.