That's a textbook symptom of bufferbloat, a widespread design flaw in networking equipment, where the equipment is too dumb to manage its queue of unsent packets when the Internet connection is being fully utilized (congested). It lets the backlog grow and grow, which just adds lag and actually prevents your networking software on your devices from noticing the congestion so their congestion control algorithms can kick in.
To confirm that you have bufferbloat, run http://dslreports.com/speedtest to check your bufferbloat grade. If you do this from a phone or tablet, be sure to request the desktop version of the results page because the mobile version doesn't include the bufferbloat grade.
Using QoS to prioritize some traffic over others doesn't solve bufferbloat, it just dump it on someone else's lap. The real solution is called "Smart Queue Management" (SQM). There are a handful of SQM algorithms to choose from, but the best is Cake, followed by FQ-CoDel. Both are available in OpenWrt and other open source Linux-based router firmware distros. So if your router is supported by OpenWrt, load OpenWrt on it and set up Cake SQM.
Router vendors are way behind on dealing with bufferbloat, so you pretty much have to go open source, although there are a few vendors with a few products known for having good SQM support, such as the evenroute.com IQrouter, Eero's products, and the Ubiquiti EdgeRouter-X.
There are sites like https://www.stoplagging.com/ that can walk you through your options for getting SQM set up on your network one way or another.
