Transferring large file to/from USB drive causes system to hang until transfer is done

Question

I have a box running Ubuntu 16.04 server. It has an internal SATA hard drive with the OS on it, and an attached USB 3 hard drive which I use for storage of large/long term files. A lot of times, this means copying a large file or folder from the internal drive to the USB drive. However, whenever I do this, the rest of the system gets incredibly slow, if not almost completely frozen, until the file transfer completes.

I've run a few tests to make sure it wasn't due to program overhead or file type. I tried both large contiguous files and large folders of many small files, and I tried using cp, pv with pipes, tar with pipes, and rsync. In all cases, the rest of the system locks up.

I also tried running each of those commands with ionice -c 3 nice -n 20 in front of them, giving the process lowest priority for processor and IO. htop shows that the process has a nice value of 19. Don't know if that's the actual minimum or if it isn't getting set right, but regardless, most other processes, including the web services, have lower nice values than 19 anyway.

The box is running several dynamic web pages, a couple with php and couple with python. Those become basically unusable for the duration of the file transfer. I assume the slowdown is caused by the overhead from USB, since, as I understand it, all USB transfers have to go through the CPU. Is this what is actually causing the slowdown? And, more importantly, is there any way for me to mitigate this?

---

EDIT (answering comments):

Internal and external drives are both using native ext4, so no FUSE. Nothing should be indexing the files that I know of, except for whatever Ubuntu might do by default (this is headless Ubuntu Server). CPU doesn't seem to be under load from htop's perspective, but uptime usage reports jump from around 0.5 to 6.0-8.0. Rate-limiting pv using -L does seem to help, so that's a solution for one tool. I usually only use pv when copying single files. Is there any way to rate-limit the other programs? I'd also prefer to change prioritization rather than just limit it. Optimize based on system load.

It's a pure install of Ubuntu server. It does have xorg installed, but I don't think there's even a window manager installed with it, I just wanted X11 forwarding over ssh.

Also, while the rate-limiters on pv, rsync, etc work well for saving the system, I'd also like to maximize the transfer rate. Is there any way, other than nice/ionice which don't seem to do much, to tell the system to do the transfer as fast as possible, but give anything else on the system higher priority if it wants it? So, for example, if I have a transfer to USB that will take an hour to complete, then half an hour in someone starts using one of the python sites, it would transfer as fast as possible for the first half hour, but then the system would start limiting it when the user starts making requests to the python site. (I'm starting to think this question should have been posted on Server Fault instead, and I apologize for that.)