It is very good advice to not use the web browser for Internet traffic on the server.
It's our policy (I work in cyber security for a federal agency) to never use a web browser (for Internet, or external websites) for any sort of Internet traffic even if it's to download a file that will be used on the server. As another user mentioned in their answer, it is a good idea to download any files or data needed on the server from a different computer; one that is trusted and is on the same network. Then, take the file(s) and verify hashes if you want to go that far, and then copy it to the server either through the network (probably preferred since you don't have easy physical access to this VM) or using a USB drive or something of that sort.
Another thing that I'd recommend if you're able to, is to use the built-in Windows firewall or a trusted third-party firewall (host-based of course) and apply a template or policy to it that doesn't allow web traffic from the server. As an example of this, we use a vendor for antivirus that has a built-in host-based firewall as part of the antivirus software. We then create a firewall policy called "Servers" and apply it to all servers and on that policy we lock down all incoming and outgoing web (usually port 80) Internet traffic so that the server cannot access the Internet even if we accidentally click on the web browser.
As far as uninstalling the browser altogether, I don't think that's a good idea. There are many times when a web browser comes in handy for things like accessing another server that runs a web console or something of that nature that you'd need to use. If you uninstall your web browser you'd have no way of accessing any of these other consoles.