I'd recommend considering what you are trying to achieve first, as it looks from the above that you may be fairly new to this. if this is a live system or has people's jobs relying on it, I recommend seeking the help of a consultant or suitable company to consider if what you are considering is suitably secure & resilient etc.
Anyway, to answer your specific questions (although not in the order you asked them):
- Should a host be purchased first so that users will use a name
instead of IP address?
Yes, Public certificates don't work on IPs. Unless this is an internal-only, non critical system I recommend you select and purchase a suitable domain name and create DNS record(s) resolving to your IP.
- Should this certificate be installed on the Windows VM or the from
GCP console.
The cert should be installed on the element(s) that encrypts SSL traffic. if this server is behind a load-balancer or reverse proxy, you may need to install it on there.
if not, most likely the server (IIS).
- How to install an SSL certificate so that the users can use HTTPS
instead of HTTP.
You'll need to follow a specific process depending on what you are installing it on (e.g. IIS).
If you've not done it before I recommend selecting who you are going to buy it from first as most vendors provide instructions on how to request and install the cert which includes steps relevant to their portal.