Can't connect to SQL Server on a non-default port

Question

I am learning how to set up SQL Server 2016, 2017 on virtual machines. I set up everything using default port, and I have no firewall on the server and everything works fine on the server and I can connect from another computer.

I change the port to use a non-default port in the SQL Network Configuration section of SQL Server Configuration manager. I can connect fine on the server. When I try to connect I get the following message. I have Windows Server 2016 and SQL Server 2014 and 2016. I tried both.

TITLE: Connect to Server
------------------------------

Cannot connect to SQLVMachine.

------------------------------
ADDITIONAL INFORMATION:

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) (Microsoft SQL Server, Error: 5)

For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%20SQL%20Server&EvtSrc=MSSQLServer&EvtID=5&LinkId=20476

------------------------------

Access is denied

------------------------------

Then I open cliconfg and make sure that TCP/IP is configured and enabled. Everything works fine on the server. But I can not connect remotely. I get a different message but I still can not connect.

TITLE: Connect to Server
------------------------------

Cannot connect to SQLVMachine.

------------------------------ ADDITIONAL INFORMATION:

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) (Microsoft SQL Server, Error: 2)

For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%20SQL%20Server&EvtSrc=MSSQLServer&EvtID=2&LinkId=20476

------------------------------

The system cannot find the file specified

------------------------------  

I am out of ideas. Did I miss anything? Could there be a group policy set up on the network I am connected to? Are there special permissions in Active Directory? Has Windows Server 2016 changed the way you install SQL Server? Are there new server setup requirements?