Answer to your question
In order to RDP to a server without showing the port to which you connect can be done by making an RDP icon that will log you in by doubleclicking on it. It will not show the server and port because it just goes too fast.
Open your RDP client, fill in the details (ip:port), then press the Show Options
button on the bottom left.
Below in the Connection settings section, you can save your RDP file as a new one. Next time you doubleclick this icon, it will connect to your server.
If you use a LAN Ip or servername, and not fill in the gateway settings in the advanced tab, this RDP will only work when the client is in the same network as the server, unless a VPN connection is used as well.
NOTE: A user can edit your RDP file to find out the port.
Alternative solution
Alternatively, you could make sure your server is only accessible from within your network by blocking the port on your router, and if your clients don't use VPN to your network, you could require establishing a VPN first before RDPing to your server.
Alternative solution 2
Create a special user for RDP sessions and limit who can RDP to the server. Even if your clients know the port, they still need a valid username and password to connect. You can even save the credentials in your RDP file, but this poses a security risk because anyone who can copy your RDP file will copy the username and password as well.
1If they see you connecting to RDP, they can find out the port in five minutes. Why are you using port numbers as a security feature? – user1686 – 2014-09-29T12:37:28.063
My user base is not technical and don't know how to get port number if it's not visible. I don't want users to go outside the company and RDP to server which has some confidential info. If there is any other way to restrict RDP just from few machines that would also work. I can't use IP Restriction as that changes and I as a administrator still want to RDP from my mobile, etc. – user3616677 – 2014-09-29T12:51:42.507
That's what passwords are for... – user1686 – 2014-09-29T13:14:45.707