How to restrict/secure access to developer's server hosted on Internet

Question

I am setting up two Ubuntu virtual private servers for my startup company. The "public" server (let's call it www.example.com) shall contain the publicly open company web site and restricted but (relatively less sensitive) services such as the LDAP and mail servers. The "dev" server (dev.example.com) will be the company's Fort Knox holding the company's source code (GIT), artefacts (Artifactory) and developer tools (Atlassian). Access to www.example.com should be open to everyone whilst the dev server should be very secured, accessible only to a very limited number of authorized developers and the other public ideally shouldn't even be able to guess that the server exists. Both servers will be hosted at a hosting company (with us having full root access).

I am planning to set up web access using the same domain name and virtual hosts. example.com will point to the "public" server. nginx installed on the "public" server will have the www.example.com virtual host (served normally) and the dev.example.com virtual host, all requests to that host will be proxied to the "dev" server running another nginx. The firewall on the "dev" server will be set up to deny all incoming traffic except HTTPS and SSH calls coming from the "public" server. So not only direct connections to the nginx server running the developer tools on the "dev" server will not be possible, but we will also have to log in to the public server first in order to be able then to SSH into the "dev" server. Thus a potential hacker wouldn't be able to access directly the "dev" server without first breaking into the public server.

If it matters, the servers will probably run Ubuntu 16.04 (we have a choice between 14.04, 16.04, 18.04 and other Linux flavours, but Ubuntu 16.04 sounds OK'ish to me). I am neither a security expert nor a system administrator.

I have two questions.

1) Does this sound like a good secure setup confirming to "best practicies" or are there better (simpler or more secure) ways to achieve same goals?

2) [Main question]. How can I restrict HTTPS access so that only developers can access dev.example.com while other users don't see this virtual host at all even if they've guessed the URL? I have two ideas so far.

• require an SSH tunnel. The dev.example.com virtual host in nginx on the "public" server will be set up to accept requests only from the "public" server's IP address itself, so the developers will have to set up the SSH tunnel to the "public" server in order to access dev.example.com. With tools like MyEnTunnel (Window) it is easy to setup a SSH tunnel so that it is automatically established during boot time. However the big disadvantage I see is that each developer will have to define that "dev.example.com" refers to localhost in their hosts file (or are there other ways?)

• Client certificates. nginx on the "public" server will basically require a client certificate in order to access dev.example.com. Each developer will have his/her own client certificate to access dev.example.com in addition to their usual login and password that they will use to access the developer tools. However I am not very familiar with client certificates and don't understand all implications. One disadvantage I see is that probably when unauthorized person tries to access dev.example.com he/she will get a "client certificate" error which will reveal that the dev server is there, while with the first approach they will simply not be able to connect.

What would you recommend me? The idea is to keep it simple, cheap and at the same time as secure as possible until the company grows to such an extent when it can afford sysadmins and its own data center.

Answer 1

Part 1:

Accessing a highly secured server through logging in to a public not-so-secured server gives me a bit bad feelings in the stomac, like passing the top in a rollercoaster.

If you want to have a secure dev server, secure it. Install required/helpful stuff only. Security is limited to what you control - your hosting provider has access anyway if they want - they have physical access.

If you want to conceal your dev server, conceal it. Drop any incoming network traffic. Maybe you want to use port knocking to open your private doors. But be aware, that your provider can see it all.

Part 2:

Yes, having all services of your dev server listen on localhost only is a simple, and effortless way to conceal and secure your server at the same time.

Requiring users to use ssh tunnels is ok. It works, is set up in a few minutes and can be done with about any ssh binary out there. Good idea in my opinion.

If you close all other doors, you have the ssh door left, which is visible to the outside world. You can move it to the backyard though, changing from port 22 to a random like 32109.

Client certificates complete this type of security to - in my opinion - more than enough. If more would be required, not having your own physical fort nox, where your server is secured against unauthorized physical access would be grossly neglegient.

Nobody will get an "client certificate" error before he has established his ssh tunnel, so don't worry about this.

No, your developers won't define, that dev.mydomain.com refers to localhost, else they wouldn't be able to ssh to it. They may safely access your dev server via localhost or a hosts entry "127.0.0.1 mydev" or whatever you name it internally. This doesn't require a publicly accessable name.

On your Ubuntu 18.04 - you are waaay ahead of time ;-) If you want to use Ubuntu, use 16.04. It will endure your company's startup phase. Other Flavours are equally ok, though. Make sure there will be updates available for a few years at least, like 04/2021 for your Ubuntu-16.04

Btw: if you want to conceal your dev server, don't give it a name service entry like dev.myserver.com; you don't need it, others will need it less. Just put it in your developer's hosts files.

Have Fun!

TomTomTom