If I may, I will take the liberty of decomposing your question, taking a step back, and helping to step through the entire solution design process so we can determine a workable solution for you. There are a number of inaccuracies in your question around the SSH protocol, VPNs, and in general, how we design secure systems. Let's work through them here.
At present, your question asks for guidance implementing specific technologies. However, your problem statement does not evaluate the specific threat you are facing, so it is premature to design a solution; in this respect, you have XY problemed us.
Any implementation you make will either not resolve the problem (because in fact the issue lies elsewhere) or add complexity where a simpler solution is adequate. Moreover, needless complexity is undesirable as it may be a source of further security vulnerabilities.
Rigorously define your problem
We should be objective focussed, not solution focussed. We should define the objects of our problem domain, ascertain any contributing factors to consider, understand the threats we face, and only then can we begin to determine which technology and process choices can be made to resolve the problem in hand. A process we may follow:
- Identify the problem – what is the specific difficulty we are seeking to resolve? The difficulty should be an objective, measurable problem, backed up by hard evidence of its existence from observations made in the field.
- Determine assumptions and constraints – are there specific items we can assume as being in a particular state, and are there other conditions imposed on the proposed solution? A significant constraint must include direct costs of implementing the solution (buying hardware, software, or consulting time) and indirect costs (making process change, training staff, accommodating lost productivity).
Identify threat actors (for security problems) – no system or process is 100% secure. We need to carefully determine the nature of all the adversaries who are likely to launch an attack in order to determine whether our solution adequately prevents such attacks. This applies to designing real-world physical security systems as much as designing for technical outcomes.
For example, in your assessment, you may consider such factors as:
- Capabilities of your adversary – how knowledgeable are they, do they have access to specific resources to aid an attack, etc.
- Their position – there is a substantial difference between a script kiddie on the last mile of a residential internet service and a nation state actor with access to positions in the network from which man-in-the-middle attacks are feasible
Your risk and risk thermostat – what motivates the actor to attack you or your organization specifically? For example, does your system store or process sensitive and/or privileged data which is normally of a restricted nature and may be of value to others (personal data, company secrets, etc.)? Does it have a privileged position in a network from which further analysis or attacks may be launched (e.g. a core router in an ISP, a border gateway on the perimeter of a sensitive network in a large corporation)?
This helps identify whether we are dealing with an Advanced Persistent Threat (APT) actor who seeks to target you specifically, or whether we're defending opportunists. It's much easier to defend against a passing opportunist by having modest protections which make you look secure relative to the competition.
Furthermore, identifying your appetite for risk (your risk thermostat) contributes to optimizing the outcome to appropriately cover the identified risks without being over-the-top.
- Implementation decision – using the information collected from this process, taking into account the constraints described and our stance on the risk, identify suitable technology and process changes to remedy the problem, or revise the constraints or risk profile if no solution can be identified.
Throughout the process, we remember security is a process, not a destination. We cannot "buy" security as a commodity off-the-shelf, and anyone who suggests as much is lying or has ulterior (likely pecuniary enrichment) motives.
Your specific problem
Your question is incredibly comprehensive, so we can follow our process in outline with your specific problems:
The problem
I have experienced a server compromise based on rkhunter analysis and want to mitigate the possibility of this happening again.
The specific problem is a past compromise of the machine, of which you wish to minimize any recurrences.
The primary goal I can identify from your question is to harden the machine against remote compromise events which may take place over a public network (such as the Internet). A secondary goal is to assure confidentiality and integrity of remote shell sessions to the remote machine.
Assumptions and constraints
Let's document these points to guide our solution:
- The public website service(s) exposed from the machine are secure
- The workstation(s) from which you initiate remote connections are not proxies for attacking the server machine in question. For example, they are not themselves infiltrated (so they are not a vector for keys leaking or being modified, or for the binaries used to make connections to be tampered with). You may wish to explore the security weaknesses of any client machines separately, or roll them into a single assessment.
- The server machine is physically secure and tampering with the hardware or software configuration by a person physically attending the machine is unlikely or discounted. (A machine to which an attacker has had physical access is unlikely to be your machine any longer.)
- The network is assumed compromised. There may be actors who have the ability to intercept or divert our packets.
- We want to use freely available software to achieve the technical aspects of our solution.
- We assume the users are adequately trained such that attacks on the wetware (human operators) can be discounted (e.g. social engineering threats). Again, in principle these are rarely mitigated adequately and are a weakness to most organizations, but this is Server Fault so besides a passing mention I will discount the non-technical aspects of the attack model.
- It is acceptable if the solution requires offline distribution or verification of keys prior to the first connection.
- Well-known cryptographic primitives are assumed to be immune to backdoor or non-publicly-disclosed attacks.
Threat model
I cannot adequately determine your threat model as I do not have visibility over your organization & infrastructure, nor do I possess an overview of the data portfolio you process or the private networks you may be connected to internally. From the public information in your profile, I can see you may work in a location which processes sensitive intellectual property on behalf of others, which would constitute a medium to high-risk collection of data for specific attack threats. (This threat may extend to personal systems which you operate.)
Implementation
Let's design a solution which meets our goals. To harden the machine, we need to consider public attack routes. It exposes two services, and we have assumed the web service is not vulnerable. Therefore, we must consider the remote shell connection.
For this purpose, SSH is more than capable of fulfilling your requirements without the added wrapper of a VPN session. Almost any Unix box is capable of running an SSH daemon, and a significant number are exposed directly or indirectly to hostile networks without infiltration.
How does SSH fit our goals?
Secure Shell (SSH) is designed to provide confidentiality and integrity of remote shell sessions. It does so using cryptographic approaches; in particular, hosts are assigned one or more host keys which can be used to positively identify the host to connecting client machines.
Man-in-the-middle attacks on SSH
As you have correctly identified, SSH is susceptible to a man-in-the-middle attack in a specific scenario: most users do not inspect the host keys presented by the machine on initial connection; they deploy a trust on first use policy. If a MitM can provide alternative host keys at this point, interception of the SSH session now and on future connections without further detection is feasible. Detection without inspecting the cached host key would require the MitM threat to be neutralized, or connection from an uncompromised network so the true host key of the remote host can be presented.
As we are concerned about MitM, we must design a solution to mitigate this. Options available to you include (non-exhaustive):
- Only connecting over trusted networks. This is not workable as it does not meet our goals or assumptions around connections over public networks.
- Distribution of the host key's fingerprint (or its entire public key) prior to initial connection. Use the
ssh-keygen command on the server to obtain the fingerprint, distribute this to users, and have them compare the fingerprint presented on first connection with the version on the server. They must only log in if the fingerprint matches.
- Publish the host keys in DNS and sign the zone using DNSSEC. Ensure all connecting clients use a DNSSEC-validating resolver and verify the DNS-based host keys. More information here. This approach avoids the burden of distributing and manually verifying the host key, but requires the presence of specific technical components which are not widespread on many networks yet.
Brute-force password attacks
Another vulnerability of a running SSH daemon is brute-force password attacks. It is common for attackers to probe your box for SSH services and attempt authentication using a common list of usernames and passwords. Boxes with usernames on the public list and weak passwords are likely to be compromised. Methods to mitigate this include:
- Switching the SSH daemon to use key-based authentication and disabling password authentication from the public internet. Generate an RSA keypair for your user account using
ssh-keygen with a large (e.g. >2048) number of bits, or an appropriate number of bits for a keypair signed with another crypto system.
- Using software like
fail2ban to watch your logs and add firewall rules to block further connection attempts from the same address after a failed login threshold is reached.
Would a VPN help?
VPN solutions may solve the same problem as you seek to solve with the SSH tunnel. They may use a different technical approach, or different crypto systems, but both are adequate to deliver on your security obligations. Both also incur similar overhead (e.g. the obligation to pre-distribute or verify the keys with each party in advance is identical).
The additional functionality provided by a VPN appears to be unnecessary in this particular instance if all you are seeking to operate is a remote shell server. Running a VPN likely carries additional risk through being another daemon running on your machine and a greater attack vector.
