19

We are engaging a consultant in India as our Linux Administrator. We don't know him well and he requires Root access to all our servers to do his job (including a security audit).

What is the best practice for enabling a remote consultant for such work such that we are protected against any malignant activities?

Thanks in advance.

Anderson Tess
  • 315
  • 2
  • 3
  • 66
    If you don't trust someone, do not give them access to your servers. Period. Hire someone you can trust. – EEAA Feb 08 '17 at 00:39
  • 6
    Can't help wondering if this is an action being mandated by higher-ups and you're looking for ammunition/good arguments against it? – Matt Feb 08 '17 at 03:04
  • 5
    LOL... Is this a good idea? – ewwhite Feb 08 '17 at 03:59
  • 1
    https://serverfault.com/questions/805333/linux-set-up-for-remote-sysadmin – user9517 Feb 08 '17 at 07:35
  • If you want to have a particular stack setup, probably the best way is to get them to create a Dockerfile for you, and send it. You can then carefully review it, and see anything fishy in seconds. – AKS Feb 08 '17 at 17:52
  • 5
    If you're giving somebody root access to your servers, you have absolutely no way at all to systemically protect yourself from malignant activities on the machine(s) the individual has root access to. – Craig Tullis Feb 08 '17 at 22:01
  • 3
    Hope you have good offline backups – CaffeineAddiction Feb 09 '17 at 20:52

5 Answers5

55

Don't. Also, you're in as much danger of ineptitude as malice from what I've seen of the typical way companies handle this.

I'd like to say, there's probably great system administrators out there in India, but the way many companies do things is terrible.

If you're going through a body shop, you're also likely seeing a pretty big cut go to them, and many of them are unlikely to have properly vetted their employees. I've talked to three, one of whom I worked for and none of them have done any technical interviews.

So, if you must hire someone remotely, for god's sake, interview him yourself and make sure he knows his work. System administration is far too important to hand over to someone blindly

Now that I've handled the "ineptitude" part of it,

Administration is a pretty broad phrase. And someone with root access can do anything. Now, personally I think creating an account for the admin, and giving him the ability to elevate himself through sudo is a better idea (which your config management system should handle if you have many servers). That said, even that relies on a certain amount of trust. There's so many stories out there of the sheer damage a disgruntled sysadmin can do. Change all your passwords? Sure you could get in eventually, but its not trivial, and it would probably cost more than you're saving.

So, consider a local. If not, consider someone you have vetted yourself and have directly hired.

Journeyman Geek
  • 6,969
  • 3
  • 31
  • 49
  • 35
    I have a hard enough time giving privileged access to **the guy one door down from me** let alone someone 12 time zones away. My mind boggles that anyone would even consider this as an option. – EEAA Feb 08 '17 at 01:32
  • 7
    If you're going through a body shop, you won't even know who is really in possession of those root credentials, you have no guarantee that the person working on your systems today is the same person who will be working on them tomorrow, you have no guarantee that these guys aren't literally going to be emailing your root-level password(s) around to each other in the clear (I've seen it way too many times), and so on. – Craig Tullis Feb 08 '17 at 22:15
  • 2
    Nobody should log into a modern Linux distro as root. The root account shouldn't even have a password. Instead, there should be users who have `su` authority to elevate themselves to root. When someone says they need "root access", this is what they should be asking for. If this person says he needs "the root password", he's not competent to do the job anyway. – Monty Harder Feb 08 '17 at 22:53
  • @MontyHarder do you mean (certain) users should have `sudo` authority to elevate themselves to root? If not, could you outline your best practices for using `su` to elevate oneself to root without having, and distributing, a root password? – MadHatter Feb 09 '17 at 08:08
  • @MadHatter What I mean is that the people authorized to perform administration on the server should have the `sudo` authority. Typically this is done by making them members of a group (traditionally known as "wheel") which is specified in `/etc/sudoers` as having the ability to use `sudo` to become root. Where I work, we use Likewise to allow authentication against Windows Active Directory, where a group of administrators is defined. Then that group is specified in sudoers. That way, our Account Administration team has a single point of control over who has that authority. – Monty Harder Feb 09 '17 at 15:44
  • 2
    @MontyHarder that makes a lot of sense, it's just not what you said the first time around; `sudo` and `su` are two completely different things. Thanks for clarifying. – MadHatter Feb 09 '17 at 15:45
  • @EEAA: How do you know India is 12 time zones away from the OP? – Lightness Races in Orbit Feb 11 '17 at 20:20
  • @LightnessRacesinOrbit I don't. 12 hours away from *me*, not the OP. – EEAA Feb 11 '17 at 20:22
  • @EEAA: I see, so personal story rather than directly related comment on the OP :) – Lightness Races in Orbit Feb 11 '17 at 20:23
33

As has been mentioned, don't do this.

The only way you'll be able to protect yourself is by doing something like this:

  1. Insist that the consultant use a configuration management system of your choosing.
  2. The consultant will write configuration management manifests for the actions you need completed.
  3. The consultant will test the manifests on a test system.
  4. When ready, the consultant will commit the configuration to a code repository.
  5. All changes are reviewed by either a member of your staff or another consultant that has absolutely no relation to the first, and has no way of contacting them.
  6. Once the changes are signed off, they are applied to the server by you or a member of your staff. The original consultant should not have access to any of your systems.

As should be clear, this is a very clumsy and inefficient process, but if you insist on accepting work from a non-trusted individual, this is one way to handle things.

As I recommended, though, you're much better off hiring a known, trusted individual.

EEAA
  • 108,414
  • 18
  • 172
  • 242
  • Why not? I am working remotely, managing around +300 dedicated servers, within one hour I could destroy everything if I want. But of course I wouldn't do that,even if I get fired. I am responsible for making backups, have highest privileged (not me only, couple of us), and we can get malicious anytime. The reason why we don't - is moral and ethics. We love company and employees and our job in general. Main thing here is to trust someone, and to find moral person for this job. – fugitive Mar 11 '17 at 17:32
  • @fugitive You're speaking of a different situation. I presume that you are trusted by the company you're consulting for, otherwise they would have not granted you the permissions you have. In the case of the OP, it is clear that they do not trust this consultant, which is why I recommended against giving this person permissions on any systems that matter. – EEAA Mar 11 '17 at 17:36
  • Well, trust have to be earned. :) – fugitive Mar 11 '17 at 20:30
10

What is the best practice for enabling a remote consultant for such work such that we are protected against any malignant activities?

From a legal perspective: due diligence beforehand and strict penalties for breach of contract.

You start with the usual good hiring practices that also apply when hiring on-premise staff (and/or service providers) which include fact checking the provided resume, ask for education transcripts and certification numbers, check and call their references, interview, maybe even a background check or security screening etc. etc.

Then apply the carrot: pay fair value, offer attractive work, amazing colleagues, good working conditions and benefits etc. (if you pay peanuts you get monkeys.)

And the stick: breach the terms of your employment/service contract and we'll sick our lawyers on you and leave you bankrupt!

Unfortunately both the above become increasingly more difficult when crossing borders and timezones.

Once you decide on hiring somebody:

  • clear directives and policies, people should be aware of what they should and may not do.
  • the principle of minimal access applies, make it difficult for people to (accidentally or on purpose) do things they shouldn't. For the typical system administrator often that still means full access, but a security auditor for instance should not need full administrator access but can simply request the existing admins to run a script on his behalf that collects the details he needs to make his report. Such a script can easily be checked beforehand.
  • trust, but verify. Simply have the existing staff check the work of a new joiner and as always collect audit information.
  • etc. etc.

This question details what I usually ask my clients to do to establish remote access for me, which might be a starting point for you as well.

Community
  • 1
HBruijn
  • 72,524
  • 21
  • 127
  • 192
  • 3
    *"if you pay peanuts you get monkeys"* Or elephants. Though come to think of it, I'm not sure if that's better or worse than monkeys. – user Feb 08 '17 at 15:54
  • Just to add, the "stick" part of your answer might be harder to enforce if the other party is in a different country. You should consult with a knowledgeable/experienced lawyer first to see what possibilities you have in case something goes wrong. – user121391 Feb 09 '17 at 10:31
  • Also, enforcement after an incident probably sucks more than working with a party you can trust in the first place. By the same token there are people you totally can trust whom you might not automatically be inclined to, and people you instinctively gravitate toward whom you probably shouldn't trust at all. – Craig Tullis Feb 09 '17 at 15:00
7

There is one systemic method of protecting yourself that comes to mind, which I have not seen mentioned.

Host your Linux instances as VM's on a virtualization hypervisor (VMware, Xenserver, Hyper-V, etc.).

DO NOT give the remote admin administrative access to the hypervisor. The remote admin would only get root access to the VM's themselves.

DO Implement a hypervisor based backup system (Unitrends, Veeam, vSphere Data Protection, etc.)

DO keep at least one snapshot per day of each Linux VM, going back as far in time as you feel is necessary.

DO NOT give the remote admin write access to the backup repositories.

If you do these things, you will have backup snapshots of each Linux instance over which the remote admin has no control. If the remote admin does something hinky, whether intentionally or accidentally, you can always mount a backup from before the hinkeness occurred to evaluate what happened and possibly recover to a clean state.

This won't be proof against a hypervisor side-channel attack, which could potentially be mounted from within a VM that the attacker has root access to.

If your backups don't go far enough back in time, this won't protect you.

You need to thoroughly trust whomever is in control of your hypervisor and the backup infrastructure.

If you're doing this in the cloud (AWS, Azure, etc.), the implementation details will differ, but the general concept would be the same.

In essence, divide responsibilities among parties who are not business partners with each other, in addition to only hiring people you trust.

Craig Tullis
  • 488
  • 3
  • 14
  • 1
    By the time you've done all that, you're the system administrator already and you're hiring a remote lackey or PFY. – Criggie Feb 08 '17 at 22:38
  • 3
    Not totally. You're administering just the hypervisor and the backups. Which, admittedly, isn't nothing. But it isn't necessarily the same skillset or administrative burden, either. Chances are that if you're doing virtualization (we are), that you have a different person or people in charge of whatever is happening on the individual VM's anyway. – Craig Tullis Feb 08 '17 at 22:45
  • 1
    Although the general idea is good, it only helps against incompetence/errors (deleted the production database etc), not against malice (unless you check every change every day and compare the contents of the changes, essentially a daily full audit). Also, the damage might be unrelated to technical stuff, for example customer data or trade secrets siphoned off cannot be contained this way, as it is only reactive. – user121391 Feb 09 '17 at 10:27
  • @user121391: I can't really disagree, particularly with the data exfiltratiion issue. Once the data is taken, it's completely out of your control. – Craig Tullis Feb 09 '17 at 14:55
-1

Give him his own user account. Then find out exactly what he needs access to and grant just that access but nothing else. For example if he needs to reconfigure an Apache web server, use an ACL to give him write access to the Apache configuration files and configure sudo to let him restart the Apache service but not execute any other commands as root. As always, keep backups of anything that you're giving him access to (in this case, your Apache configuration files).

Micheal Johnson
  • 151
  • 5
  • 3
    Having permission to configure and restart an Apache running as root is essentially giving out root privileges. It's easy enough to have an arbitrary binary executed by the apache process, e.g. to pipe logs to that. Better set up Apache so it can run as non-root and still bind to privileged ports. [That's what I did](https://gist.github.com/gagern/00c6168af466964022715737fd41ea61) in this situation in the past. – MvG Feb 09 '17 at 12:31