A previous IT worker probably left some backdoors. How can I eliminate them?

Question

I started working for a company that fired a previous IT worker for leaking data.

I can only say the following things:

We use a Firebird DB with an application written by another company, Proxmox, for virtualization of Windows Server 2008 R2, SQL Server, cloud core Mikrotik router, and a few other Mikrotik devices.

I am not 100% sure, but is there some quick way to check if there are some backdoors left, without interrupting internal processes and reformatting everything?

This previous guy was really good, having written software in C++ and C#. I also know that he did some assembler and cracked a few programs in ollydbg.

Answer 1

The only way to be absolutely certain is to wipe every system clean and to reinstall from scratch. You will also need to audit all of the locally generated software and configurations to ensure that they do not contain backdoors. This is a non trivial task which comes with a non trivial cost.

Beyond that then there isn't really much you can do.

Obviously while you're deciding what to do

• Audit all firewall rules for validity
• Audit all accounts for validity
• Audit all sudoers files for validity
• Change all passwords and keys

but that is only scratching the surface.

Answer 2

First, the most important goal of a fired sysadmin is to clean up his past, particularly if it was a departure in bad standing. If an attack on his previous system were to happen he wouldn't stand to gain anything (particularly not his old job), but he could lose a lot. I've been faced with similar fears many times, but in my opinion, they are largely unfounded. I think it is way more likely that if you were to pose him some questions, he would be very nice and helpful to you (which you should then in turn mention to your boss).

---

Now consider the very unlikely case, that he really wants to do something harmful.

Make an archive of your whole network into some - for him - unreachable location (behind a firewall out of his responsibility, hard disk in a locked cabinet, etc).

As soon as you've made this backup, he can't cover his tracks any more. In the case of a rogue attack, it will serve as evidence.

---

You can't ever be 100% sure (except in the case of a reinstallation of the whole network). You can pass through a checklist, but you can't ever be sure that you've checked everything.

Even in the case that you find a hole, you can't prove that it was placed there intentionally. Note, the same problem exists for software developers. Bad work is not a criminal offense, and you can't prove that he "forgot", for example, to change the default database admin password, intentionally. Or that users, whose password was set by him, "accidentally" were granted the privileges required to connect to your top-secret databases.

---

In most cases, the most important part of your system isn't your OSes, but the data which is managed on them. It is particularly so, if this data is private data and the property of your customers. He could have stolen them long before his last workday, encrypted them and saved them in places known only to him. So make sure to also check for traces that indicate that he made copies of your data before he left. Note though that if he did it "properly", you won't find anything.

Answer 3

@JonasWielicki compared this question to one of our canonical security questions (How do I deal with a Compromised Server). I stand by my answer to that question but there's an important difference.

In that question the server was known to be compromised. As far as I understand this question that hasn't been established in this case. As such, I'm not sure we need to be breaking out the flamethrower just yet.

People leave an organisation all the time without anything bad happening, even when they've left under poor terms. Simply being good at programming, which is all the "evidence" you've shown us in the question OP, does not mean that someone is a hacker per se, nor that they're likely to attack you now that they've left.

If you're truly concerned then I would suggest hiring an outside security firm to audit your system. Not only will that hopefully uncover any little surprises that may have been left by the former sysadmin, it will also serve as a good baseline for all your security issues and concerns going forward.

Answer 4

The only way to ensure that no backdoor exist for sure is nuking the system like you said.

If that isn't entirely possible,

1. Consider a baseline secure configuration and analyse of the current configuration is deviant from that.
2. Verify all the suid programs.
3. Analyse all the running processes.
4. Perform a portscan on the system to identify open ports and services.
5. Routinely monitor all outbound and inbound connections and look for rogue connections.

Answer 5

You're going to need to decide how sure you want to be. The cost-benefit is never going to pan out for nuking from orbit.

Are managers demanding assurances or are you just trying to do some reasonable examination of the systems you've inherited?

If it's managers, now you get to find out how reasonable they are. Are they willing to settle for "pretty sure"? Maybe you can follow the fired guy to his new job!

If you're looking to examine your own systems, I'd start with setting up a network monitoring system, like snort. Look for unexpected traffic, like " why is the system talking to this one server in Russia every day?" or "why are people doing IRC over my web server?" (that one happened to me).

I think @peterh's suggestion about making a big archive is a really good idea. I also think his suggestion about the fired guy being helpful is totally realistic. Problems like this turn out to be stupid management 90% of the time.

Answer 6

You should do an audit of current functionality and usage, but honestly you should start from scratch.

Meaning, don’t just “nuke it from orbit” but make note of all core functionality and requirements such as—for example—what ports might need to be open on the Firewall for the box, IP address or hostname restrictions and the sundry “plumbing” like that.

Then recreate a new environment based on what you saw, run some tests to confirm functionality is the same if switched over and then schedule maintenance to do an official switchover: Meaning copy data from the old setup onto the new one and then either switch IPs on the box to take over the old IPs from the old box or setup other systems that need access to be able to connect to the new setup.

But that said, even if you do that then you might not be able to trust the data within the database, but at least in a case like this the core system is moved over to something more stable and “sane” so you have a cleaner starting point.

Also, you mentioned the previous I.T. folks were able to program in assembler and C++. My question would be: Why? So do an audit of any custom code that might exist and assess functionality. Because while the other people might have been “fancy” in their programming skills, who knows if they programmed something in C++ that—for example—can easily be recreated in Python or a Bash/Batch script. I’ve met lots of C++ programmers nowadays who are really over-using C++ code when simpler tools can be used to achieve equivalent functionality.

But at the end of the day, rebuilding architecture from the ground up might be the only—and safest—thing to do.

Answer 7

The first thing, is everyone change their password. Second, you need to secure the firewall, and anything that directly connects to it. Change ALL passwords everywhere.

If he can't get through the firewall, he will have to rely on something establishing a connection to him, and make getting back in much harder.

You will have to learn everything about the firewall,rules, and configuration and check them line by line for things that don't belong. Even then you might be better off getting a new router, and transferring the config over manually. Verifying that each change to the configuration is both correct,not a backdoor, and that it is even still needed. This is the perfect time to clean the cruft out of your system.

Answer 8

Consider: he could have installed his back-door into the boot sector. Depending on just how good he was and how cautious he was about getting a persistent backdoor, you might need a new hard drive and motherboard. And depending on exactly what he has done, you might have to leave all your data behind. If it was me making the decision of what to do, I wouldn't look at the person for clues about what he may have done; I would act on the risk of damage that the data you have tucked away in those databases could potentially do.

If you want a neat tool to categorize by risk, the Canadian Military has a pretty neat method for that:

Data Protected "A" - data that can cause significant damage to an individual without risk of damage to others using the same system.

Protected "B" - data which can cause significant damage to any more than 1 person, but is still limited to people whose data is stored in the same system, but without risk of damage to the organization which carries the data.

Protected "C" - data which if in the wrong hands poses a threat to national security, organization-wide security, or "root" access to the entire database where the data stored can cause the same widespread damage.

So, if your databases house personal information, and your server belongs to an entity who is legally responsible to people to keep that data confidential, then I would trash anything that can store data and start fresh. It's a hefty cost in most cases, but if there's potential for a law suit, then lawyer up first and go from there. Find a lawyer who specializes in data security breech. The law for these sorts of things can be very muddy, so it's best to get a lawyer who's very experienced, or to just trash your hardware and start fresh. Hope this is useful. Best of luck and sorry to hear about your predicament.