47

I want to prevent potential theft of my web application (source code + database) by my local hosting company, that I don't fully trust for some reason (but have no other choice but to use as they give, by far, the best latency over here).

I'm planning to encrypt a partition with cryptsetup, and store the web directory + database files there.

The only issue is that I'll have to unlock the partition on every reboot (even uncontrolled ones), before I can start the web & database servers; but I guess I'm ready to live with that for now.

However, as far as I can read all over the web, encryption keys are stored as is in memory, and can be read given physical access to the machine, even from a dedicated machine using cold boot attacks. On a VM, I guess this is even simpler, as the hypervisor can take a snapshot of the whole server state (storage + RAM).

So as far as I can tell, it should be pretty easy to take a memory dump, and locate the encryption key at any time.

Is there any way to prevent this?

I know that controlling the hypervisor gives a huge advantage to someone attempting to steal data, and as such I'm not looking for a perfect solution here; but I'm rather trying to make it as hard as possible to get access to the data, so that it would not be economically viable to spend time & resources to get them.

I'm not really optimistic here, as encryption keys must be stored in some form in memory for decrypting, but I hope I missed something.

Edit - clarification

From my comment below:

I'm sure the hosting company's hypervisors are not modified in any way to perform any malicious operation; their business is hosting, not stealing things, and the company is relatively trustworthy. What I'm trying to protected against is someone bribing an employee to sell them a snapshot of the data. As long as the encryption key cannot be recovered from any snapshot the stock hypervisor can do, I consider the solution good enough for my use case.

BenMorel
  • 909
  • 1
  • 7
  • 13
  • If homomorphic encryption ever becomes viable I'm sure some hosting company will offer secure VM hosting. – user Aug 26 '19 at 17:27
  • 5
    @user Homomorphic encryption isn't magic "compute things on untrusted media" sauce. – forest Aug 27 '19 at 06:47
  • 2
    ... why bother with the encryption keys, when your code/database is going to be decrypted when running? That is, just snapshot the decrypted code directly. – Clockwork-Muse Aug 27 '19 at 20:23
  • @Clockwork-Muse If you're talking about the executable code (opcodes) residing in memory, database server memory pages, and disk cache, then yes, at least a part of the source code + database will be in memory when snapshotting. However, I consider the source code and the database valuable only as a whole, not as a 1,000,000 pieces puzzle that one would spend months trying to make sense about. As long the work to recover anything useful out of the snapshot is non trivial, I'm happy. – BenMorel Aug 27 '19 at 21:07
  • just rent a small apartment near that datacenter location (since it's known to give low latency to your target area) and get an appropriate internet subscription and host it yourself? – user1067003 Aug 28 '19 at 17:21
  • 2
    Have you tried asking the company what measures they take to prevent a malicious individual employee from doing something like this? – smithkm Aug 28 '19 at 17:52
  • deleted an applicable answer, proven to be working - because this question is somewhat hypothetical and based upon nothing but security paranoia... probably not even looking for a solution. –  Aug 30 '19 at 12:02
  • @MartinZeitler If only I could understand your comment, maybe I could respond. – BenMorel Aug 30 '19 at 15:17

9 Answers9

110

You can't, plain and simple. If you don't trust the hosting company, you don't host with them. This is law #3 from 10 immutable law of security:

Law #3: If a bad guy has unrestricted physical access to your computer, it's not your computer anymore.

The hypervisor always have privileged position over your virtualised machine, you can't protect yourself from rogue hypervisor.

One way you can do to benefit from their network location but still keeping your code secure is to do a colocation hosting. In a colocation hosting, you bring your own machine that you put in someone else's datacenter.

Basic colocation itself should discourage most opportunistic attackers, but a targeted/determined attacker might connect into your machine anyway. If this is a concern for you, you should put your machine in a tamper-evident/tamper-proof server casing, so that the only unlocked port to the outside world is the power cable and network cable, and you encrypt all network traffic going into/out of the machine. This way you can put your machine in their network and benefit from the network's low latency, and they won't be able to steal data from your server unless they break the server's chassis.

If protecting your data is very important, you would also want to do a regular audit that the case hasn't been tampered with. If you have infinite budget, it may also be possible to design tamper proof chassis that have various sensors that can raise an alarm and initiate shutdown (discarding all sensitive/encryption keys) if they detect that the chassis had been tampered with, but of course these gets expensive really quickly.

Lie Ryan
  • 31,089
  • 6
  • 68
  • 93
  • 36
    ...and the next step up, of course, is to set up your own hosting facility, complete with whatever physical (locks, doors, ...), electronic (alarm systems, surveillance cameras, ...) and human (guards, receptionists, ...) security elements are needed to ensure a proper level of security. Of course, that gets expensive real quick, but hey, what's the price for a good night's sleep anyway? – user Aug 26 '19 at 09:37
  • I'm on a budget, so I'm looking for a "good enough" solution that will just make it not economically viable to attempt to decrypt the data. What about [TRESOR](https://www.cs1.tf.fau.de/research/system-security-and-software-protection-group/tresor-trevisor-armored/?q=content%2Freadme), which seems to keep the encryption key in CPU registers and not in RAM? – BenMorel Aug 26 '19 at 09:48
  • 10
    @Benjamin: TRESOR assumes that the hypervisor is trustworthy. A rogue hypervisor can fake it so that your program thinks it's running code on the CPU, but turns out your instructions are actually running on an emulated CPU whose behaviour and debug registers is fully controlled by the attacker. – Lie Ryan Aug 26 '19 at 10:10
  • @LieRyan I could live with that. I'm sure their hypervisors are not modified in any way to do these kind of things, as their business is not stealing things, and the company is *relatively* trustworthy. What I'm trying to protected against is someone bribing an employee to sell them a snapshot of the data. As long as the encryption key cannot be recovered from any snapshot the stock hypervisor can do, it's good enough for me. – BenMorel Aug 26 '19 at 10:16
  • 9
    @Benjamin: the host machine is privileged over the guest machine. It's pretty simple for the hypervisor running on the host machine to read the debug registers of the guest vcpu. This isn't even difficult as many hypervisor supports guest debugging API that includes reading debug registers among other things. Even in real hardware, the attacker can attach a JTAG debugger to read out the debug registers. These type of attacks sit well outside TRESOR's threat model, but will be relevant to yours. – Lie Ryan Aug 26 '19 at 12:48
  • 17
    That. If I were that rogue hosting company employee... well, I wouldn't take the bribe, but if bad guys had a member of my family hostage, I *absolutely could* obtain a register dump. And could do it in a completely undetectable way, assuming that the hypervisor in question supports live migration or (full-system/live) snapshots. Dump the (emulated) hardware state, copy that data to personally-owned media, and come back to analyze it later. And I'm not a pentester or RE specialist, just a systems engineer with some sysadmin and C experience. – Charles Duffy Aug 26 '19 at 17:32
  • @LieRyan Well, some boards actually have an option to disable JTAG physically by burning stuff inside chip, but you hold your point as it only makes stuff much trickier. – val is still with Monica Aug 26 '19 at 23:42
  • 1
    Strictly speaking, you could use remote attestation and run all the sensitive computations in an SGX enclave (and SGX disabled JTAG), but it would be difficult to accomplish, needing a complete rewrite of the hypervisor. At least it would move the trust from your VM host to Intel's hardware. The only full solution that I'm aware of is vCage, which is commercial and quite expensive. The only issue with it is that it does not disable JTAG, as far as I am aware. – forest Aug 27 '19 at 06:40
  • 1
    @LieRyan TRESOR does not work correctly under a hypervisor. I've worked on a patch to get KVM to support TRESOR without a huge performance hit, and it's non-trivial. But you're right that it doesn't protect from a malicious host anyway. It only protects from RAM being _read_ (e.g. a cold boot attack). – forest Aug 27 '19 at 06:41
  • @val It's the chipset that supports that, but usually you need the Intel chip in manufacturing mode to set those OTP fuses. However, pretty much _all_ Intel CPUs don't allow JTAG without a specific, per-serial number password that can only be obtained through signing an NDA with Intel. But an NDA is not a security barrier! – forest Aug 27 '19 at 06:46
  • 1
    @Benjamin: [Here's Jeff Atwood](https://blog.codinghorror.com/the-cloud-is-just-someone-elses-computer/) on the colo prices he is paying. _Someone_ still has the potential for physical access to your computer, but they'd have to tinker with your hardware to get a memory dump. – Michael Aug 27 '19 at 12:51
  • There are chassis with an open switch that notice when the side panel has been opened. That's useful to detect when someone might have stolen the RAM out of a desktop in a computer lab or something (if it logs in the firmware without needing the CPU powered on), but probably also for this if software can be notified. – Peter Cordes Aug 28 '19 at 07:09
  • @aCVn there is one step in-between: hire a guard that sits next to the physical machine placed into their datacenter ^^ – Frank Hopkins Aug 28 '19 at 19:38
16

so that it would not be economically viable to spend time & resources to get them.

I hate to break it to you, but you simply aren't that important. No-one knows you or your web app. So it's already not economically viable.

Consider the cost-benefit. For the hosting company, if this happens once, they'd hemorrhage customers. So they're going to have proper logs. Any employee caught doing this will not only be fired, but also arrested. So the bribe would need to be serious. We're talking millions, because it's got to compensate the employee for being imprisoned and never being able to work again when released. And if that's the bribe, your company must be worth at least 10x that. That values your company in the tens of millions, and more likely hundreds of millions.

Honestly, is your company worth that? Everything you've said so far says it isn't. And if it is, you have no reason not to run your own servers.

Graham
  • 581
  • 3
  • 7
  • 22
    -1 for _I hate to break it to you, but you simply aren't that important_. The people who assume that are the people who end up being screwed over the most. Even if OP is not that important, OP might have access to something that is (maybe he used to work at a major ISP, and an attacker is looking for ex-employees who might have old credentials saved). – forest Aug 27 '19 at 06:43
  • Yes and no. Sure, you might not be important yourself but you might be just a stepping stone in a [watering hole attack](https://en.wikipedia.org/wiki/Watering_hole_attack) – Gaius Aug 27 '19 at 08:59
  • 2
    You're perfectly right, the company is rather small and the project is not that important. However, the database contains a decent number of customers from the whole island, and the app took many thousand man-hours to write. Plus, I wouldn't trust that the company is actually keeping logs of their employees' actions or that the employee in question is risking anything at all (let alone prison) if he did break into our server. Therefore I wouldn't be surprised if someone actually sold a snapshot for a couple **thousands**, not millions. THIS is what I'm trying to protect against. – BenMorel Aug 27 '19 at 09:58
  • 3
    @Benjamin Then you need to check that the company *is* keeping those logs, and you need to verify that the audit mechanism exists for you to see all administrative actions on your VMs. Especially if you're in a country with privacy laws (GDPR is the obvious one; do you do any business with European clients?), your clients' security depends on your hosting company's security. And if you really have that low an opinion of your hosting company then you shouldn't be using them, regardless of their latency. – Graham Aug 27 '19 at 11:50
  • @forest That's hypothetical what-if stretching. The OP is looking for security on his current project. Sure, he's put a lot of effort into it himself, but that doesn't mean he's a significant player in that business, nor that his competitors care enough to go to illegal lengths to sabotage him. Ransomware would be a possibility, but then as a small startup he isn't going to have the cash to pay off anything much (otherwise he wouldn't be using this hosting company). And if the issue is the security of clients' confidential data, he simply shouldn't host that data on an untrusted server. – Graham Aug 27 '19 at 11:56
  • @Gaius How? This isn't about user security, it's about server hosting security. – Graham Aug 27 '19 at 11:57
  • 4
    @graham the assumption that you won’t be attacked if you’re unimportant is unsafe – Gaius Aug 27 '19 at 12:04
  • 3
    @Gaius The watering hole attack isn't relevant to his situation. And whilst I agree that it is unsafe to assume you won't be attacked, safety is *always* a function of cost and probability. As an engineer where "safety" on my projects has meant "people die", I'm more aware of that than most! :) The OP needs to decide what is "safe enough" given the probability of *this specific attack vector*, and unless there are other factors, the probability looks vanishingly low. Of course if his company makes it big then he needs to reassess, but then he can probably afford better hosting too. – Graham Aug 27 '19 at 13:04
13

In theory, you should be able to use the trusted hardware features of modern CPUs to run your disk encryption, or even your entire VM, inside a tamper-resistant part of the CPU, having all the data on disk and in memory encrypted with keys that are only accessible inside that tamper-resistant trusted enclave.

While exposing Intel's SGX trusted computing system to VMs doesn't appear to be possible on stock hypervisors, AMD has a feature called Secure Encrypted Virtualization, or SEV, which sounds like exactly what you are looking for: you can set up your VM in such a way that it is protected from the hypervisor with keys known only to AMD or someone who is willing and able to disassemble the host CPU.

Unfortunately, it's unlikely that the particular hosting company that you need to use provides AMD SEV-supporting hosts or makes the feature available to their clients.

interfect
  • 305
  • 1
  • 4
  • Interesting, thanks. Indeed there is no chance that this hosting company uses this, though. – BenMorel Aug 26 '19 at 22:01
  • 3
    Anyway, regarding the SEV thing, quickly reading about it, it doesnt seem its provable that the host is really using it. What would stop a malicious server running the VM on an emulated system that pretends to use SEV but just fakes the appropriate instructions? – Vality Aug 26 '19 at 22:12
  • @Vality if you hosting provider gives you access to hosting machine and you are the one who setups server with SEV, this _might_ work. – stiv Aug 26 '19 at 22:27
  • @Vality As described in https://github.com/AMDESE/sev-tool#guest-owner it looks like you encrypt (the sensitive part of) your VM with a key shared between you and the trusted hardware in the CPU, so the data can only be used in a guest running the image you specified. Image verification is accomplished somewhat like secure boot, I think, except instead of the TPM verifying measurements about the boot process of the host, you yourself verify and approve measurmenets about the boot process of the VM. – interfect Aug 26 '19 at 22:56
  • 1
    @Vality You can verify it just like you can verify that a TPM is real and not emulated (for a TPM, that's through its Endorsement Key). I'm not sure exactly how SEV does that, or even if AMD supports remote attestation beyond simply checking firmware, but it might. – forest Aug 27 '19 at 06:48
4

Microsoft has a possible solution for this called Shielded VMs, these are specifically intended for the attack vector that some malicious actor has administrative access to the hypervisor. An example of this would be machines deployed either in the cloud or in a colo.

The downside is that you will need to maintain a Guardian Server under your physical control. That may be overkill for your needs.

Gaius
  • 810
  • 6
  • 7
  • 1
    Even more overkill is IBM Secure Service Container. It runs on a LinuxONE machine (cost starts around 100k, If I recall correctly)... – ThoriumBR Aug 26 '19 at 19:05
  • Unfortunately I have absolutely no control over the choice of hypervisor! – BenMorel Aug 26 '19 at 22:00
4

I don't fully trust for some reason

Trust them in what way? Do you believe they may be incompetent or do you think they may be malicious?

If merely incompetent, you'll probably get reasonable protection from the encryption you've described. It is of course possible that their host machine gets compromised and the attacker finds a way to grab the current encryption keys from RAM, but in typical web host compromises the attackers won't go that far. They'll just find an easy place to dump their malware or easy to grab passwords.

But if they are actively malicious, you cannot protect the data on the VM. At best, you could configure it as a proxy server, which could give you the latency benefits without exposing your database and source code.

jpa
  • 951
  • 6
  • 11
  • As clarified in the question, I was thinking of a malicious employee who would be bribed to get a snapshot and hand it to someone for money. Regarding the proxy server, I thought about this idea as well; this may be a reasonable fallback if everything else fails. Obviously not everything can be cached by the proxy, especially in a web application, but if it gives a 80% or 90% of the perceived performance, this may be good enough. This would require an in-depth review of the application, though, as it's not been built with caching in mind so far. – BenMorel Aug 26 '19 at 21:59
4

Lie Ryan's answer is the right one, but beyond colocation, which still has physical attack surface at the host's premises, one additional possibility is using the hosting provider just to get you a public-facing IP address, and tunneling that to an on-own-premises server via a decent home or business class broadband link.

If you want to get really fancy, you can even do a mixed approach where the private keys for certificates never leave your own on-premises server and the server at the hosting site only has access to generate session keys. Cloudflare offers something like this; I'm not sure exactly how it works though. I think it's even possible (not with theirs, but if you roll your own, making fancy use of cryptographic primitives) to have the data at the hosting site, but encrypted with symmetric keys that are never on the hosting site, and handing off the connection to the hosting site in such a way that only the client, but not the hosting site, can decrypt the data.

R.. GitHub STOP HELPING ICE
  • 7,813
  • 1
  • 23
  • 34
1

The cloud ultimately is someone else's computer, and someone else's rules. If you are looking for physical access security, the best path may be to host your own physical server, or get a dedicated physical server. With all the tools available to orchestrate this today, and the likely fact that you will be wanting to host projects for a long time, it's not a bad idea..

Jas Panesar
  • 111
  • 2
0

Your question seems to switch pace a few times, but if your main concern is someone cloning your work, the following should be simple protection:

Make sure there is nothing interesting to steal

Rather than letting the source code for everything live on their machines, work on the code in a different environment and deploy only the binaries.

This should prevent anyone from copying your work as they cannot update the binaries, if you are really paranoid you could even put a time bomb in the logic to make sure things stop working even if they copy the current version.

Keep in mind that even if a sizable amount of your code does need to live in the server, it already becomes financially unattractive if you are able to block anyone from using one or two core parts. Of course they can be reverse engineered but this is likely prohibitively expensive.

This solution of course does not offer protection against data leaks, but based on your question it seems like preventing competition is more of a concern than data becoming public.

Community
  • 1
Dennis Jaheruddin
  • 1,715
  • 11
  • 17
  • 1
    The code is written in PHP, so source code == production code. There is currently no way to ship only the compiled opcodes (that are machine-dependent), and even if there was, decompiling opcodes would probably get you something very close to the source code in this language. PHP code obfuscators are usually nowhere near production quality, and ruin any chance of debugging any issue on the server itself. And finally, I don't want the database stolen either! – BenMorel Aug 27 '19 at 15:06
  • In that case I understand it will not work for you, but will leave this answer here as it may help future visitors with the same question in a different setting. – Dennis Jaheruddin Aug 28 '19 at 09:06
0

With AMD's SEV it is possible, but your cloud provider will have to support this: https://security.stackexchange.com/a/241566/84564

Ole Tange
  • 301
  • 1
  • 9