9

As far as I understand, a problem with the idea of a pepper is that, if it's stored as part of your code, then the hacker can read it if they can access your code.

So I was wondering, would it not be better to store the pepper in memory? I'm thinking of running the server in such a way that the pepper is made available to the server's memory, without it being visible in the environment or in the process list or in the shell history. To obtain it, the hacker would need to run a memory debugger as the user that's running the server, or as root. Maybe even run the process under something like RamCrypt, to encrypt its memory at runtime.

The scenario is that of running a server on a cloud Linux machine.

In that case, is there any better way of storing a pepper? Or is there something wrong with the in-memory idea?

rid
  • 327
  • 2
  • 7
  • 2
    A pepper is commonly used to protect database data. So, the attacker has your database data AND your application code? There's not much left to protect... – Neil McGuigan Aug 19 '20 at 17:08
  • @NeilMcGuigan, I was thinking that if they have the database and the code, then they'd still be missing the pepper, which is not stored anywhere. So they'd need to extract it from the server's memory. Until they do, the code is useless and the data is still protected by the pepper. – rid Aug 20 '20 at 10:10

1 Answers1

12

First: you are definitely correct, it can't be stored alongside your code, no secret should.

It obviously needs to be held in memory by the server since it needs to be used. However, it can't be in-memory only. As long as a single password hashed with the pepper remains, it must be available. If it's only held in-memory, you'll have a different pepper for every server, and for every restart.

The real question becomes: where to persist it.

At this point, this is the same as storing any sensitive cryptographic material (eg: a private key). You have a lot of options, some of which are:

  • encrypt/store the pepper using a cloud KMS/Vault (eg: Google/AWS KMS, Azure Vault, etc...). Either directly or encrypt the pepper using a cloud KMS and store the encrypted pepper (the encrypted pepper can be stored somewhere less sensitive, but still with restricted permissions). But the key stays inside the KMS.
  • same as above with HSM.
  • roll your own secrets storage system. Things like keywhiz and Vault can be very powerful, but these are not simple systems to deploy and manage.
  • deploy the secret to the server at setup or startup time. This moves the responsibility for fetching the pepper but the ultimate persistent storage recommendations still apply (it just gives you more options).

The general idea behind all of those is that to have access to the pepper, you should show that you have control of the server. This does nothing to stop attackers with sufficient privileges on the server but that's not what peppers are for. If they have that level of access, they can just replace your binary with one that logs the plaintext passwords.

Marc
  • 4,091
  • 1
  • 17
  • 23
  • 2
    Storing a pepper in code, while obviously not ideal, still provides protection in the event of a database only breach. Also, neither a KMS not an HDMI store anything, so you can't store your pepper in either. – Conor Mancone Aug 04 '20 at 15:32
  • I did mention "store it directly" (which applies to some cloud kms/vault systems), "or encrypt it using ...". Fixed the language to be clearer . And we should never say it's ok to store secrets in code, ever. – Marc Aug 04 '20 at 15:35
  • 1
    We should never say, "it's never okay to do *this*" :p I do agree with your general point though. Storing a pepper in code is definitely not a good choice, but a pepper is also a tertiary line of defense. If that's the only place someone can find to store it, then it is better than nothing. Then again a pepper *is* a tertiary line of defense, so if you are trying to setup peppers but still have secrets stored in your code base, then your priorities are wrong... – Conor Mancone Aug 04 '20 at 15:51
  • 1
    @ConorMancone: point taken, I was being overly dramatic. But as you mentioned, if the pepper is in the code there's a good chance the private key for their certificate is too. Solving this problem has gotten slightly easier now that cloud providers have secret storage systems with decent permission and access configurations. – Marc Aug 04 '20 at 15:54
  • Thank you for the answer. Your last point is what I was suggesting - every time you start the server, you also supply it with the pepper in memory. The pepper would be constant, no matter where the server is started or how many instances there are, and it would be stored on the computer that starts the server, which is outside the network the server runs on. – rid Aug 04 '20 at 16:16
  • You'll still need to persist it properly, a single computer is not a reliable storage device. – Marc Aug 04 '20 at 16:17
  • 1
    Well, it doesn't *have* to be stored on that computer, it could be stored in multiple secure places, such as external encrypted storage media. It just wouldn't be available anywhere within the network the server is hosted on. – rid Aug 04 '20 at 16:20
  • If you have a secret storage system available, then yes. But you personal (or work) computers are not that. – Marc Aug 04 '20 at 16:57
  • 2
    @rid what you are suggesting certainly makes sense, but in a way just moves the problem one more "deep". Where do you store the pepper so that it can be readily available when a server starts? If you are talking about an auto scaling environment or the like, then it must always be available. Therefore you would need something like a secret manager that can supply the pepper to the new server as it starts. – Conor Mancone Aug 04 '20 at 19:18
  • @ConorMancone, what I was thinking was to have something external that would be in charge of managing the server(s). One idea would be to restart the server manually when necessary from a machine with physical access to the pepper. Another would be to have an external system that could monitor the server and restart it when needed, which would also have access to the pepper from a secure source. This system would not listen on any port and would only connect via something like SSH to restart the server. – rid Aug 04 '20 at 19:29
  • @ConorMancone, my idea was to try limiting the ways in which the pepper can be obtained as much as possible. You can't read it from the disk, you can't connect to a server to obtain it, the source code doesn't leave any hint as to where it got it, it just has it in a global variable that was somehow initialized when the server started. The only way to obtain it would be to read the server's memory, which can in turn be encrypted, making things harder for an attacker, or so I would think. – rid Aug 04 '20 at 19:54
  • 3
    @rid I think you need to consider your [threat model](https://security.stackexchange.com/questions/225012/what-is-a-threat-model-and-how-do-i-make-one). Every security feature has trade offs, so you want to find a good balance between "more secure" and "worth the effort". It's just as possible to over do security as it is to under do it. For instance, if an attacker has just a database dump then a pepper in the code base will stop them from reversing hashes. – Conor Mancone Aug 04 '20 at 19:57
  • 3
    If however an attacker has server access, then hiding the pepper won't help you because the attacker can just modify the application to have it send off plain passwords as users login. Therefore, jumping through hoops to protect the password in the event that an attacker has completely compromised your system is of limited value, and you're probably best off sticking to simple, tried-and-true, methods. "Typical" secret management will serve you well here. – Conor Mancone Aug 04 '20 at 19:58