Is there something insecure about storing secrets in plain text on the host FS?

Question

There are recommendations on this website and in the internet suggesting to never store credentials (e.g. login/password to the database that certain web application is using, or S3 access key on non-AWS instance, etc.) in plain text on the filesystem, apparently due to possibility to recover password from the disk.

The alternative suggest transmitting secrets from a some sort of remote secret manager over the network, I believe this model expects application that makes use of corresponding secret to store it in memory. Variations of this scheme also support storing secrets in encrypted files on the target host's filesystem and transmit encryption key to the application in some secure way over the network.

Given that I'd still like to store credentials unencrypted on the local filesystem - let's say due to framework limitation that expects unencrypted password in some local file - would it be possible to still store secrets in plain text in certain file and mitigate the risk by:

1. Using encrypted filesystem - so malicious actor who steals the disk still won't be able to read secrets on the encrypted partition.
2. Using ram FS (let's forget about necessity to transmit secrets first for now).
3. In addition to either 1 or 2 limit access to file to a certain user which is used to run the application.

P.S. there are similar questions about this matter, e.g. Storing database password in plain text? (this one is perhaps too vague and therefore it did not receiving a clear answer) and Is it okay for API secret to be stored in plain text or decrypt-able? (more broad in scope; the answers though didn't address particular aspect I'm asking about).

In addition, I want to clarify that (1) password IS NOT stored in the application configuration, let's just say it is distributed via some secure mechanism to the host during application's deployment; (2) end user who uses web application I'm referring to does not have access to the host.

My primary question is whether there is something inherently insecure in filesystems that essentially leaves me with options that expect me to never store credentials (for accessing other services from my applications) unencrypted on any filesystem. In order to narrow down the scope, let's limit this question to modern Linux OS.

Answer 1

OWASP comments on this in Cleartext Passwords in CATALINA_HOME in the context of managing Tomcat's web server database credentials. Their view can be summarized as

best practices is to never to store clear text passwords, but it is very difficult to avoid

Their primary recommendation is to use file system security to secure the credentials file and limit the database user permissions to the minimal required.

If you have heard the expression, "it's turtles all the way down", it nicely summarizes the OWASP view that hiding the password in another place isn't really resolving the key issue and may provide a misplaced trust that the issue has been solved.

Enterprise security teams are sometimes humorless and likely lacking in appreciation for philosophical arguments involving turtles and the problem of infinite regression. Therefore, there are several password vault solutions with integration for web servers including Tomcat Vault, OracleWallet, CyberArk, etc.

Answer 2

With any question like this it's all about risk mitigation and usability.

The first thing to accept is that if a third party has unrestricted access to hardware you won't be able to stop them. The same largely goes for having root access. At some point a secret will have to exist unencrypted in memory, and at that point it is possible to retrieve it. So let's deal with the case where we have a user who has access to a server and is able to run commands, but who isn't an admin. How best do you store your secrets?

Ideally, you don't store them on the server at all. You can utilise a secrets management system and pull in a secret that is regularly rotated that way and not have to worry about storing it at all. This is widely used and deployed on the cloud in things like AWS Secrets Manager or Azure Key Vault. In this scenario it doesn't matter about the security of your web host to a degree since the security is handled off-host.

Let's say for whatever reason it has to exist on the same server. Well you made sure it's not hard-coded into the code, good. Next thing is that the web application needs to read it. That doesn't mean though that any other user should be able to. The sensitive files should be have 400 permissions ideally in a separate folder outside of the web root. If they must sit under the web root, use something like .htaccess to make sure they can't be viewed.

What if someone steals your server? Well, yes it's good that everything is encrypted, but given enough computing power and time that could be broken. What if you have restored an offsite backup in the meantime and then our person has all the keys? That's the last and most important bit - regular rotation of unique secrets so that even if they have been compromised the amount of time that compromise is useful is low.

The final thing to do might be to just eliminate secrets altogether - connect to your database via presentation of a certificate rather than a password, restrict access to internal services to only come from the local machine, etc.

A filesystem is a way of cataloguing and organising a bunch of binary on a hard disk into something useful for an end user. There are good and bad filesystems depending on what you are looking for - speed, redundancy, or security. The principles are largely the same for all, and in your scenario where the user only has access to the web application and not the host then the security of the filesystem should not come into it as long as that application has been designed securely.