Should failed login attempts be logged? My doubt is that if there is a distributed brute force attack, it might exhaust the available disk space of the database. What is the best practice for this?
I'm protecting a public-facing web server with sensitive data.
Based on the answers so far, one other question that occurred to me is whether web server logs would be enough for logging such attempts. Would it be redundant to log them in the database?
Yes, failed login attempts should be logged:
It's also very important - older Windows logging process never emphasized this enough - to log successful login attempts as well. Because if you have a string of failed login attempts, you really really really should know if the last one was followed by a successful login.
Logs are relatively small. If there was enough login attempts that logging would cause a problem, then "not knowing about the attempts" is probably a worse-case problem than "found out about them when we ran out of disk."
A quick caveat - as @Polynomial points out, the password should not be logged (I seem to recall that 25 years ago some systems still did that). However, you also need to be aware that some legitimate login attempts will fail when people enter their password into the username field, so passwords do get logged. Doubt me? Trawl your logs for Windows Event ID 4768:
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4768
EventType=0
Type=Information
ComputerName=dc.test.int
TaskCategory=Kerberos Authentication Service
OpCode=Info
RecordNumber=1175382241
Keywords=Audit Failure
Message=A Kerberos authentication ticket (TGT) was requested.
Account Information:
Account Name: gowenfawr-has-a-cool-password
Supplied Realm Name: TEST.INT
User ID: NULL SID
Correspondingly, you should limit access to these logs to the necessary people - don't just dump them into a SIEM that the whole company has read access to.
Update to address question edit:
Based on the answers so far, one other question that occurred to me is whether web server logs would be enough for logging such attempts. Would it be redundant to log them in the database?
Best practices are that logs should be forwarded to a separate log aggregator in any case - for example, consider PCI DSS 10.5.4. In practice, such an aggregator is usually a SIEM, and functions like a database rather than flat log files.
So, yes, it's "redundant" by definition, but it's the kind of redundancy that's a security feature, not an architectural mistake.
The advantages of logging them into a database include searching, correlation, and summation. For example, the following Splunk search:
source="/var/log/secure" | regex _raw="authentication failure;" | stats count by user,host
Will allow us to roll up authentication failures by user and host:
Note that the ability to query discrete fields like 'user' and 'host' is dependent upon the SIEM picking logs apart and understanding what means what. The accessibility of those fields here is a side effect of Splunk automagically parsing the logs for me.
Given that your original question dealt with space constraints, it should be pointed out that any database or SIEM solution is going to take more disk space than flat text file logs. However, if you use such a solution, you'll almost always put it on a separate server for security and space management reasons. (There are even SIEM-in-the-cloud solutions now to make life easier for you!)
As a complement to @gowenfawr's answer that explains why you should log those attempts, I would like to say that there are ways to ensure that logs will never exhaust your disks.
At least in the Unix-Linux world, tools like logrotate or rotatelogs allows to change the log file when its size goes beyond a certain threshold. They are commonly used with the apache server (rotatelogs comes from Apache foundation) or with the syslog system.
For example logrotate is used to rename a log file (in a ring of a number of copies, generally about 10 of them) eventually compress it, and warns the program generating the log to reopen its log file by sending it a dedicated signal or via any arbitrary command.
That way, if your server is under a DoS attack, the size of your log files will remain under control. Of course you will loose older events, but that is definitely better than crashing the server because of an exhausted disk partition.
It really depends on what value you think you could derive from the information. There's limited value in having pages of logs telling you that your server is under attack; it's internet facing and will likely be under various degrees of constant bombardment for it's lifetime.
Depending on the configuration of your server, it is quite possible to end up creating an availability issue because you've exhausted the available disk space with logs. It does happen. Gowenfawr was right to state that logs don't take up much space but this is why issues with disk space exhaustion can take years to pop up but they're a major pain when they do.
If you decide to log, then you need to design a log management strategy and consider some of the following:
Speaking personally, I tend to find logs only useful for forensic analysis - they help work out what happened after a successful breach. As Gowenfawr mentioned; logging successful attempts to log into a system are just as (probably more) important than the failed ones.
One last point, your login mechanism should be built such that the likelihood of a distributed brute force ever working is vanishingly small. You haven't given a lot of detail on what you've built but using strong backend algorithms, particularly computationally expensive hashes and introducing backoff timing into login attempts can greatly reduce the chances that an attacker will ever gain access in this way.
