0

I made some changes to my config as per this suggestion:

SecAction \
    "id:901321,\
    phase:1,\
    pass,\
    t:none,\
    nolog,\
    initcol:global=global,\
    initcol:ip=%{x-forwarded-for}_%{tx.ua_hash},\
    setvar:'tx.real_ip=%{x-forwarded-for}'"

But nothing seems to happen. I noticed that my apache error_log was using the default error log format and logging everything as 127.0.0.1.

So I changed my ErrorLogFormat to:

ErrorLogFormat "[%{u}t] [%-m:%l] [pid %P:tid %T] %7F: %E: [client\ %{X-Forwarded-For}i] %M% ,\ referer\ %{Referer}i"

That made my logs better, but ModSecurity is still not doing any blocking. What's weird is that most of the ModSec logs in the apache error_log have an extra client IP section in them:

[Wed May..2019] [err] [pid X:tid X] [client XXX.XX.XX.XXX] [client 127.0.0.1] ModSecurity: Warning...

I have no idea where the extra [client 127.0.0.1] is coming from - I know it's definitely only doing this for ModSecurity logs in the error_log.

It seems like either ModSecurity is either constantly trying to block 127.0.0.1 or just not blocking anything..

So how can I get ModSecurity to block using the X-Forwarded-For header?

NOTE

  1. I do have SecRuleEngine On set properly.
  2. Versions: Apache 2.4, ModSecurity 2.9, OWASP 3
Yes Barry
  • 170
  • 1
  • 16

1 Answers1

1

The first part of your linked suggestion has this

SecAction "phase:1,nolog,pass,initcol:IP=%{REQUEST_HEADERS.x-forwarded-for}"

As you can see it specifies that x-forwarded-for is part of REQUEST_HEADERS but that is missing from your version of this.

Do be aware that just logging the IP will not cause blocking. It is used to persist data that can be used in subsequent rules (e.g. log a counter against an IP and iterate it with each request, and then block if it goes above a certain limit for basic DoS protection). So make sure you’ve got some rules configured to DO something with that IP address!

Also, as discussed in the comments, ModSecurity logs collections in a disk based file. This causes contention when lots of Apache processes try to access it at the same time. And the cleanup that ModSecurity does also can fail meaning the file grows and grows until it eats up all disk space or slows Apache to a crawl. I am not a fan of using them, especially on a site with an volume. I see them as a proof of concept of what ModSecurity could do to expand its single request rules based engine to a cross request one, but it’s not production ready IMHO. For more on this see here: https://sourceforge.net/p/mod-security/mailman/message/34393121/

The double client_ip logging is a remnant of when ModSecurity used to use a non-standard way of logging. They moved to Apache native logging (which has more features - like the ability to log HTTP Headers like x-forwarded-for) but only noticed late the double client_ip (as both Apache standing logging module and ModSecurity is now logging this). They left as is to avoid breaking anyone who depended on the original client_ip logging. See here for more details: https://github.com/SpiderLabs/ModSecurity/pull/840

Barry Pollard
  • 4,461
  • 14
  • 26
  • Thanks and good answer, (+1 for the quality info); but after making this change I still see `[Timestamp] UNIQUE_ID_HASH 127.0.0.1 XXXXX 127.0.0.1 XX` in the audit log. – Yes Barry May 24 '19 at 16:06
  • Is there something I'm misunderstanding? I thought this configuration change is to basically tell modsec to treat the x-forwarded-for IP(s) as the real client IP and use _that_ to block the user when there's a flagged request for _any_ SecRule in OWASP? – Yes Barry May 24 '19 at 16:19
  • 1
    No that’s not how it works. IP address is used for specific rules. Look at DOS rules or Brute Force rules which are in the experimental folder if on v2 of the CRS. They are not used to block all messages after a single failed rule. You could write such a rule based on DOS rule but I could imagine that would be quite dangerous as prone to false positives. – Barry Pollard May 24 '19 at 16:28
  • Also the disk-based method of storing collections (for IP addresses or other persisted information between requests) is not very good and I do not recommend using to be honest as you get lots of contention on that file and “can not access file” errors and it grows and grows so you have to delete it periodically. It needs a rethink and improved before it should be used. See here for more of my thoughts on that: https://sourceforge.net/p/mod-security/mailman/message/34393121/ – Barry Pollard May 24 '19 at 16:31
  • Awesome, I will read that. So then how _does_ it work? I was under the impression that anytime modsec decides to block a user (for whatever reason) for breaking a rule, it would use their IP address or give said user a 403 or whatever. That is not happening. Does it absolutely _have_ to be `REMOTE_ADDR` for that to work (e.g. do I have to for sure use mod_remoteip then?)? (Btw, I am trying to block SQLi attacks, and many other attacks not just DoS) – Yes Barry May 24 '19 at 16:37
  • 1
    In general ModSecurity blocks requests and not users. So it interrupts Apache processing if a rules is fired and instead of returning the result it returns a 403. Other requests from that IP still work (assuming they don’t fire rules) as they don’t check IP collections. If you have the DoS rules enabled then they do check the IP collections each time and they pass certain thresholds then it blocks ALL requests from that IP for a period of time. However the DoS limits are typically not set to 1 bad request as it seems you want - though could configure like this I suppose. If collections worked! – Barry Pollard May 24 '19 at 16:42
  • Hence your thoughts "Ultimately I don't think storing collection data in a disk based file is ever going to work particularly well for any site with any volume." I see what you mean there. I went back through my crs-setup.conf and checked hte `SecDefaultAction`, mine is in anomoly mode. But it's weird I am trying to "fake attack" my site with SQLi and I don't get 403'd... In the comment above this `SecDefaultAction` it says `"By default, offending requests are blocked with an error 403 response."` So is it that the SQLi isn't enough, it needs more "offenses" to block me?? – Yes Barry May 24 '19 at 16:47
  • Yeah ModSecurity is really designed as a “per request” system and the collections is a bit of a fudge to attempt to add state to allow “cross request” rules to be built - like DoS (too many requests in total in short period of time - e.g. 1,000 requests in a second) or Brute Force (too many requests against a login function in a short space of time - e.g. 10 requests). But then you need a shared place to store that state as multiple requests may not go through same Apache process/thread/event. Disk based shared stored is easiest to implement but won’t work under scale. I see them more a PoCs. – Barry Pollard May 24 '19 at 16:54
  • 1
    And yes anomaly scoring requires blocking rules to be above an anomaly score before it blocks. This may be just triggered by one bad rule failure (as I would expect with an obvious SQLi) or it might be lots of little rules fails. Counting separate rules for one request doesn’t require shared storage as all rules for one request are handled by one process so can share in memory. Error log should show anomaly score for each rule that is fired. Which is one reason I dislike anomaly scoring - it’s very noisy! https://serverfault.com/questions/888168/mod-security-anomaly-scoring-in-the-audit-log – Barry Pollard May 24 '19 at 17:00
  • Are you sure you’re not running in DetectionOnly mode? – Barry Pollard May 24 '19 at 17:01
  • Yes I am sure. I just changed the `SecDefaultAction` to "phase:1,log,auditlog,deny,status:403" and sure enough that worked. So I think what really needs to happen is that I need to change the anomoly scoring a bit or something. I am in paranoia level 1 so maybe I turn that up a bit! – Yes Barry May 24 '19 at 17:04
  • Would you mind adding all of this info to your answer in relevant/summary form so I can mark this as accepted? I think I need to edit my question too but I am struggling to think of _how_ it should be edited so it doesn't lose relevance or it's original purpose and stays informative and helpful :| – Yes Barry May 24 '19 at 17:06
  • 1
    Ah so you are on CRSv3. Yeah Paranoia level 1 doesn’t always block when it should. But then also doesn’t usually accidentally block when it shouldn’t. WAFs are very prone to false positives and blocking legitimate requests when they shouldn’t. So need to be tuned to avoid this. As you move up the Paranoia levels you need more tuning based on your site and the requests you get. Paranoia level 1 usually needs no tuning but, as I say, can allow requests that should be blocked. – Barry Pollard May 24 '19 at 17:08
  • Updated answer to include some of this info. – Barry Pollard May 24 '19 at 17:17