Weird apache mod_rewrite behavior

Question

I'm new to the site and I hope you can help me with an issue that happened to me trying to install Rhodecode. Here's the (long) story:
I've managed to install Rhodecode inside a virtualenv in a linux box. Using the development server (paster serve production.ini), i see it is running perfectly. However, I wanted to use Apache as a frontend to SSL, using mod_rewrite to redirect http requests to https. Here's my configuration:

default-vhost.conf

<VirtualHost _default_:80>
  ServerName hg.mydomain.com
  ServerAdmin admin@mydomain.com
  ServerAlias rhodecode.mydomain.com

  DocumentRoot "/srv/www/htdocs"
  RewriteEngine On
  RewriteCond %{HTTPS} off
  RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
  HostnameLookups Off
  UseCanonicalName Off
  ServerSignature Off
  ....
</VirtualHost>

I use mod_rewrite instead of a redirect because I want my site to be reachable by two domain names. We have a site at hg.mydomain.com we are planning to replace with the new rhodecode.mydomain.com, so I wanted to preserve the hostname in the rewrite rule. With this directive:

Redirect permanent / https://hg.mydomain.com/

the site works perfectly and no redirection problems occur. But when I browse to http://rhodecode.mydomain.com, I get redirected to the other site, and I can't do that until the site at hg.mydomain.com is discarded and hg.mydomain.com points to the same IP as rhodecode.mydomain.com.

The problem
Rhodecode sometimes includes urls in responses to actions that require authentication. For example, if you are a guest and try to se a private repo, you are redirected to the login screen with a url like this:

https://rhodecode.mydomain.com/_admin/login?came_from=%2F

where the %2F is an encoded '/'.
After the login, I get redirected to https://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var, and the apache default page for a 404 error is displayed. After that, browsing to https://rhodecode.mydomain.com/ shows I'm in session in the site. Why I get redirected to that strange HTTP_NOT_FOUND.html.var document?? Here is the rest of my configuration and the relevant parts of the logs:

default-vhost-ssl.conf

<VirtualHost _default_:443>

  ServerName hg.mydomain.com
  ServerAdmin admin@mydomain.com
  ServerAlias rhodecode.mydomain.com

  DocumentRoot "/srv/www/htdocs"
  HostnameLookups Off
  UseCanonicalName Off
  ServerSignature Off

  SSLEngine on

  certificate stuff ...

  WSGIDaemonProcess hg.mydomain.com user=rhodecode group=users threads=5 \
  home=/home/rhodecode/rhodecode-env python-path=/home/rhodecode/rhodecode-env/lib/python2.7/site-packages
  WSGIScriptAlias / /home/rhodecode/rhodecode-env/dispatch.wsgi
  WSGIPassAuthorization On

  <Directory /home/rhodecode/rhodecode-env>
    WSGIProcessGroup hg.mydomain.com
    WSGIApplicationGroup %{GLOBAL}
    Order deny,allow
    Allow from all
  </Directory>

</VirtualHost>

Rewrite Log

172.17.1.49 - - [04/Mar/2014:00:06:24 +0000] [rhodecode.mydomain.com/sid#7f6a03266f00][rid#7f69fd68d7a0/initial/redir#1] (2) init rewrite engine with requested uri /error/HTTP_NOT_FOUND.html.var
172.17.1.49 - - [04/Mar/2014:00:06:24 +0000] [rhodecode.mydomain.com/sid#7f6a03266f00][rid#7f69fd68d7a0/initial/redir#1] (3) applying pattern '(.*)' to uri '/error/HTTP_NOT_FOUND.html.var'
172.17.1.49 - - [04/Mar/2014:00:06:24 +0000] [rhodecode.mydomain.com/sid#7f6a03266f00][rid#7f69fd68d7a0/initial/redir#1] (4) RewriteCond: input='off' pattern='off' => matched
172.17.1.49 - - [04/Mar/2014:00:06:24 +0000] [rhodecode.mydomain.com/sid#7f6a03266f00][rid#7f69fd68d7a0/initial/redir#1] (2) rewrite '/error/HTTP_NOT_FOUND.html.var' -> 'https://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var'
172.17.1.49 - - [04/Mar/2014:00:06:24 +0000] [rhodecode.mydomain.com/sid#7f6a03266f00][rid#7f69fd68d7a0/initial/redir#1] (2) explicitly forcing redirect with https://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var
172.17.1.49 - - [04/Mar/2014:00:06:24 +0000] [rhodecode.mydomain.com/sid#7f6a03266f00][rid#7f69fd68d7a0/initial/redir#1] (1) escaping https://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var for redirect
172.17.1.49 - - [04/Mar/2014:00:06:24 +0000] [rhodecode.mydomain.com/sid#7f6a03266f00][rid#7f69fd68d7a0/initial/redir#1] (1) redirect to https://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var [REDIRECT/301]

Pay attention to the rid#7f69fd68d7a0/initial/redir#1 part. When I send URLs without %2F, that part does not appear in the log.

Access Log

hg.mydomain.com:443 172.17.1.49 - - [04/Mar/2014:02:09:13 +0000] "POST /_admin/login?came_from=%252F HTTP/1.1" 302 186 "https://rhodecode.mydomain.com/_admin/login?came_from=%252F" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"
hg.mydomain.com:80 172.17.1.49 - - [04/Mar/2014:02:09:14 +0000] "GET /_admin/%2F HTTP/1.1" 301 268 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"
hg.mydomain.com:443 172.17.1.49 - - [04/Mar/2014:02:09:14 +0000] "GET /error/HTTP_NOT_FOUND.html.var HTTP/1.1" 200 1132 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"
hg.mydomain.com:443 172.17.1.49 - - [04/Mar/2014:02:09:14 +0000] "GET /favicon.ico HTTP/1.1" 404 618 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"

The first line is the POST request with the authentication data. It is successful, so it redirects me to the admin page. In the second line, you can see the redirect location sent me back to http (it entered port 80) to /_admin/%2F and as such, gets redirected to https but magically transformed to /error/HTTP_NOT_FOUND.html.var

HTTP request (Some headers ommited for brevity)

POST /_admin/login?came_from=%252F HTTP/1.1
Host: rhodecode.mydomain.com
Cache-Control: no-cache
Pragma: no-cache
Origin: https://rhodecode.mydomain.com
Content-Type: application/x-www-form-urlencoded
Referer: https://rhodecode.mydomain.com/_admin/login?came_from=%252F
Cookie: rhodecode=3af58050ce87a93caa5a4c6809c5dacef4afb29d8e74b152c97f469199c554b6f67f7aa7
...

HTTP Response

HTTP/1.1 302 Found
Date: Tue, 04 Mar 2014 02:24:11 GMT
Server: Apache/2.2.22 (Linux/SUSE)
Pragma: no-cache
Cache-Control: no-cache
Set-Cookie: rhodecode=f0a94a155738490da032b46354f4d72338902da2d69bc1177bcf4086aa8158f4719526e0; httponly; Path=/
Location: http://rhodecode.mydomain.com/_admin/%2F
...

HTTP Request 2

GET /_admin/%2F HTTP/1.1
Host: rhodecode.mydomain.com
Cache-Control: no-cache
Pragma: no-cache
Cookie: rhodecode=f0a94a155738490da032b46354f4d72338902da2d69bc1177bcf4086aa8158f4719526e0
...

HTTP Response 2

HTTP/1.1 301 Moved Permanently
Date: Tue, 04 Mar 2014 02:24:12 GMT
Server: Apache/2.2.22 (Linux/SUSE)
Location: https://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var
...

And finally, trying to get /error/HTTP_NOT_FOUND.html.var does not give me a 404 error, but a 200 OK response!!
I thought browsers were doing something weird under the hood, so I sent a raw HTTP request and got the same results:

[Rober@yue ~]$ nc rhodecode.mydomain.com 80
GET /%2F HTTP/1.1
Host: rhodecode.mydomain.com

HTTP/1.1 301 Moved Permanently
Date: Tue, 04 Mar 2014 00:07:26 GMT
Server: Apache/2.2.22 (Linux/SUSE)
Location: https://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var
Content-Length: 268
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var">here</a>.</p>
</body></html>

Look at the location header in the response. Why Apache is doing this instead of just changing the request to https???

Sorry for the looong text of my question, but I wanted to give as much information as possible so you can help me to debug this :)

Thanks to all in advance!

Edit
As suggested, I wanted to find out if Rhodecode was sending the redirects. So I changed my wsgi script for the one shown in the documentation: http://modwsgi.readthedocs.org/en/latest/configuration-guides/running-a-basic-application.html#wsgi-application-script-file. It always returns a "Hello World" without the possibility of redirection. I sent a raw request to the site with the same results, so Apache must be changing the url somehow. Here is the result:

GET /%2F HTTP/1.1
Host: rhodecode.mydomain.com

HTTP/1.1 301 Moved Permanently
Date: Thu, 06 Mar 2014 04:23:31 GMT
Server: Apache/2.2.22 (Linux/SUSE)
Location: https://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var
Content-Length: 268
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var">here</a>.</p>
</body></html>

But when I send a normal URL (without %2F), it redirects OK:

GET /abffr HTTP/1.1
Host: rhodecode.mydomain.com

HTTP/1.1 301 Moved Permanently
Date: Thu, 06 Mar 2014 04:25:19 GMT
Server: Apache/2.2.22 (Linux/SUSE)
Location: https://rhodecode.mydomain.com/abffr
Content-Length: 244
Content-Type: text/html; charset=iso-8859-1

Apache seems to not like the %2F thing...

This is your apache config... looks like this Rhodecode web app is doing the redirect to /error/HTTP_NOT_FOUND.html.var. — ETL, Mar 04 '14 at 03:40
@ETL: Hi, I changed my application script to test whether Rhodecode was sending the redirect, but I found it wasn't. I've edited my question with the details. — rober710, Mar 06 '14 at 04:18

score 0 · Accepted Answer · answered Mar 16 '14 at 02:15

Turns out there were two things that were causing this error. First is the default value of AllowEncodedSlashes, from the docs:

With the default value, Off, URLs which contain encoded path separators (%2F for / and additionally %5C for \ on according systems) are refused with a 404 (Not found) error.

I discovered this by turning on maximum verbosity in apache logs and found this:

[Mon Mar 10 04:53:43 2014] [info] [client 172.17.1.49] found %2f (encoded '/') in URI (decoded='//'), returning 404

Thus, all my requests with %2F were rejected. Also, I had this configuration that came in my server by default:

<IfModule mod_negotiation.c>
<IfModule mod_include.c>
    <Directory "/usr/share/apache2/error">
        AllowOverride None
        Options IncludesNoExec
        AddOutputFilter Includes html
        AddHandler type-map var
        Order allow,deny
        Allow from all
        LanguagePriority en cs de es fr it ja ko nl pl pt-br ro sv tr
        ForceLanguagePriority Prefer Fallback
    </Directory>

    ErrorDocument 400 /error/HTTP_BAD_REQUEST.html.var
    ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
    ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var
    ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var
    ErrorDocument 405 /error/HTTP_METHOD_NOT_ALLOWED.html.var
    ErrorDocument 408 /error/HTTP_REQUEST_TIME_OUT.html.var
    ErrorDocument 410 /error/HTTP_GONE.html.var
    ErrorDocument 411 /error/HTTP_LENGTH_REQUIRED.html.var
    ErrorDocument 412 /error/HTTP_PRECONDITION_FAILED.html.var
    ErrorDocument 413 /error/HTTP_REQUEST_ENTITY_TOO_LARGE.html.var
    ErrorDocument 414 /error/HTTP_REQUEST_URI_TOO_LARGE.html.var
    ErrorDocument 415 /error/HTTP_UNSUPPORTED_MEDIA_TYPE.html.var
    ErrorDocument 500 /error/HTTP_INTERNAL_SERVER_ERROR.html.var
    ErrorDocument 501 /error/HTTP_NOT_IMPLEMENTED.html.var
    ErrorDocument 502 /error/HTTP_BAD_GATEWAY.html.var
    ErrorDocument 503 /error/HTTP_SERVICE_UNAVAILABLE.html.var
    ErrorDocument 506 /error/HTTP_VARIANT_ALSO_VARIES.html.var
</IfModule>
</IfModule>

So everytime a 404 error was sent to the client, the Includes filter combined with the ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var directive caused a redirect to be sent to the browser to http://rhodecode.mydomain.com/error/HTTP_NOT_FOUND.html.var
You can see this behavior in my rewrite logs, where the expression to be rewritten is initialized with /error/HTTP_NOT_FOUND.html.var:

172.17.1.49 - - [10/Mar/2014:04:04:23 +0000] [rhodecode.mydomain.com/sid#7f98fedd7f00][rid#7f98f91fe7a0/initial/redir#1] (2) init rewrite engine with requested uri /error/HTTP_NOT_FOUND.html.var

Also note that redir#1 indicating that an internal redirect occurred. So, I changed my config to

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
AllowEncodedSlashes NoDecode

And changed the ifmodule directive to

<IfModule !mod_include.c>

so no ErrorDocument directives are executed and everything worked perfectly!
This time, the rewrite logs show the %2F in the url and no redir happens:

172.17.1.49 - - [10/Mar/2014:04:55:40 +0000] [rhodecode.mydomain.com/sid#7f9fd1f83f00][rid#7f9fcc3a90a0/initial] (2) init rewrite engine with requested uri /%2F

score 0 · Answer 2 · answered Aug 28 '15 at 10:21

0

RhodeCode has also a special flag to tell it to that you want to force use of ssl, meaning all redirects would go always to https.

In you .ini flag change:

force_ssl = true

answered Aug 28 '15 at 10:21

marcinkuzminski

185
1
4

Weird apache mod_rewrite behavior

2 Answers2