16

System Configuration : Apache2, Django 1.10, Python 3, Ubuntu 16.04 LTS

Django debug=True.


/var/log/apache2/error.log

[52:53.057967] [wsgi:error] [pid 4303] [client 1.1.1.22:24409] Timeout when reading response headers from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466726] [wsgi:error] [pid 4305] [client 1.1.1.10:9787] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466729] [wsgi:error] [pid 4304] [client 1.1.1.4:18417] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466726] [wsgi:error] [pid 4307] [client 1.1.1.22:35116] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466756] [wsgi:error] [pid 4306] [client 1.1.1.22:19242] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.467164] [wsgi:error] [pid 4336] [client 1.1.1.4:34187] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.467212] [wsgi:error] [pid 4342] [client 1.1.1.22:28212] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py, referer: http://example.org/
[52:58.467282] [wsgi:error] [pid 4331] [client 1.1.1.22:31045] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.467426] [wsgi:error] [pid 4341] [client 1.1.1.70:22784] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py, referer: http://example.org/

I do not know the cause of the error. But I have narrowed it down to the Django wsgi process. Since the server is hosting static files properly.

While Cloudflare sometimes shows 502 : Bad Gateway Error, the server itself shows 500 : Internal Server Error.

I have already tried restarting the server and checking Django's (debug) log file. There is no error information in the Django log files (at all).


How should I debug the problem? Since Django didn't log anything, I assume the problem could be caused in wsgi.

Note : The server was working fine earlier. I made some changes* (which are reverted back as-is); The Django shell works fine.

Changes*

  1. Installed django-pandas, django-model-utils, numpy, scikit-learn
  2. A program that utilises above libraries. (This change is reverted to original)

In other similar questions, the problem is caused during upload of a big file.

Suraj
  • 419
  • 1
  • 3
  • 12
  • Could you share the relevant apache config? It sounds like either you might need to tune/check your [LimitRequestBody](https://httpd.apache.org/docs/2.2/mod/core.html#limitrequestbody) settings, and/or the equivalent django settings. – iwaseatenbyagrue May 11 '17 at 17:27
  • @iwaseatenbyagrue please check my answer. – Suraj May 14 '17 at 14:12

5 Answers5

15

The cause of the problem was numpy.

Python C extension modules, like numpy, are known to cause timeouts when used under mod_wsgi.

Source : Answer by Sean F on Timeout when reading response headers from daemon process

A similar question which I didn't find on initial search answered and explained by author of mod_wsgi says -

Some third party packages for Python which use C extension modules, and this includes scipy and numpy, will only work in the Python main interpreter and cannot be used in sub interpreters as mod_wsgi by default uses. The result can be thread deadlock, incorrect behaviour or processes crashes.

Source : Answer by Graham Dumpleton on Non-responsive apache + mod_wsgi after installing scipy

Solution

Add the below line to your httpd.conf. In my case the file was /etc/apache2/apache2.conf.

WSGIApplicationGroup %{GLOBAL}
Suraj
  • 419
  • 1
  • 3
  • 12
  • 2
    Can you explain what change when you use "%{GLOBAL}"? – Nazkter Sep 29 '17 at 20:44
  • 4
    I get this error even with `WSGIApplicationGroup %{GLOBAL}` in my apache config. – Cerin Oct 27 '17 at 05:13
  • @Cerin did you make sure that that directive applies to _all_ Python files being interpreted? (I.e. not buried in some Apache `` that limits the scope.) – Ezekiel Victor Nov 08 '17 at 22:34
  • @EzekielVictor I don't understand your question. Suraj specified the line to be placed inside `/etc/apache2/apache2.conf`, which, by definition, makes it apply to all sites. – Cerin Jun 11 '19 at 17:37
  • I tried putting this in both my global `/etc/apache2/apache2.conf` as well as each site's `/etc/apache2/sites-enabled/.conf` file, and neither had any effect. – Cerin Jun 11 '19 at 17:44
  • 1
    @Cerin it's possible that it could be inside a `` tag anywhere but it sounds like you made sure that wasn't the case. Did you also restart Apache? – Ezekiel Victor Jun 12 '19 at 19:02
  • I had this error come in random (probably because of numpy) and crash my site. I just did what you said and I don't know if it works. Do you have any idea on how I could try to reproduce the error ? – Sam May 04 '20 at 08:20
  • 1
    I had to restart Apache after that. Thanks. – huseyin39 Sep 12 '20 at 18:34
  • I got the error "truncated ..." suddenly after installing Apache+mod_wsgi under Python3.10. No problem under 3.7/3.8/3.9. Thanks a lot for the Solution. WSGIApplicationGroup %{GLOBAL} helped. – Razenstein Feb 05 '22 at 09:24
7

As others have mentioned, this is caused by the process crashing for a number of potential reasons. One reason being that some Python dependency isn't thread safe.

If that's the problem, one work around is to switch Apache's MPM type. The prefork type doesn't use threads, so if the problem is numpy crashing due to thread misuse, then that should fix it. The worker and event types use less memory, but also use threads, so they can encounter this error.

To determine which type you're currently using, run:

apachectl -V | grep -i mpm

If you see Server MPM: prefork then you're already using prefork, meaning the cause of the error might be something else. If it says "worker" or "event", then you can switch to prefork by running:

sudo a2dismod mpm_event
sudo a2dismod mpm_worker
sudo a2enmod mpm_prefork
sudo service apache2 restart

Note, the main downside of prefork is that, since it's not using threads, it consumes more memory.

Edit: I've since encountered this error due to other causes. Most recently, the problem is caused by a bug in Ubuntu's precompiled mod-wsgi system package as well as a bug in the Python psycopg2 package.

The solution for this was to switch from the system mod-wsgi to the Python package, as well as switch to the psycopg2-binary package:

sudo apt purge libapache2-mod-wsgi*
sudo apt install apache2-dev
pip uninstall psycopg2
pip install mod_wsgi psycopg2-binary

I also had to update my apache site configuration file by adding to the top:

LoadModule wsgi_module /usr/local/myproject/.env/lib/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py27.so

And change my WSGIDaemonProcess statement to use python-home instead of python-path, to something like:

WSGIDaemonProcess myproject python-home=/usr/local/myproject/.env processes=5 threads=15 display-name=%{GROUP} user=www-data group=www-data

I've encountered this for both Python2.7 and Python3.7 and the solution is the same, but the path to the mod_wsgi.so changes.

Cerin
  • 3,497
  • 17
  • 57
  • 72
  • I had the same problem but in the end, simply removing `psycopg2` and installing `psycopg2-binary` is what fixed it. This directive `WSGIApplicationGroup %{GLOBAL}` didn't have any effect and my app is currently working fine without it. – musashiXXX Aug 22 '19 at 14:16
3

I have received a similar error while trying to run opencv using mod_wsgi and apache. I guess the problem was probably multiple threads with underlying C code trying to acquire GIL and failing.

I solved it by setting threads=1 and processes=32(in my case it was appropriate) in WSGIDaemonProcess directive.

PS: Late to the party but thought it could help someone.

mind_stone
  • 31
  • 1
0

In my case, the problem was pymongo and PHP. As described in this GitHub issue https://github.com/GrahamDumpleton/mod_wsgi/issues/351:

If PHP is loading a client for MongoDB it can [create a conflict]. (...) PHP often preloads all extensions. Doesn't matter if hosted application uses it or not.

The way I solved it:

  1. I ran my flask app from mod_wsgi-express: mod_wsgi-express start-server api.wsgi --user=www-data --group=www-data --host=0.0.0.0 --port=8443.
  2. Then, in my httpd.conf, I redirected all traffic to my flask app alias using:
SSLEngine on
SSLProxyEngine On
ProxyPass /api http://localhost:8443/
ProxyPassReverse /api http://localhost:8443/
prm296
  • 101
0

In my case I had to change the WSGIDaemonProcess line from:

WSGIDaemonProcess wsgi processes=2 threads=4 display-name=%{GROUP} \
  python-path=/var/www/appname:/var/www/appname/venv/lib/python2.7/site-packages user=wsgi group=wsgi \
  home=/var/www/appname

to:

WSGIDaemonProcess appname user=wsgi group=wsgi processes=2 threads=4 display-name=%{GROUP} home=/var/www/appname
RyanH
  • 327
  • 1
  • 6