Nginx uWSGI responses truncated

Question

I tried to write a web service as a joke today at http://dont-tread-on-memes.controversial.io. It's a flask app that serves fairly large images. The Flask app works well on its own, as does an independent uWSGI server, but when I try to plug uWSGI into NGINX via uwsgi_pass, suddenly every other request is truncated at 9.99KB across browsers.

After reading about similar truncation with proxy_pass I tried:

Setting uwsgi_buffering to off in my config file
Increasing the buffer size to 1024k with uwsgi_buffers 1024 1024k; uwsgi_buffer_size 1024k;
sendfile: off
Checking buffer file permissions (all the files in /var/lib/uwsgi are owned by the www-data user and the www-data group, so I think my permissions are good.)

I'm left with my current config, which still exhibits the issue:

server {
    listen 80;
    server_name dont-tread-on-memes.controversial.io;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/var/www/dont-tread-on-memes/dont_tread_on_memes.sock;
        uwsgi_buffers 1024 1024k;
        uwsgi_buffer_size 1024k;
    }
}

The strangest part is that this issue appears only on every second request. It has to be something to do with NGINX cache, since I'm not using multiple NGINX instances or anything. Yet it has to be something to do with my NGINX config, since uWSGI running on its own does not exhibit the issue.

Any thoughts on what could be causing this issue, and how to fix it?

The responses are chunked. You can try to disable `chunked_transfer_encoding`, but I think it depends on how are you serving the images within the app. Anyway, inspect the HTTP headers when you're requesting the Flask app directly and look for differencies. That could give you a clue. — charli, Jan 07 '17 at 21:48
The only header differences are that NGINX includes `Date`, `Server`, and `Connection: close` while uWSGI omits all three of these. — Luke Taylor, Jan 07 '17 at 21:59
@charli Try refreshing. Every second response is truncated to 9.99KB — Luke Taylor, Jan 07 '17 at 22:12
Yeah, it happens only every 2 requests. This is really weird. Do you have a load balancer in front of 2 nginx or something similar? I can't explain that behaviour... — charli, Jan 07 '17 at 22:27
I don't think I do, @charli. I have no idea why it could be. — Luke Taylor, Jan 07 '17 at 22:29
Sure, @MichaelHampton, it's https://github.com/controversial/dont-tread-on-memes/ — Luke Taylor, Jan 08 '17 at 02:27

score 1 · Answer 1 · edited May 23 '17 at 12:41

After double-checking that the uwsgi command I was testing against matched all the options I'd provided in my .ini file, I realized that my .ini file contained processes = 5 while the uwsgi command I was testing with did not. If I add --processes=5 to my uwsgi command, I can reproduce the truncation issue on every hit, not just on every second request. Each time I start the uwsgi server with --processes=5, the first request succeeds, the second yields a 500, and all subsequent requests are truncated to 9.99MB with the following error in the console:

[2017-01-08 04:16:08,959] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "./flaskapp.py", line 13, in main
    flag = dont_tread_on_memes.dont_me(caption)
  File "./dont_tread_on_memes/__init__.py", line 30, in dont_me
    return tread_on("don't {} me".format(phrase))
  File "./dont_tread_on_memes/__init__.py", line 16, in tread_on
    flag = BLANK_FLAG.copy()
  File "/usr/local/lib/python3.4/dist-packages/PIL/Image.py", line 1010, in copy
    self.load()
  File "/usr/local/lib/python3.4/dist-packages/PIL/ImageFile.py", line 226, in load
    "(%d bytes not processed)" % len(b))
OSError: image file is truncated (0 bytes not processed)

My suspicion is that this is an issue with Pillow and the way uwsgi handles threading. Perhaps the every-other-request behavior was due to the way uwsgi decides to spawn new processes and kill old processes, or due to caching by NGINX. Either way, I've fixed the truncation issue.

I also found this on StackOverflow from someone with my same issue.

If anyone else can provide an answer for why this occurred, or an answer as to how I can both keep this resolved and can let uwsgi spawn multiple processes, I'd certainly consider that a more complete answer and accept it.

score 1 · Answer 2 · edited May 23 '17 at 12:41

This almost always means that there is a problem with your images (i.e., there is missing data at the bottom). I have processed images > 25mb using PIL (provided you have enough RAM) and it works fine. The workaround here might work for you. Copy & pasted for ease of reading:

if img and img.meta_type == 'Image':
    pilImg = PIL.Image.open( StringIO(str(img.data)) )
elif imgData:
    pilImg = PIL.Image.open( StringIO(imgData) )

try:
    pilImg.load()
except IOError:
    pass # You can always log it to logger

score 0 · Answer 3 · edited Dec 18 '21 at 14:16

0

I had a similar problem. Some javascript files was turncut. Settings buffer:

uwsgi_buffers 1024 1024k;

uwsgi_buffer_size 1024k;

fixed the problem

edited Dec 18 '21 at 14:16

Paul

2,755
6
24
35

answered Dec 17 '21 at 11:35

Marcin Sobieszczyk

1

Nginx uWSGI responses truncated

3 Answers3