UnicodeEncodeError when uploading files in Django admin

Question

Note: I asked this question on StackOverflow, but I realize this might be a more proper place to ask this kind of question.

I'm trying to upload a file called 'Testaråäö.txt' via the Django admin app.

I'm running Django 1.3.1 with Gunicorn 0.13.4 and Nginx 0.7.6.7 on a Debian 6 server. Database is PostgreSQL 8.4.9. Other Unicode data is saved to the database with no problem, so I guess the problem must be with the filesystem somehow.

I've set

http {
    charset utf-8;
}

in my nginx.conf. LC_ALL and LANG is set to 'sv_SE.UTF-8'. Running 'locale' verifies this. I even tried setting LC_ALL and LANG in my nginx init script just to make sure locale is set properly.

Here's the traceback:

Traceback (most recent call last):

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 307, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 93, in _wrapped_view
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/views/decorators/cache.py", line 79, in _wrapped_view_func
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/sites.py", line 197, in inner
return view(request, *args, **kwargs)

File "/srv/django/letebo/app/cms/admin.py", line 81, in change_view
return super(PageAdmin, self).change_view(request, obj_id)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 28, in _wrapper
return bound_func(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 93, in _wrapped_view
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 24, in bound_func
return func(self, *args2, **kwargs2)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/transaction.py", line 217, in inner
res = func(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 985, in change_view
self.save_formset(request, form, formset, change=True)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 677, in save_formset
formset.save()

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 482, in save
return self.save_existing_objects(commit) + self.save_new_objects(commit)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 613, in save_new_objects
self.new_objects.append(self.save_new(form, commit=commit))

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 717, in save_new
obj.save()

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 460, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 504, in save_base
self.save_base(cls=parent, origin=org, using=using)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 543, in save_base
for f in meta.local_fields if not isinstance(f, AutoField)]

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/fields/files.py", line 255, in pre_save
file.save(file.name, file, save=False)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/fields/files.py", line 92, in save
self.name = self.storage.save(name, content)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 48, in save
name = self.get_available_name(name)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 74, in get_available_name
while self.exists(name):

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 218, in exists
return os.path.exists(self.path(name))

File "/srv/.virtualenvs/letebo/lib/python2.6/genericpath.py", line 18, in exists
st = os.stat(path)

UnicodeEncodeError: 'ascii' codec can't encode characters in position 52-54: ordinal not in range(128)

I tried running Gunicorn with debugging turned on, and the file uploads without any problem at all. I suppose this must mean that the issue is with Nginx. Still beats me where to look, though. Here are the raw response headers from Gunicorn and Nginx, if it makes any sense:

Gunicorn:

HTTP/1.1 302 FOUND
Server: gunicorn/0.13.4
Date: Thu, 09 Feb 2012 14:50:27 GMT
Connection: close
Transfer-Encoding: chunked
Expires: Thu, 09 Feb 2012 14:50:27 GMT
Vary: Cookie
Last-Modified: Thu, 09 Feb 2012 14:50:27 GMT
Location: http://my-server.se:8000/admin/cms/page/15/
Cache-Control: max-age=0
Content-Type: text/html; charset=utf-8
Set-Cookie: messages="yada yada yada"; Path=/

Nginx:

HTTP/1.1 500 INTERNAL SERVER ERROR
Server: nginx/0.7.67
Date: Thu, 09 Feb 2012 14:50:57 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Vary: Cookie

500

UPDATE: Both locale.getpreferredencoding() and sys.getfilesystemencoding() outputs 'UTF-8'. locale.getdefaultlocale() outputs ('sv_SE', 'UTF8'). This seem correct to me, so I'm still not sure why I keep getting these errors.

Check what you get for outputs from `locale.getpreferredlocale()` and `sys.getfilesystemencoding()` in your python code - something's not playing the UTF8 game, since it's trying to shoehorn into ASCII for the `os.stat()` call. — Shane Madden, Feb 14 '12 at 19:06
I do have the same exact problem. I hooked some code in django/db/models/fields/files.py to know what is the type of the string name in line 92. and it's a unicode. have you found the fix? — Treddit, Feb 26 '12 at 17:58

score 2 · Answer 1 · answered Jun 23 '15 at 11:14

I had the same issue with genericpath.py giving a UnicodeEncodeError when attempting to upload a file name with non ASCII characters.

I was using nginx, uwsgi and django with python 2.7.

Everything was working OK locally but not on the server

Here are the steps I took 1. added to /etc/nginx/nginx.conf (did not fix the problem)

http {
    charset utf-8;
}

I added this line to etc/default/locale (did not fix the problem)

LANGUAGE="en_US.UTF-8"

I followed the instructions here listed under the heading 'Success' https://code.djangoproject.com/wiki/ExpectedTestFailures (did not fix the problem)
```
aptitude install language-pack-en-base
```
Found across this ticket https://code.djangoproject.com/ticket/17816 which suggested testing a view on the server to what was happening with locale information

In your view

import locale
locales = "Current locale: %s %s -- Default locale: %s %s" % (locale.getlocale() + locale.getdefaultlocale())

In your template

{{ locales }}

For me, the issue was that I had no locale and no default locale on my Ubuntu server (though I did have them on my local OSX dev machine) then files with non ASCII file names/paths will not upload correctly with python raising a UnicodeEncodeError, but only on the production server.

Solution

I added this to both my site and my site admin uwsgi config files e.g. /etc/uwsgi-emperor/vassals/my-site-config-ini file

env = LANG=en_US.utf8

Your solution worked for me too! This should be an accepted answer! — Maciek, Jun 02 '16 at 10:28

score 1 · Answer 2 · answered Oct 15 '19 at 05:58

Just in case anyone comes here with Apache using mod_wsgi, this can be solved by adding the lang and locale options to the WSGIDaemonProcess line in your site's config file.

Should normally be set to en_US.UTF-8 according to the docs:

https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html

Like so:

lang=en_US.UTF-8 locale=en_US.UTF-8

For example:

WSGIDaemonProcess processName python-home=/path/to/venv python-path=/path/to/mydjangoproject lang=en_US.UTF-8 locale=en_US.UTF-8

After adding these options, restart Apache and you should be good to go.

score 0 · Answer 3 · answered Feb 26 '12 at 18:39

I had the same problem and I found the fixes for my cases (I run Ubuntu).

I found a good unanswered thread here, where Karen give some usefull information:

https://groups.google.com/forum/?fromgroups#!topic/django-users/hwNL7i6IeIY

which led me to this:

https://code.djangoproject.com/wiki/ExpectedTestFailures

I installed the missing "language-pack" for Ubuntu. and I ran in a shell as root:

locale-gen en_US.UTF-8

and it's now working.

Look like, in low level layers of Python, something call Glibc which need some compiled files generated by locale-gen to work correctly.

I've looked in http://packages.debian.org and there is no package like this in Debian.

Try to run the following and restart your daemon before trying again:

locale-gen sv_SE.UTF-8

or maybe install any Debian specifics locale/language packages, if there are.

UnicodeEncodeError when uploading files in Django admin

3 Answers3