WSGI cannot access a file, but permissions are correct

Question

I am debugging a problem where MoinMoin on CentOS is throwing a permissions error, but I can't track down where the problematic file / directory is.

I ran strace -vp <pid> on the apache pid; when I have the problem I see this:

epoll_wait(10, {{EPOLLIN, {u32=3487534344, u64=140367313734920}}}, 2, 10000) = 1
accept4(6, {sa_family=AF_INET6, sin6_port=htons(52621), inet_pton
    (AF_INET6, "::ffff:105.193.30.91", &sin6_addr), sin6_flowinfo=0, 
    sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 11
## Later on...
read(7, 0x7fffa658ad7f, 1)              = -1 EAGAIN (Resource temporarily 
    unavailable)

However, since apache is already running, I see no corresponding open() on the file referred to as 7; thus I see the permissions problem, but I still don't know which file is the problem.

I know I could try to catch all the file opens when I respawn apache, but I'm hoping there is a way to map file 7 to a real filename... is there a way to do this?

EDIT 1:

Using @lain's guidance, I ran lsof | grep 266474069, but the results are unclear...

[root@lnxlmf moin]# ls -la /proc/9707/fd/7
lr-x------. 1 root root 64 Aug 28 15:39 /proc/9707/fd/7 -> pipe:[266474069]
[root@lnxlmf moin]#

[root@lnxlmf moin]# lsof | grep 266474069
httpd      9703      root    7r     FIFO                0,8          0t0  266474069 pipe
httpd      9703      root    8w     FIFO                0,8          0t0  266474069 pipe
httpd      9705    apache    7r     FIFO                0,8          0t0  266474069 pipe
httpd      9705    apache    8w     FIFO                0,8          0t0  266474069 pipe
httpd      9706    apache    7r     FIFO                0,8          0t0  266474069 pipe
httpd      9706    apache    8w     FIFO                0,8          0t0  266474069 pipe
httpd      9707    apache    7r     FIFO                0,8          0t0  266474069 pipe
httpd      9707    apache    8w     FIFO                0,8          0t0  266474069 pipe
httpd      9733    apache    7r     FIFO                0,8          0t0  266474069 pipe
httpd      9733    apache    8w     FIFO                0,8          0t0  266474069 pipe
[root@lnxlmf moin]#

I see that this is a FIFO pipe, but what does this mean for my system configuration? How can I trace the root cause for the EAGAIN issue?

EDIT 2:

Thanks to @Alan Curry, running strace -fp <pid_of_wsgi_proc> seems to get me a bit farther...

[pid  9731] stat("/opt/moin/share/moin/wikiconfig.py", {st_mode=S_IFREG|0770, st_size=6463, ...}) = 0
[pid  9731] stat("/opt/moin/share/moin/data/pages", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
[pid  9731] access("/opt/moin/share/moin/data/pages", R_OK|W_OK|X_OK) = -1 EACCES (Permission denied)

However, both apache and wsgi run as the apache user...

[root@lnxlmf moin]# ps auxw | grep -E "apache|wsgi"
apache   10187  0.0  0.1 373488  5884 ?        Sl   17:18   0:00 moin_http_wsgi
apache   10188  0.0  0.1 373488  5884 ?        Sl   17:18   0:00 moin_https_wsgi
apache   10189  0.0  0.1 185096  5824 ?        S    17:18   0:00 /usr/sbin/httpd
root     10243  0.0  0.0 103240   848 pts/1    S+   17:21   0:00 grep -E apache|wsgi
[root@lnxlmf moin]#

Yet, when I run the following commands and restart apache, I still can't fix the problem...

[root@lnxlmf moin]# pwd
/opt/moin/share/moin
[root@lnxlmf moin]# chown -R apache:apache data/
[root@lnxlmf moin]# sudo chmod -R ug+rwx data/
[root@lnxlmf moin]# sudo chmod -R o-rwx data/

I am using this in my wiki http config:

<VirtualHost *:443>
  ServerName netwiki.foo.com
  DocumentRoot /opt/moin/share/moin
  WSGIScriptAlias / /opt/moin/share/moin/server/moin.wsgi
  WSGIDaemonProcess moin_https display-name=moin_https_wsgi \
      user=apache group=apache \
      processes=1 threads=10 maximum-requests=1000 umask=0007
  WSGIProcessGroup moin_https
  WSGIApplicationGroup %{GLOBAL}

  # Generate with...
  # openssl req -new -x509 -days 365 -nodes -out netwiki.pem -keyout netwiki.key
  SSLEngine on
  SSLCertificateFile /etc/httpd/ssl/netwiki.pem
  SSLCertificateKeyFile /etc/httpd/ssl/netwiki.key
</VirtualHost>

It should be noted that `EAGAIN` is not a permissions issue. It's just a "buffer is empty right now" indicator. It appears quite often in the strace output of programs that are working perfectly. How sure are you that this particular `EAGAIN` is connected to your problem? — Alan Curry, Aug 28 '12 at 22:08
Well, you're probably right... bad call on my part. I will run `strace` again and see if I can nail this down. — Mike Pennington, Aug 28 '12 at 22:12
Since you're looking at an `access()` failure, I wonder if the real uid doesn't match the effective uid. Look at the `Uid:` line in the process's `/proc/PID/status` and if it has different numbers in the different fields, that'll be a clue. — Alan Curry, Aug 28 '12 at 22:59
Well, everything is showing up as euid of 48... and `grep 48 /etc/passwd` yields `apache:x:48:48:Apache:/var/www:/sbin/nologin`... so I think I'm ok there... the apache user and group have `rwx` on the whole directory tree — Mike Pennington, Aug 28 '12 at 23:08

user9517 · Answer 1 · 2012-08-28T21:15:13.267

5

Take a look in /proc/<PID>/fd which should list all of the open files that PID has open.

On my CentOS system fd 7 is

lrwx------. 1 root root 64 Aug 28 22:01 7 -> socket:[1872522]

I can then use netstat -ane | grep 1872522 to get

tcp    0  0 :::443              :::*               LISTEN      0          1872522

You can use

lsof | grep 266474069

to get information on the pipe.

edited Aug 28 '12 at 21:15

answered Aug 28 '12 at 20:36

user9517

114,104
20
206
289

Symlinked to a pipe??? `/proc/9707/fd/7 -> pipe:[266474069]` – Mike Pennington Aug 28 '12 at 20:40
Uh, lain was faster, sorry. Yes, a file descriptor can also point to a pipe. – PythonLearner Aug 28 '12 at 20:41
@MikePennington: on my system it's a socket `lrwx------. 1 root root 64 Aug 28 21:34 7 -> socket:[1869276]` – user9517 Aug 28 '12 at 20:42
+1, good info... please see my update – Mike Pennington Aug 28 '12 at 22:05

score 3 · Answer 2 · answered Aug 28 '12 at 20:40

Looking at my little VPS, I can determine the fd number the following way:

 ll /proc/17684/fd/ |colrm 1 46

0 -> /dev/null
1 -> /dev/null
10 -> /var/www/vhosts/censored.xenuser.org/statistics/logs/error_log
11 -> /var/www/vhosts/censored.de/statistics/logs/error_log
12 -> /var/www/vhosts/censored.org/statistics/logs/error_log 
13 -> /var/www/vhosts/xenuser.org/statistics/logs/error_log
14 -> /var/log/apache2/access.log

[and so on, where 17684 is the PID of the process I straced earlier]

+1, please see my update – Mike Pennington Aug 28 '12 at 22:06 — Mike Pennington, Aug 28 '12 at 22:06

score 0 · Accepted Answer · answered Sep 13 '12 at 19:18

0

The problem was that I had SELINUX=enforcing in /etc/selinux/config.

After I set SELINUX=permissive, SELINUXTYPE=targeted, and rebooted wsgi can access all files correctly.

answered Sep 13 '12 at 19:18

Mike Pennington

8,266
9
41
86

WSGI cannot access a file, but permissions are correct

3 Answers3