3

I have a Nextcloud server running NextCloudPi (NCP) on a Raspberry Pi 3. NCP is Raspbian/Apache2/PHP/MariaDB stack so to speak. It's run reliably for years. But recently my Nextcloud died! All it does on a page load is report:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

Really? More details can be found in the server log? Not by me. I have tried everything and looked everywhere. Here is my sledge hammer approach:

First I make sure to exclude all proxies by accessing it locally by IP. I can and do access my server through a global domain name but that's farmed out through a local reverse proxy and so I can and do access it by an internal .lan name or it's LAN IP as well without problem, at will, regularly. All no dramas. Been working for years.

What's caused this? At a mild loss, but suspect I did an update, with ncp-update. Not 100% sure. But that's a candidate. Cause becomes by-the-by though as no matter what broke it, I should be able to diagnose it. So here goes:

I'm on a bash terminal on the nextcloud server and I do this:

  1. Store a time reference: touch /tmp/foobar
  2. force a page reload in my case I have https://192.168.0.14 open and CTRL+F5 in Firefox and the page reloads and I see the above "Internal server error"
  3. Search for any filesystem changes since the reference time: sudo find / -regextype posix-extended -regex "/(sys|srv|proc|dev)" -prune -o -newer /tmp/foobar

Alas all this yields is:

/run/sudo/ts/cirrus
/dev
/sys
/var/log/apache2/other_vhosts_access.log
/var/log/apache2/nc-access.log
/var/log/auth.log
/proc
/srv

in short the ONLY logs I see changed are access logs and the sudo auth logs.

How can this be? How can apache be issuing an internal server error and leave no error log?

What gives? What can I learn from this?

Is this Apache, or the PHP subsystem at play? What can I do to get a deeper diagnostic out of it, given "More details can be found in the server log." is clearly untrue.

Also of note is that ErrorLogs are not disabled and I do use them from time to time, but to be sure:

$ grep -R ErrorLog /etc/apache2/
/etc/apache2/apache2.conf:# ErrorLog: The location of the error log file.
/etc/apache2/apache2.conf:# If you do not specify an ErrorLog directive within a <VirtualHost>
/etc/apache2/apache2.conf:ErrorLog ${APACHE_LOG_DIR}/error.log
/etc/apache2/sites-enabled/nextcloud.conf:    ErrorLog  /var/log/apache2/nc-error.log
/etc/apache2/sites-available/default-ssl.conf:      ErrorLog ${APACHE_LOG_DIR}/error.log
/etc/apache2/sites-available/nextcloud.conf:    ErrorLog  /var/log/apache2/nc-error.log
Bernd Wechner
  • 151
  • 8
  • It is certainly possible to configure Apache not to log errors. I've never used a Raspberry Pi before, so I don't know how Apache is typically installed or configured on it. However, you should be able to grep the Apache conf files for `ErrorLog` to see which log file it is supposed to be writing errors to (hopefully not `/dev/null`). – Stephen Ostermiller Nov 27 '19 at 00:06
  • Thanks for the thought. Indeed that didn't escape me, but there are error logs present, I should have mentioned and I can see errors in them and use them time to time to diagnose other issues. But I will add to the question a quick note about that! Thanks. – Bernd Wechner Nov 27 '19 at 00:21
  • 1
    Increase the log level. – Gerald Schneider Nov 27 '19 at 07:19
  • A great idea, thanks. So I upped it to debug. Alas all that does is indeed show a pile of output in the error logs, but nothing of use :-(. Just a very long run of `authorization result of Require all granted: granted` and `authorization result of : granted` messages. No indication of an actual error :-(. I could up it to a trace level but would be looking at volumes of output to analyze and still no error clues I expect. – Bernd Wechner Nov 27 '19 at 09:35
  • You could check if php error logging is enabled like this: https://askubuntu.com/a/185291 I don't know how it is in apache, but nginx usually has a separate error log to which php errors are written. I also recommend to test your setup independently from nextcloud by creating a php file containing something like ` – zuim Dec 03 '19 at 13:46
  • Thanks. I checked and error logging not enabled as per that link. Which might explain the lack of logs? Alas hard to check for sure now as I happened upon a likely culprit and fix. In `/etc/apache2/conf-enabled` were two files `php7.3-fpm.conf` and `php7.0-fpm.conf`. I deleted the latter and have been running error free for two days since. Looking hopeful. If I find time I may put it back enable logging and see if I get logged errors and then file an answer to the original question (about how it's possible not to have logs!). – Bernd Wechner Dec 11 '19 at 06:31

0 Answers0