I have a Nextcloud server running NextCloudPi (NCP) on a Raspberry Pi 3. NCP is Raspbian/Apache2/PHP/MariaDB stack so to speak. It's run reliably for years. But recently my Nextcloud died! All it does on a page load is report:
Internal Server Error
The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.
Really? More details can be found in the server log? Not by me. I have tried everything and looked everywhere. Here is my sledge hammer approach:
First I make sure to exclude all proxies by accessing it locally by IP. I can and do access my server through a global domain name but that's farmed out through a local reverse proxy and so I can and do access it by an internal .lan name or it's LAN IP as well without problem, at will, regularly. All no dramas. Been working for years.
What's caused this? At a mild loss, but suspect I did an update, with ncp-update. Not 100% sure. But that's a candidate. Cause becomes by-the-by though as no matter what broke it, I should be able to diagnose it. So here goes:
I'm on a bash terminal on the nextcloud server and I do this:
- Store a time reference:
touch /tmp/foobar
- force a page reload in my case I have https://192.168.0.14 open and CTRL+F5 in Firefox and the page reloads and I see the above "Internal server error"
- Search for any filesystem changes since the reference time:
sudo find / -regextype posix-extended -regex "/(sys|srv|proc|dev)" -prune -o -newer /tmp/foobar
Alas all this yields is:
/run/sudo/ts/cirrus
/dev
/sys
/var/log/apache2/other_vhosts_access.log
/var/log/apache2/nc-access.log
/var/log/auth.log
/proc
/srv
in short the ONLY logs I see changed are access logs and the sudo auth logs.
How can this be? How can apache be issuing an internal server error and leave no error log?
What gives? What can I learn from this?
Is this Apache, or the PHP subsystem at play? What can I do to get a deeper diagnostic out of it, given "More details can be found in the server log." is clearly untrue.
Also of note is that ErrorLogs are not disabled and I do use them from time to time, but to be sure:
$ grep -R ErrorLog /etc/apache2/
/etc/apache2/apache2.conf:# ErrorLog: The location of the error log file.
/etc/apache2/apache2.conf:# If you do not specify an ErrorLog directive within a <VirtualHost>
/etc/apache2/apache2.conf:ErrorLog ${APACHE_LOG_DIR}/error.log
/etc/apache2/sites-enabled/nextcloud.conf: ErrorLog /var/log/apache2/nc-error.log
/etc/apache2/sites-available/default-ssl.conf: ErrorLog ${APACHE_LOG_DIR}/error.log
/etc/apache2/sites-available/nextcloud.conf: ErrorLog /var/log/apache2/nc-error.log