0

The Issue (Modified: with REPRO STEPS!)

To repro, spin up a new VPS with Ubuntu 16.04 (probably works with any Debian distro, but not tested) and follow these instructions:

Through experimentation I boiled down the instructions to the absolute minimum necessary to repro

apt-get install php7.0 php7.0-dev php-pear
apt-get install libsodium-dev
pecl install libsodium
echo "extension=libsodium.so" >> /etc/php/7.0/fpm/php.ini
shutdown -r 0

When I reboot the server (shutdown -r 0) PHP-FPM is dead (confirm via service php7.0-fpm status)

If I run service php7.0-fpm start, it starts up in a matter of milliseconds with no issue.

I've created an issue on the libsodium-php tracker in hopes someone can find a solution there: https://github.com/jedisct1/libsodium-php/issues/94

The Question

Now that I've narrowed down the issue, what I'm really looking for is either:

  1. A way to figure out why it's timing out (is there a good way to profile what resources are being accessed or what a boot-time process is doing?)

  2. A way to fix it (is there some way to alter the boot order so that libsodium boots last, in case it's a dependency issue?)

New Updates

I've continued to look into this and figured out that all of PHP is stalling during the initialization of the libsodium library (the call to sodium_init()). This call takes about 3-4 minutes to complete, and until this time has passed even a call to php -v fails.

As a workaround I have added the following to /lib/systemd/system/php7.0-fpm.service:

TimeoutStartSec=600

This causes my web server to return HTTP status code 502 for a couple of minutes after a reboot but eventually it starts working again (better than manually having to log in and fix it)

I will try to get in contact with the developers of libsodium and libsodium-php to find a better solution, however for now I believe I have solved this myself:

The Solution

  1. When something isn't running at startup, begin investigation with journalctl -u <service>

  2. If it's timing out, try a fresh install -- disable all extensions, modules, plugins, remove custom config, etc. If the fresh install works, enable things one at a time until you find the cause

  3. If you absolutely need the plugin / extension / whatever then you can increase the timeout by modifying the *.service file and adding TimeoutStartSec=...

stevendesu
  • 113
  • 9
  • Any reason the server info was removed? It's a VPS that was spun up earlier this morning and has no personal info, no client code,... nothing of value on it. At worst if a bot scrapes this site and starts using the server to DDoS something I could click a button and destroy the server in seconds. – stevendesu Jul 27 '16 at 20:47
  • Even if you're genuinely looking for help, posting credentials on a public forum is irresponsible. If you want help, seek it privately, from someone you trust. – EEAA Jul 27 '16 at 20:48
  • @EEAA: Posting my personal passwords would be irresponsible, giving root access to a server on a private network behind a firewall would be irresponsible, giving access to just about anything else might be irresponsible. As it was, I was giving access to a brand new server with nothing on it. Someone could get the same thing by spending 0.7 cents per hour at DigitalOcean. @Michael_Hampton: I'll post the results of `systemctl status php7.0-fpm` in the original post – stevendesu Jul 27 '16 at 20:50
  • 1
    Even if it was a new server with nothing on it, if you give the whole world access, they can compromise the server within seconds and without your knowledge, so you would have to destroy it anyway, and start all over with a new instance. You would never be able to trust that it hadn't been compromised. At that point you would have to do all of this work again. – Michael Hampton Jul 27 '16 at 20:55
  • @MichaelHampton: "all of this work again" was literally 60 seconds of `apt-get` and `wget ... | tar -xvz`. I wasn't concerned about sharing access because if doing so would shave 5 minutes off the debug time it would be a net gain for me. Besides, I created an image of the hard drive prior to changing the root password so I can spin up a duplicate configuration on a different server with a different public IP in seconds. Believe it or not, I thought this out :) – stevendesu Jul 27 '16 at 20:57
  • That may be so, but we couldn't know that. We have a long history of getting people who post their credentials without thinking it out, and so we remove those instantly whenever we come across them. – Michael Hampton Jul 27 '16 at 20:58
  • @MichaelHampton: Fair enough. 99% of the people who post their root password probably don't even know what a hard drive image is, so I guess it's better to err on the side of caution. – stevendesu Jul 27 '16 at 21:00
  • Welcome to Serverfault! Your question is about Ubuntu. You would probably get better expertise, and therefore better answers on [Ask Ubuntu](http://askubuntu.com). Please flag your question to be migrated, or delete your question and re-ask it over there. Please do not cross-post (i.e. ask your question on [Ask Ubuntu](http://askubuntu.com) before you delete here or your question is migrated). You may ask your question there once you have deleted your question here. – fosslinux Jul 28 '16 at 02:29
  • 1
    @ubashu I feel my question is more about `systemd` / `init.d` than specifically Ubuntu – stevendesu Jul 28 '16 at 02:31
  • Hmm. Your choice. I haven't (and won't) flag your question to be migrated. – fosslinux Jul 28 '16 at 02:36

1 Answers1

0

The issue lies with the libsodium PHP extension. I have initiated conversation with the developer of libsodium to work on a fix. For now:

The Solution

  1. When something isn't running at startup, begin investigation with journalctl -u <service>

  2. If it's timing out, try a fresh install -- disable all extensions, modules, plugins, remove custom config, etc. If the fresh install works, enable things one at a time until you find the cause

  3. If you absolutely need the plugin / extension / whatever then you can increase the timeout by modifying the *.service file and adding TimeoutStartSec=...

stevendesu
  • 113
  • 9