I have Rabbitmq server version 3.5.7-1ubuntu0.16.04.2 installed on Ubuntu 16.04 server.
When the server is rebooted Rabbitmq randomly fails to start.
Rabbit server is installed from official Ubuntu repository.
There is no clustering, it is just one independent server.
When the problem happens this is what gets logged to Journalctl on reboot:
Jun 03 09:50:52 hostname systemd[1]: Starting RabbitMQ Messaging Server...
...
Jun 03 09:51:23 hostname rabbitmq[1067]: Waiting for 'rabbit@hostname' ...
Jun 03 09:51:24 hostname rabbitmq[1067]: pid is 2455 ...
At 09:51:29 application that tries to use Rabbit is unable to connect: [Errno 111] Connection refused
Few seconds later Rabbit is started according to /var/log/rabbitmq/rabbit@hostname.log:
=INFO REPORT==== 3-Jun-2018::09:52:22 ===
Server startup complete; 6 plugins started.
* rabbitmq_management
* rabbitmq_web_dispatch
* webmachine
* rabbitmq_management_agent
* mochiweb
* amqp_client
But at the same moment, Rabbit goes into error state because of Start-post operation timing out:
Jun 03 09:52:22 hostname systemd[1]: rabbitmq-server.service: Start-post operation timed out. Stopping.
Jun 03 09:52:22 hostname systemd[1]: Failed to start RabbitMQ Messaging Server.
Jun 03 09:52:22 hostname systemd[1]: rabbitmq-server.service: Unit entered failed state.
Jun 03 09:52:22 hostname systemd[1]: rabbitmq-server.service: Failed with result 'timeout'.
Logs in /var/log/rabbit don't show any errors warnings.
Looking at the Systemd unit file I see that Startpost operation executes script /usr/lib/rabbitmq/bin/rabbitmq-server-wait which contains following code:
. `dirname $0`/rabbitmq-env
/usr/lib/rabbitmq/bin/rabbitmqctl wait $RABBITMQ_PID_FILE
After this running command service rabbitmq-server start starts the service without problems.
I am unable to understand what is purpose of this Start-post operation and why it is timing out. What could be causing the problem and how should I proceed researching it?
