0

Debian 9.5 - postgresql cluster 9.4. It's not a fresh installation - it was a perfectly decently running system, until I tried restarting the service

all commands: /etc/init.d/postgresql start and service postgresql start and systemctl start postgresql behave the same: the the prompt hangs for about 10-15 seconds, and then nothing happens - postgres isn't running!

here's the syslog:

Jul 18 18:15:35 intranet systemd[1]: Starting PostgreSQL Cluster 9.4-main...
Jul 18 18:16:06 intranet postgresql@9.4-main[27610]: The PostgreSQL server failed to start. Please check the log output.
Jul 18 18:16:06 intranet systemd[1]: postgresql@9.4-main.service: Control process exited, code=exited status=1
Jul 18 18:16:06 intranet systemd[1]: Failed to start PostgreSQL Cluster 9.4-main.
Jul 18 18:16:06 intranet systemd[1]: postgresql@9.4-main.service: Unit entered failed state.
Jul 18 18:16:06 intranet systemd[1]: postgresql@9.4-main.service: Failed with result 'exit-code'.

The postgres-log (/var/log/postgresql/) is empty.

journalctl -xe:

-- Unit postgresql@9.4-main.service has begun starting up.
Jul 18 18:16:06 intranet postgresql@9.4-main[27610]: The PostgreSQL server failed to start. Please check the log output.
Jul 18 18:16:06 intranet systemd[1]: postgresql@9.4-main.service: Control process exited, code=exited status=1
Jul 18 18:16:06 intranet systemd[1]: Failed to start PostgreSQL Cluster 9.4-main.
-- Subject: Unit postgresql@9.4-main.service has failed

systemctl status postgresql

● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2018-07-18 17:42:02 CEST; 37min ago
 Main PID: 29934 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/postgresql.service

Jul 18 17:42:02 intranet systemd[1]: Starting PostgreSQL RDBMS...
Jul 18 17:42:02 intranet systemd[1]: Started PostgreSQL RDBMS.

systemctl status postgresql@9.4-main

● postgresql@9.4-main.service - PostgreSQL Cluster 9.4-main
   Loaded: loaded (/lib/systemd/system/postgresql@.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-07-18 18:16:06 CEST; 1 day 14h ago
  Process: 27610 ExecStart=postgresql@9.4-main --skip-systemctl-redirect 9.4-main start (code=exited, status=1/FAILURE)

Jul 18 18:15:35 intranet systemd[1]: Starting PostgreSQL Cluster 9.4-main...
Jul 18 18:16:06 intranet postgresql@9.4-main[27610]: The PostgreSQL server failed to start. Please check the log output.
Jul 18 18:16:06 intranet systemd[1]: postgresql@9.4-main.service: Control process exited, code=exited status=1
Jul 18 18:16:06 intranet systemd[1]: Failed to start PostgreSQL Cluster 9.4-main.
Jul 18 18:16:06 intranet systemd[1]: postgresql@9.4-main.service: Unit entered failed state.
Jul 18 18:16:06 intranet systemd[1]: postgresql@9.4-main.service: Failed with result 'exit-code'.

So there are absolutely no indicators about what's wrong. Strangely, the system was running without any problems all along - my application just crashed, and I tried restarting postgres - and now it wont start again.

After several hours of searching, I discovered, that strangely, it can be launched manually.

I logged as the postgres-user and used the command:

/usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/9.4/main -c config_file=/etc/postgresql/9.4/main/postgresql.conf

it just works fine - no errors, no thing. I even tried adding the '-d 3' to get more detailed output ....however there is nothing out of the ordinary. it runs fine - the problem are just the init-scripts. How do I fix this - or at least find out the root of the problem?

RazorHail
  • 165
  • 1
  • 8
  • Use `systemctl status postgresql@9.4-main` and/or look at log files in `/var/log/postgresql` – Daniel Vérité Jul 18 '18 at 17:54
  • @DanielVérité I've added the out for `systemctl status postgresql@9.4-main` to the original post. The path `/var/log/postgresql` however doesn't contain any new logs – RazorHail Jul 20 '18 at 07:01
  • Please show the systemd unit configuration file of postgres in `/lib/systemd/system/postgresql@.service`. Also, have you tried starting by `systemctl start postgresql@9.4-main.service`? – Fanatique Jul 20 '18 at 07:43
  • The part of interest in `/var/log/postgresql/postgresql-9.4-main.log` is what was written between `Jul 18 18:15:35` and `Jul 18 18:16:06`. Another question would be whether it still fails to start with systemd. One theory is that systemd was not patient enough with PG's recovery (can be long following an unclean shutdown on a busy db). In that case it would work now. – Daniel Vérité Jul 20 '18 at 08:57

1 Answers1

2

I think I solved this. Thanks for the commenters for guiding me towards finding the problem.

The permissions for /var/log/postgresql were somehow changed. The folder (and the log-files in it) wasn't owned by the postgres-user anymore. So apparently it wasn't able to write/open the log anymore.

After doing a chown postgres:postgres /var/log/postgresql/ -R, I was able to launch postgres via init-scripts

RazorHail
  • 165
  • 1
  • 8