1

I have been running this instance of apache 2.4.17 for a long time. It is used as a proxy to a number of applications at the back. Recently I upgraded Piwik/Matomo, and it caused a bit of issue in the apache, but now it shoulbe fixed. However, after restart the physical server and apache instances numerous times, it will still crash apache pretty badly after a number of hours later, and need to apachectl restart to get it going again.

Checked apache's error.log, and there are quite a few entries like this:

[Sun Mar 21 16:15:21.126158 2021] [ssl:info] [pid 1274] [client  x}:0] AH01998: Connection closed to child 28 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.136567 2021] [ssl:info] [pid 1271] [client 216.245.221.83:44352] AH01998: Connection closed to child 25 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.146466 2021] [ssl:info] [pid 1270] [client ?x}:0] AH01998: Connection closed to child 24 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.153012 2021] [ssl:info] [pid 1269] [client  x}:0] AH01998: Connection closed to child 23 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.164055 2021] [ssl:info] [pid 1262] [client ?v}:0] AH01998: Connection closed to child 16 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.167462 2021] [ssl:info] [pid 1263] [client ?2):0] AH01998: Connection closed to child 17 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.169270 2021] [ssl:info] [pid 1264] [client ?x}:0] AH01998: Connection closed to child 18 with abortive shutdown (server <domain>:443)

where the client IP value is randomised... x}, ?x}, ?v} and etc.

There aren't anything strange in applications' log files.

How should I troubleshoot this problem?

charleslcso
  • 21
  • 3

1 Answers1

1

The abortive shutdown messages just mean that the SSL sessions didn't terminate via the normal process - so a malformed client, or a robot, etc could just be abandoning connections. It's typically indicative of a client issue of some sort. Unless there are a ton of them, and users reporting issues, it's probably not your cause.

For further troubleshooting...

What does dmesg say around the time of the crashes?

Is there a pattern at all to the crashes?

Any cron jobs running?

What do system resources look like before you restart the failed Apache service?

Have you set the logs to full debug to see what it says?

tilleyc
  • 914
  • 4
  • 11
  • 1. Will try dmesg. It is available on my osx. But it is hard to sit there and issue the command at around the time apache crashes. 2. I'm looking for patterns too... can't spot any obvious ones. 3. 2 simple cron jobs are running in the apache host, but the crashes don't coincide with the job run time. 4. sufficient memory, volume space left. Haven't checked all system parameters though 5. just set apache log to debug and left it over night. – charleslcso Mar 22 '21 at 02:45
  • If some bad bots call my server and abort the SSL connections frequently enough then it will continue to crash my apache. Is there any way to fix this? – charleslcso Mar 22 '21 at 02:46
  • You can't absolutely say that the aborted connections are a root cause. At most, they're a symptom, but you can't even definitively say that until you investigate and see exactly what is causing the crash. More often than not, these logs aren't an issue, so I would not begin thinking of it as the issue and chasing that until you have something for sure pointing to it. – tilleyc Mar 22 '21 at 16:19