What could cause deeply nested apache prefork children?

Question

I'm running apache 2.2.25 with prefork and mod_perl on CentOS (mostly 6.4). Recently one instance went particularly haywire, forking off children of children of children:

<snip>
\_ /usr/sbin/httpd
|   \_ /usr/sbin/httpd
|   |   \_ /usr/sbin/httpd
|   |       \_ /usr/sbin/httpd
|   |           \_ /usr/sbin/httpd
|   |           |   \_ /usr/sbin/httpd
|   |           |       \_ /usr/sbin/httpd
|   |           |       |   \_ /usr/sbin/httpd
|   |           |       \_ /usr/sbin/httpd
|   |           \_ /usr/sbin/httpd
|   \_ /usr/sbin/httpd
|       \_ /usr/sbin/httpd
|       |   \_ /usr/sbin/httpd
|       |   |   \_ /usr/sbin/httpd
|       |   |   |   \_ /usr/sbin/httpd
|       |   |   \_ /usr/sbin/httpd
|       |   \_ /usr/sbin/httpd
|       \_ /usr/sbin/httpd
\_ /usr/sbin/httpd
|   \_ /usr/sbin/httpd
\_ /usr/sbin/httpd
\_ /usr/sbin/httpd
</snip>

At the same time the box ran out of memory (I'm not sure which came first, the deep forking or the memory running out).

I've never seen apache do this before and haven't yet narrowed down the cause. The two possibilities I'm investigating are a memory leak in mod_perl (or rather in our code which then persists because of mod_perl) or a security exploit that allows for some limited fork bomb (can't fork off arbitrary children but can fork off additional apache children).

I'm not sure what would cause apache to fork like this (edit I originally mentioned a mistaken belief about how graceful worked) but my thought here is that a circular ref type memory leak (either directly or indirectly by just using all the available memory on the box) caused some kind of mod_perl confusion which resulted in additional worker children forking from children instead of the main parent. I have no real reason for this, just attempting to reconcile the two problems (deep forking and memory usage).

Alternatively, we may have a security exploit that I need to track down.

Has anyone seen this type of behavior from apache before and know of a solution?

That part about a child becoming the new parent is not accurate. The parent is fixed during a graceful restart — covener, Jan 14 '14 at 01:47

score 0 · Accepted Answer · answered Jan 15 '14 at 19:11

It turns out this was code forking under mod_perl (which is already a bad idea) and then making it worse by not cleaning up the child. Under mod_perl, when you fork, you're not forking your code, you are forking apache. So then that child goes on to continue serving content as any other forked apache child but doesn't get included in the count and therefore won't ever get culled if you get too many instances.

What could cause deeply nested apache prefork children?

1 Answers1