Is there any way to monitor and kill Phusion child processes that are
using too much memory or cpu with monit?
It is a symptom of application leaking memory. I would rather fix the source of problem rather than expecting a tool to monitor and kill processes.
That said, Passenger Enterprise ( non-free product ) has the option to automatically kill the appserver process and respawn when the application reaches a certain amount of memory.
Note: I do not have any affliation with Phusion, the creator of Passenger. I'm just an user of their Enterprise product.