Infinite loop of cron jobs with intermediate output from each job

Question

I have a program that outputs stuff to standard output. I am using cron to schedule it which is fine because I am collecting the stuff that it prints outs via email and do some statistical work on that. I would like this to go on essentially forever.

Similar to question 538622, I would also like the job to "re-submit" itself when done. This can not be done via an infinite-loop script because I need to continously collect what gets emailed from cron.

If possible, I would also like to have N jobs going on in parallel when the computer has N CPU's.

So far, I have scheduled the jobs to run every X minute but since the computer is used by others, this is not very efficient use of time and there is a risk of filling up the machine with my jobs which take longer and longer time (so that X minutes in not enough).

Any idea on how to do this?

Answer 1

Based on the script in the referred question you can fetch command output to a file, and email the file content after one iteration completes. In that case you can have an infinite loop and when you start it using nohup into the background it can run as long as the host is up.

Sample script (called parse.sh in this case) can be:

while true
do
    command &> /tmp/output_thread_$1
    cat /tmp/output_thread_$1 | mail -s "output from thread $1" email@recipient
done

If you call the script using

for thread in 1 2 3 4; do nohup parse.sh $thread & done;

where 4 is the number of cpus. It will start (in this case) 4 processes doing your stuff