0

I have the following set up.

In a remote machine I have a process continuously running which subscribes to some data and writes it to stdout (lets call the process the subscriber). I call the subscriber from a .sh script which redirects stdout into a file which contains the date on the name, lets call that the logfile. Big picture, what I want to achieve is to break those logfiles into different dates so I can bring them from the remote machine to my local.

There's one thing which is particular to my situation that make the solution I'm familiar with not work: The logfile contains snapshot data plus a stream of deltas. The implication of this is that I can't just logrotate the logfile, because the beginning of the next logfile would be missing the snapshot. The way I've been dealing with this now is manually stop the subscriber, and restart it from the .sh script. The .sh script will redirect the stdout to the next dated logfile. Missing a few seconds of data in between restarts is not problematic for my purposes. What IS problematic would be having a logfile that is missing its initial snapshot.

So my question is: what is the correct way of automating this manual process?

I have a partial solution that I don't like.

  • have a new .sh script called from the crontab every day at a certain time
  • this script then kills all the processes by name (i.e. subscriber)
  • re-starts the subscriber with stdout pointing to new logfile, but instead of naming it .log name it .log.part, inspired by this answer.
  • then I would have to find out how to guarantee that when the subscriber dies, the .log.part logfile gets renamed to .log, signalling that that logfile is available to be moved out
  • then run rsync from the local machine to bring over all .log files.

Even if I found out how to do the last step, I don't like the solution because I've never seen a robust pipeline like this built on top of cronjobs.

So if someone knows a robust way to do this, what is it? And if not, how do I guarantee that a .log.part file gets renamed to .log when the subscriber dies?

Thank you

sedapon
  • 1

1 Answers1

0

Your present cron script likely looks something like this:

setsid subscriber > subscriber.date.log &

You want to have a shell wait around for subscriber to exit and then rename it's output. Something like this would do:

log_date_str=`date ...`

watcher()
{
        subscriber > subscriber.${log_date_str}.log.part
        mv subscriber.${log_date_str}.log.part subscriber.${log_date_str}.log
}

killall subscriber
setsid watcher &

Since watcher is started in the backgound it is in a separate shell process. The parent shell starts the shell function and then exits. The watch function, in it's own shell process, starts subscriber and then waits for it to finish. Once it does it preforms the rename.

There are a couple of gotchas:

If subscriber forks into the background like a daemon and isn't backgrounded by the shell script then you'll need to figure out how to keep it from doing so (-d is common). This is an unlikely gotcha as it would be strange to output to stdout and fork itself.

Second, if your shell doesn't have setsid built in you'll have to split watch into it's own shell script and call that instead of the function.

I'd recommend using pstree to see the parent child relation ship.

Also, make sure that you're not doing a set -e in your script (or turn it off in watcher) or the mv won't be done when subscriber exits with an error.

Ed Schaller
  • 1