netcat output into separate files using string delimiters

Question

I want netcat to pipe the output of a tcp/ip connection to commands that examine the output and split it into distinct files separated by the strings "msg" and "/msg" from the output, something like

nc host port | while :
do
split 'msg' '/msg' > timestamp.txt
done

but I can't get the syntax right. In an ideal world, I'd also like the whole of the output of nc appending to another file, say big_data.txt. Maybe I should use grep?

I know how to send nc output to a file and can work out how to split it, but the file will keep appending for 12 hours each day. Also, I think it's more efficient to split the data as it arrives.

Answer 1

I did it all in R.

    stream_processor <- function(host_,port)
     con2 <- socketConnection(host=host_, port=port_)
    # as non-blocking, may need to loop for input
    readLines(con2)
    while(isIncomplete(con2)) {
        Sys.sleep(1)
        z <- unlist(readLines(con2,1000))
     write(z, file=paste(make.names(date()),'.stream',sep=''))
  }
}