0

I need to launch a command when two files from different sources are present. Every file could arrive in different time, but I want to trigger the command when both were received. I have tried to do it with incrond, watching both directories for IN_CLOSE_WRITE and IN_MOVED_TO

/dir/path1 IN_CLOSE_WRITE,IN_MOVED_TO command
/dir/path2 IN_CLOSE_WRITE,IN_MOVED_TO command

The question is: when the first watch is triggered, how to wait for the second?

For example:

Time: 12:05 UTC - File1 arrives to Path1 command is waiting for Path2 watch

Time: 12:09 UTC - File2 arrives to Path2 command is launched

The command is coded in Go, but I could not find anything that help me.

More specifically, the system works as follows: There are two remote servers recording data in pcap, in periods of 6 hours, same periods. They record the same information, but from different VLANs. When the recordings have finished, they are uploaded to two different directories of the main server. Both recordings need to be uploaded to be able to compare and mix them, discarding duplicate packets. Recordings are not always uploaded in the same order, nor do they arrive at the same time. It depends on the communications between the remote servers and the main server.

The file names are allways the same pattern: _yyyymmddTHHMMSS.pcap. From example, files started recording 2020-10-15 18:01:00:

File1: vlan1_20201015T180100.pcap

File2: vlan2_20201015T180100.pcap

I am looking for a solution related with incrond and bash or a solution within the command in Go. A solution or a clue, since I am currently at a standstill.

Jose Ángel de Pascual Viciana
  • 103
  • 4
  • can you provide more details on the criteria for which makes the files match? is there a specific order, or can either file come first/second? are they always named the same, or is there a pattern? – Tom Oct 15 '20 at 17:07
  • There are two remote servers recording data in pcap, in periods of 6 hours, same periods. They record the same information, but from different VLANs. When the recordings have finished, they are uploaded to two different directories. Both recordings need to be uploaded to be able to compare and mix them, discarding duplicate packets. Recordings are not always uploaded in the same order, nor do they arrive at the same time. It depends on the communications between the remote servers and the main server. The name of the files are _yyyymmddTHHMMSS.pcap – Jose Ángel de Pascual Viciana Oct 15 '20 at 17:20

2 Answers2

1

I would create a wrapper script that was called by incrond, something like

if [ -f /dir/path1 -a -f /dir/path2 ]; then
  command
else
  echo "both files don't exist yet"
fi

This would run every time either file triggered IN_CLOSE_WRITE or IN_MOVED_TO, but would only run when both files were present.

Update

based on the comments, it seem like tracking the completed state of the uploaded file is necessary. (I didn't test the following) The idea is to record the completion of the uploaded file, and

#!/bin/bash

# $@ set to event filename by incrond
filepath=$@

# this assumes both files are in the same directory, otherwise you would
# have to do some logic which switches directories as well as filename

# check file matches pattern
if [[ "${filepath}" =~ vlan.*\.pcap$ ]]; then

  # mark current file as completed
  if [ ! -f "${filepath}.complete" ]; then
    touch "${filepath}.complete"
  fi

  filename=$(basename ${filepath})
  dirname=$(dirname ${filepath})

  # find other filename by toggling vlan number
  vlan_num=${filename:4:1}
  [[ "${vlan_num}" == "1" ]] && vlan_alt=2 || vlan_alt=1
  # construct the other filename
  other_file=${dirname}$/{filename:0:4}${vlan_alt}${filename:5}
  echo $other_file;

  # see if other filename completion file exists
  if [ -f "${other_file}.complete" ]; then
    command
  else
    # other file not uploaded yet
    echo "completion file for both files not exists"
  fi

else
  echo "file didn't match pattern"

fi
Tom
  • 10,886
  • 5
  • 39
  • 62
  • what happen if Path1 have triggered IN_CLOSE_WRITE with File1 but File2 is uploading yet? I think the command would be executed but file2 would not be complete. And the command will be triggered again when path2 will trigger IN_CLOSE_WRITE with File2. – Jose Ángel de Pascual Viciana Oct 15 '20 at 17:36
  • If the files are progressively uploaded to their final destination by repeatedly opening, appending data, and closing, then I would expect IN_CLOSE_WRITE to be called multiple times, which would create a problem. There would then be no way to know when the file was actually finished – Tom Oct 15 '20 at 17:45
  • No, each file only trigger one IN_CLOSE_WRITE. But they could be uploaded at the same time, and when one file is completed, the other file is uploading yet. I think I could check that with lsof. – Jose Ángel de Pascual Viciana Oct 15 '20 at 17:53
  • re your first comment, "what happen if Path1 have triggered IN_CLOSE_WRITE with File1 but File2 is uploading yet? I think the command would be executed but file2 would not be complete." ... the wrapper script would be run twice, but the command only once, because `[ -f /dir/path1 -a -f /dir/path2 ]` is checking that both files are present, which they would be not, while File2 is still uploading. maybe I am misunderstanding your comment – Tom Oct 15 '20 at 17:56
  • ls return filename while uploading and size is increasing. I don't really know if -f test only files closed. – Jose Ángel de Pascual Viciana Oct 15 '20 at 18:02
  • Ah, in that case, then you would need to track the state, and wait until IN_CLOSE_WRITE had been called twice, once for each file. – Tom Oct 15 '20 at 18:13
  • I have tested it and it works like a charm. It is a good solution without the need to be root. However, I have coded a solution based on lsof. – Jose Ángel de Pascual Viciana Oct 20 '20 at 16:04
0

Another workaround using lsof. This solution must be run as root.

root incrontab

/path/to/vlan1  IN_CLOSE_WRITE,IN_MOVED_TO  /usr/local/bin/monitor.sh $@/$# /path/to/vlan2
/path/to/vlan2  IN_CLOSE_WRITE,IN_MOVED_TO   /usr/local/bin/monitor.sh $@/$# /path/to/vlan1

Wrapper script

#!/bin/bash
# monitor.sh
FILE_FINISHED=$1
FIND_PATH=$2

f=`basename $FILE_FINISHED`

FILE2=`ls $FIND_PATH/*${f:11:13}*`
STATUS=$?
if [ $STATUS -eq 0  ]
then
    lsof $FILE2 > /dev/null 2>&1
    STATUS=$?
    if [ $STATUS -ne 0 ]
    then
        echo "Command"
    else
        echo "Still uploading"
    fi
else
    echo "File not exist"
fi
Jose Ángel de Pascual Viciana
  • 103
  • 4