script to count the occurence of the particular string in the given time interval

Question

We are trying to write a script "sendemail.sh" to count the number of occurrence of a particular string in a log file "SendEmail.log" within the given interval. We have a log file. In that we are searching for a pattern "ReqInputMsgLog" and need to count the number of times it occurred in the given period for eg: from "2014-08-19 11:30" to "2014-08-19 11:34". And our script look like this:

#!/bin/sh
enterdate=$1
echo $enterdate
enddate=$2
enterdate1=`date +%s -d $enterdate +"%Y-%m-%d %H:%M"`

echo $enterdate1
enddate1=`date +%s -d $enddate +"%Y-%m-%d %H:%M"`
echo $enddate
count=0
cat SendEmail.log | grep "ReqInputMsgLog" | awk -F "[" '{print $3}' | awk -F "," '{print $1}' > /con/scripts_server/file.txt
for line in `cat /con/scripts_server/file.txt`
do
logdate=`echo $line | awk -F : '{print $1":"$2}'`
if [[ $logdate < $enddate1 ]];
        then
        count=`expr $count + 1`
        fi
done
echo $count

But when we are trying to execute the script by the below command its not showing the proper count.

./sendemail.sh "2014-08-19 11:30" "2014-08-19 11:34"

Log file is very big one. Small chunk has been posted here.

INFO [SIBJMSRAThreadPool : 5] [2014-08-19 11:18:24,471] SendEmail - 8/19/14 11:18 AM,ECCF25B0-0147-4000-E000-1B830A3C05A9,ReqInputMsgLog,SendEmail,<?xml version="1.0" encoding="UTF-8"?>
<in:sendEmailRequestMsg xmlns:in="http://EmailMed/EmailMedInterface" xmlns:ns0="wsdl.http://EmailMed/EmailMedInterface" xmlns:ns1="http://EmailMed/EmailMedInterface" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:me="wsdl.http://EmailMed/EmailMedInterface" xsi:type="me:sendEmailRequestMsg">
<in:sendEmail xmlns:xci0="http://EmailMed/EmailMedInterface">
INFO [SIBJMSRAThreadPool : 7] [2014-08-19 11:18:14,235] SendEmail - 8/19/14 11:18 AM,ECCEFDB2-0147-4000-E000-1B830A3C05A9,ReqInputMsgLog,SendEmail,<?xml version="1.0" encoding="UTF-8"?>
<in:sendEmailRequestMsg xmlns:in="http://EmailMed/EmailMedInterface" xmlns:ns0="wsdl.http://EmailMed/EmailMedInterface" xmlns:ns1="http://EmailMed/EmailMedInterface" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:me="wsdl.http://EmailMed/EmailMedInterface" xsi:type="me:sendEmailRequestMsg">
<in:sendEmail xmlns:xci0="http://EmailMed/EmailMedInterface">
INFO [SIBJMSRAThreadPool : 7] [2014-08-19 11:18:14,241] SendEmail - xmlText: <?xml version="1.0" encoding="UTF-8"?>

after awk command we will get a file "/con/scripts_server/file.txt" which looks similar like below:

2014-08-19 11:28:03
2014-08-19 11:28:06
2014-08-19 11:28:17
2014-08-19 11:28:53
2014-08-19 11:29:02
2014-08-19 11:29:47
2014-08-19 11:29:57
2014-08-19 11:30:07
2014-08-19 11:30:17
2014-08-19 11:30:19
2014-08-19 11:30:19
2014-08-19 11:30:22
2014-08-19 11:30:25
2014-08-19 11:30:25
2014-08-19 11:30:36
2014-08-19 11:30:51
2014-08-19 11:30:56
2014-08-19 11:30:59
2014-08-19 11:30:59
2014-08-19 11:31:08
2014-08-19 11:31:25
2014-08-19 11:32:19
2014-08-19 11:32:22
2014-08-19 11:32:27
2014-08-19 11:32:28
2014-08-19 11:32:41
2014-08-19 11:32:49
2014-08-19 11:32:59
2014-08-19 11:33:27
2014-08-19 11:33:41
2014-08-19 11:34:07
2014-08-19 11:34:14
2014-08-19 11:34:21
2014-08-19 11:34:25
2014-08-19 11:34:38
2014-08-19 11:34:50
2014-08-19 11:34:58

Seeing your `/con/scripts_server/file.txt` there is an easy way to achieve this: Try it on the `file.txt` directly `awk '/2014-08-19 11:30/,/2014-08-19 11:34/{count++} END{ printf "There are %s lines\n", count}' /con/scripts_server/file.txt` — Valentin Bajrami, Aug 20 '14 at 11:19
Thank.. It worked... But it is showing the incorrect value We have 30 entries from 2014-08-19 11:30 to 2014-08-19 11:34 but it is showing as 24 lines. 2014-08-19 11:30 2014-08-19 11:34 There are 24 lines Please help us to get the correct value. — pruthvi, Aug 21 '14 at 04:52
I edited my answer. The `"${1%:*}"` removes the last two digits including the colon. So that means seconds were removed. — Valentin Bajrami, Aug 21 '14 at 07:16

Valentin Bajrami · Answer 1 · 2014-08-21T07:15:06.040

Use the following to calculate the lines between two time variables. Put the following code in a file called countOcurrences.

#!/bin/bash

 awk "/$1/,/$2/"'{count++} END{ printf "There are %s lines\n",  count}' con/scripts_server/file.txt

Run it as follow.

./countOcurrences "2014-08-19 11:30:07" "2014-08-19 11:34:07"

If file.txt is filled with a new date/time each time a pattern match occurs, then the above will work.

score 1 · Answer 2 · answered Aug 22 '14 at 15:34

First of all I encountered 2 errors trying to recreate your problem date: extra operand 11:34:14' Trydate --help' for more information. ./script.sh: line 15: 1408448098: No such file or directory

From what I understand you isolated the dates which interest you into file.txt according to user input and you want to count occurrences in there.

I coded this:

#!/bin/bash

#Start/End dates to encolse count range
startDate="2014-08-19 11:28:00"
endDate="2014-08-19 11:35:00"
#Concert these dates to seconds since Epoch
startDateEpoch=$(date --date="$startDate" +%s)
endDateEpoch=$(date --date="$endDate" +%s)

#Read file.txt count occurences
while read line
do

processingDate=$(date --date="$line" +%s)
  if [ $processingDate -lt $endDateEpoch ] && \
  [ $processingDate -gt $startDateEpoch ]; then
    echo "APOEL FC";
  fi

done < file.txt

Your file.txt as appended here contains 37 lines, so:

sysadmin@omg:/tmp$ ./script.sh | wc
     37      74     333

This looks correct for

startDate="2014-08-19 11:28:00"
endDate="2014-08-19 11:35:00"

Changing the dates to:

startDate="2014-08-19 11:28:03"
endDate="2014-08-19 11:34:58"

sysadmin@omg:/tmp$ ./script.sh  | wc
     35      70     315

returns 35 occurrences which looks correct since first and last dates should be excluded.

So, the conversion to seconds since Epoch must have had something wrong in the syntax and the < operator in your if statement was causing a small mess.

As general guidelines prefer to use -lt -gt (less than, greater than) operators for if. Also try to avoid enclosing command in backticks - Prefer to use $(command).

Happy coding.

script to count the occurence of the particular string in the given time interval

2 Answers2