5

I'm stumped. I have two log files being watched by the AWS CloudWatch agent. The first one, /var/log/nginx/access.log, works perfectly fine. The second, /var/log/otherserver/access.log, is not having any changes picked up. Not eventually, not ever UNLESS I restart the agent, whereupon it picks up the changes and sends them to CloudWatch as expected.

/var/log/otherserver/access.log is a log file rsync'ed periodically from another server that can't have the agent installed on it. Command looks like:

rsync -av user@host:/var/log/access.log /var/log/otherserver/access.log

The agent definitely can read it, because it reads the changes in after being restarted.

The position of the entry within the config file doesn't appear to matter.

The dates in the log entries for the rsync'ed log file are the same as the server (everything is UTC).

If I move the log file the agent starts complaining with:

2015-12-14 16:02:26,158 - cwlogs.push.stream - WARNING - 3344 - Thread-1 - No file is found with given path '/var/log/otherserver/access.log'.

The configuration of this second log file is almost identical to the first (below).

[website.access.log]
#datetime_format = 09/Dec/2015:14:15:02 +0000
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/otherserver/access.log
log_stream_name = master-platform.sh
initial_position = start_of_file
log_group_name = web-access

[app.access.log]
#datetime_format = 09/Dec/2015:14:15:02 +0000
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/nginx/lax.access.log
log_stream_name = {hostname}
initial_position = start_of_file
log_group_name = web-access

Does anyone have any clue what is going on here? Or can suggest an alternative agent that is less ... screwy?

lsh
  • 148
  • 1
  • 12
  • The only difference I can think of is that rsync probably overwrites the entire log file (as opposed to appending to it). Not sure how CloudWatch detectes changes but maybe it gets confused when the whole file is overwritten? Have you tried manually appending some lines `echo test > /var/log/otherserver/access.log` to it to see if the CloudWatch agent can pick those up? – Jukka Dec 22 '15 at 21:16
  • Good point, a little correction, for appending it should be: `echo test >> /var/log/otherserver/access.log`. – Diamond Dec 22 '15 at 22:05
  • Sorry for slowness in getting back, Christmas break. I tried appending and it didn't work and I should have included it in the question. I appended garbage, new log lines, the head contents of the same log file, the head contents of a similar log file, an entire other log file - nothing. I thought in the beginning the changes were simply too small to be picked up but this proved false as the number of bytes appended grew larger. – lsh Jan 04 '16 at 11:03
  • How did you end up solving this? – Parthapratim Neog Feb 20 '20 at 06:32
  • oh wow - this was a very long time ago. I ditched Cloudwatch Agent and went with syslog-ng sending logs to Loggly and went with NewRelic for monitoring. – lsh Feb 21 '20 at 07:05

1 Answers1

1

It's possible your agent state file is corrupted because you kept making changes to the configuration. There are two ways to fix this:

  • Option 1: Use a new name for your configuration block header.
    • That is, change [website.access.log] to [something.else].

  • Option 2: Delete the agent state file after stopping the service.

    sudo service awslogs stop
sudo rm /var/lib/awslogs/agent-state
sudo service awslogs start

Please note that Option 2 may initially cause duplicate logs to be pushed to CloudWatch as a new state file is created.

ADTC
  • 143
  • 2
  • 11