4

I am trying to analyze a series of application requests. There are web logs, application server logs and data server logs, all with slightly different timestamps on each line. I'd like to splice them together to see the various logs in the order they occurred.

Before I whip up a script (using awk and friends) to do this, is there a tool or existing project/recipe that does something similar?

Rog
  • 235
  • 1
  • 8

3 Answers3

2

Check out Splunk.

dmourati
  • 24,720
  • 2
  • 40
  • 69
1

Disclaimer: i haven't done this before.

I think syslod is the deamon for logging in *nix systems, maybe there are better ones. But essentially works for similar goals. I think if your application can be configured to use syslogd instead of they having their own logging file then i think you can have a centralized place to see , compare all the logs. This requires you to know the configuration options of your application or tweaking its code if it does not have the options yet and knowledge about configuration options in syslogd or other similar tools. Happy hunting and learning :) .I would be interested to know how this is actually implemented. Please post the steps you took to achieve this is a blog or website or something so that can be found through google.

bagavadhar
  • 538
  • 4
  • 14
  • I'm familiar with syslogd, and you're correct that it would do the job if the various services were (and could be) configured to use it. This is more of a one-time analysis task where the logs are already written, and some of the services aren't easy to wire up to syslogd. Good thoughts – Rog Jun 21 '11 at 23:40
0

I ended up writing a python script.

Given a series of files, the script determines the type of file in each case (looking for keywords in the filename such as access_log ), and based on that type extracts a consistent timestamp value for each line. The script then splices the lines from each file into a sorted list of tuples (timestamp, file, log line), optionally filters for a given time period and prints the result.

Rog
  • 235
  • 1
  • 8
  • Hi Rog, I feel it will be great if you could publish the script. As i think it will be helpful to many. I know there are many situation that a sys admin comes across when he have to review mutliple logs simulatenously. What i do for that is., I do a screen session and split the window and tail -f the logs and see how they are reacting and solve the problem. But the script in your way will be much easier than my approach. So can you please publish the script ? – bagavadhar Jun 22 '11 at 12:02