1

After running a simulation(using a simulator(QualNet)) of a simulated network I end up with ip stats stored in a database, I then extract the data to a csv file So now I have 750mb of raw network stats(time stamp, packet id, source ip, source port, protocol, etc). What are the common ways of analyzing large amounts of data like above, if you want to know things like packet loss, throughput, delay, congestion, etc.

user90500
  • 111
  • 3
  • I've had great success using something called an "IT bitch" for similar tasks. If you use a smart IT bitch, it will usually employ a number of useful tools to complete the task, which you can then reuse in future work. – HopelessN00b Oct 03 '12 at 15:50
  • This sounds like a joke, please provide the link for the tool – user90500 Oct 03 '12 at 19:58
  • I don't have a link for our current IT bitch because it doesn't have a LinkedIn or Facebook account, or whatever. Plus, I don't share my bitches. If you want to find an IT bitch to call your own, advertise the job openign. I think the HR-friendly synonym for "IT bitch" is "IT intern," if that's helpful. – HopelessN00b Oct 03 '12 at 20:08
  • @user90500 It is a joke. What you need is a [PFY](http://en.wikipedia.org/wiki/Bastard_Operator_From_Hell), or maybe an intern. – sysadmin1138 Oct 03 '12 at 20:57

1 Answers1

0

Without knowing just what data you are interested in we can only give very broad and general replies.

Others will no doubt have their own preferred methods but what I do with that much data is to create Perl scripts to read the data directly from the database, using queries designed to read only those parts I am interested in for a particular report. The script would then produce output in a suitable form, most often as spreadsheets or charts.

The hardest part is not filtering the data and producing the reports. The hardest part is determining just what parts you do want to see. With that much data you have to filter it down if you want to make any kind of sense out of it.

John Gardeniers
  • 27,262
  • 12
  • 53
  • 108
  • Thanks for the suggestion John, I am not too familiar in scripting with Perl, what is your suggestion on how to get started with writing Perl scripts that can filter database stats? How do you ultimately plot the info? – user90500 Oct 03 '12 at 15:08
  • It doesn't have to be Perl, that's just what I prefer to use. Nevertheless, it's the best language for such jobs. There are numerous tutorial sites on the Internet. As for plotting, I generally create Excel charts, mainly because Excel is already on all the machines that may be used to view the charts. This is surprisingly easy to do with the "Excel::Writer::XLSX" module. – John Gardeniers Oct 03 '12 at 22:17
  • Thanks again John, can you please guide me to the most helpful tutorial that will help me start learning about making use of Perl scripts with databases – user90500 Oct 09 '12 at 18:39
  • Which tutorial might be best for you will depend a bit on your previous experience with programming and scripting languages, so it's hard to recommend one but if you Google for "perl tutorial" and flick through the results you should be able to find one that is aimed at your level. – John Gardeniers Oct 10 '12 at 04:26