record the websites/remote servers visited from my computer

Issue Summary and Description

I would like to record all outgoing requests from my computer (in this case a mac) to any websites or remote servers.

I mainly program magento applications and with this you can create what's called an observer which can be automatically called anytime a particular class->function() is called eg. if a product is added to the shopping cart you could write an observer that adds the customers name to a database.

I am trying to find a way of doing something like this on my computer so that everytime the computer sends a request to the router my application can listen for this request and add it to a csv file.

I know there are some browser extensions that do something like this but I would like the browser to be irrelevant and work with the final point of the request from the client side.

My question is based on the following points:

Maybe I am over simplifying this task but it sounds like it could be something that is possible to do with a command line entry, is this correct or am I way off?
If not my preferred languages are php and javascript but I am happy to try out whatever would be most appropriate for the job. Do I have to use the language that this part of the OS software is programmed in and if so what is that language and how do I incorporate my code with this part of the OS?
Once I have figured out how to do this am I likely to have ongoing permissions issues?
What, regarding these points, would I need to consider differently if approaching this application to work on a windows machine?

develophper

Posted 2014-10-05T23:14:50.963

Reputation: 73

Answers

On both Windows and OSX there is a command line utility known as NETSTAT. The output of this is literally all open network connections to network locations.

The output looks like this:

TCP    10.99.1.155:51551      lhr14s23-in-f2:https   ESTABLISHED
TCP    10.99.1.155:51555      stackoverflow:http     ESTABLISHED
TCP    10.99.1.155:51556      stackoverflow:http     ESTABLISHED
TCP    10.99.1.155:51557      stackoverflow:http     ESTABLISHED

If you are comfortable with python, capturing stdOut and parsing the Foreign Address should be straight forward.

You can add command line parameters to include process ID, process name and to make it run continuously etc etc.

Patrick

Posted 2014-10-05T23:14:50.963

Reputation: 1 132

I assume due to this being a command line utility it doesn't need to be done in Python or do think python has other benefits over php for example? Also, do you know if this would use up much resources if ran constantly or is it possible to just get reports every few hours or minutes? – develophper – 2014-10-17T09:24:00.597

You assume correct. No need for python, it just seemed to me that'd be the easiest way to handle it. Personally I'm a windows man so would have done it in powershell. What ever suits your needs.

Netstat won't actually run continuously but you can set the repeat interval to 1 second so it can do an equivalent of. Its realtime though, so if you did it every hour you'd miss stuff that connected and closed again. The performance impact would be negligible though, its just listing existing connections. – Patrick – 2014-10-17T09:25:53.670

ok mate thanks a lot. I think this is enough to get me on track. I reckon I can work out the rest. – develophper – 2014-10-17T09:27:20.163

I believe Magneto is some sort of online CMS? If so, have you tried using your browser's developer tools network tab? Firefox, Chrome & Internet Explorer all have one.

If you're looking instead to find everything going out from your machine, you want Wireshark. Wheras the other 3 need no special permissions, Wireshark will require some sort of Admin privileges to snoop on your network interface.

All of these will let you filter down what you see to limit the information overload. Especially in Wireshark's case, you're probably not interested in DNS queries etc!

Pricey

Posted 2014-10-05T23:14:50.963

Reputation: 4 262

Magento is an ecommerce software but it's not really relevant to what I want. I just used the observer as an example of how i would like to program it. I know there are other softwares I could get to do this but I want to know how to actually develop it. – develophper – 2014-10-10T14:42:33.003

Aha so you don't access it through the browser? In that case, Wireshark's the one to go for! – Pricey – 2014-10-10T14:43:22.290

Yes but I'm trying to program it. Hence my asking about which programming languages to use and how to incorporate my code with the code of the OS. – develophper – 2014-10-10T14:45:50.853

Maybe you could explain a little more what programming needs you have for this task. Are you looking for any solution that works, or do you want to program it yourself for educational purposes ? – lemonsqueeze – 2014-10-11T08:43:03.813