What is more transparent, curl or wget in this web crawler project?

I have files on a Windows 7, xampp server (using altered port, 181). I am triggering the script from another computer (linux) using curl http://xxx.xxx.xxx:181/phpproject.php

This PHP project is a web crawler using PHP cURL, that shuffles a list of proxies (http). I also have a script in the linux system that can fire wget -O - http://xxx.xxx.xxx:181/phpproject.php as another alternative. Is one more secure and are there ways for me to test for visibility.

The linux system happens to be BackTrack, so I have a good toolbox if any of that software could be of use for this project. I do have Task Scheduler dialed in on the Windows system, but wanted to try out remotely, leading to questions on what is more secure.

Shane

Posted 2013-06-21T00:04:24.640

Reputation: 135

1Which part of this chain are you asking about the security of? What do you mean by secure - secure from who/what? What threats are you trying to protect against? – Duncan Lock – 2013-06-21T09:55:47.047

@DuncanLock I am looking at never leaking my IP or cookies that can identify my primary IP to the scraped source. if one of the proxies allowed view of anything, I would like one system to be the net for this leak. That way I can set-up local IP's in accordance to not accidentally disrupt related services to the scraped source. Secondly, I just want to understand as a visual person, what I am looking for and how to "see" what is going on. – Shane – 2013-06-21T22:17:08.020

Answers

CURL & WGET are the same, from a transparency point of view - they're both plain text and anyone sniffing packets on the network can see what they're doing. Running them over SSH will create an encrypted tunnel and fix this.

To see what's going on inside your network visually, you need a copy of WireShark - a free, open source network protocol analyzer and packet inspector. If you run this on either end, it will show you everything that that machine is sending/receiving over the network. It's just a desktop application and runs on both Windows and Linux. If you run it as Admin and have a capable network card, it can be run in 'Promiscuous Mode' which allows you to see everything happening on a local network from one machine.

If you're touching the public internet with this project - which it sounds like you are - then you will need something that creates encrypted tunnels (like SSH does for servers you control/login to) - but for the public internet in general. You also want something that will mask/change your public IP address. This thing is called a VPN - there are numerous VPN providers that you can sign up for, such as OverPlay, who will provide you with VPN service for a fee.

Also, remember: always use your powers for good, not evil ;)

Duncan Lock

Posted 2013-06-21T00:04:24.640

Reputation: 252

1Great answer thank you. No evil here, but I do like to bypass quotas, max limitations and such. – Shane – 2013-06-22T01:20:45.347