How can I automate sending a form and saving the result web page?

Question

I have a site where on the page is text input field and submit button.

I need to put numbers 1 to 100 in that text field and save the resulting page after the submit button has been clicked. So I need to have 100 pages downloaded.

In these downloaded pages I will look for the information I need.

How can I do that?

Answer 1

If the website is submitting the form's data using GET, when you click on submit, you should see the submitted data on your browser's address bar. Something like:

http://www.somesite.com/someform?numbers=1

If that is the case, you can run the command in a for loop:

1. Install wget for windows.

2. Modify (change the paths according to your needs) and run this command:

   FOR /L %i IN (1,1,100) DO ("C:\Program Files\GnuWin32\bin\wget.exe" --output-document "C:\Users\SomeUserName\SomeDir\%i.html" http://www.somesite.com/someform?numbers=%i)
   

EXPLANATION:

The for loop has this format: (start,step,end). So, the loop above will go through all the numbers from 1 to 100 in increments of 1. At each iteration, it will set the value of variable %i to the current number.

Wget will download a webpage from a given URL. Using the address from the previous step, it will replace the number in numbers=1 with each of the numbers from 1 to 100, download the corresponding webpage and save it as "number".html in the directory C:\Users\SomeUserName\SomeDir\%i.html.

If you need more help, please post the URL of the website form you will be using.