Main Area and Open Discussion > General Software Discussion

navigate webpage results automaticaly and save them

(1/5) > >>

kalos:
hello

first, I am talking about webpages that have 1,2,3,next etc links (like google results)
I need a 'bot' that will click 'next' in a webpage, go to the next webpage, save it and then click next, go to the next, save it, etc

is there anything like this?

thanks

PS: in any browser, I have no specific preference

jgpaiva:
I think you can use httrack for that. Just pass it the page's address and configure it to download all the pages linked by that one at a depth of '1'.
If the page has other links not related to the search, they will be downloaded too, but I suppose you could delete those manually or something. I think httrack can ignore domains, so if those other pages are all in the same domain (the domain of the original page), you could just ignore that one and you'd get only the interesting pages ;)

kalos:
the problem is that I need to do this inside the web browser, because the website needs authentication, which is not easy to achieve in webpage offline downloaders (it is not webpages in http://user:[email protected] format, but it requires web form authentication)

4wd:
the problem is that I need to do this inside the web browser, because the website needs authentication, which is not easy to achieve in webpage offline downloaders (it is not webpages in http://user:[email protected] format, but it requires web form authentication)
-kalos (August 28, 2008, 05:43 PM)
--- End quote ---

Sounds like a job for either GreaseMonkey, AutoIt and AutoHK but unless you're willing to provide some details I don't think anyone will be able to help:

eg.

GreaseMonkey - you need to provide access to the site so as to be able to create a userscript to do the actions you want.
AutoIt/AutoHK - you might get away with providing a screenshot of the site so as to give reference to mouse movement/actions and/or key input.

I think these are the most likely automated options barring a dedicated program.

If the website is using a form for verification then it most likely sets a cookie and you could use a website downloader that can use the cookie.

Try FireFox with DownThemAll! - it can supposedly download all links on a page.

kalos:
unless you're willing to provide some details I don't think anyone will be able to help
-4wd (August 28, 2008, 07:50 PM)
--- End quote ---

let's say I search in google for 'something' and it returns a webpage that it displays the google results, where at the bottom there is 1,2,3,4,next

each of the number of the google results webpages has this url:
http://www.google.com/search?q=something&start=10
http://www.google.com/search?q=something&start=20
etc

what I want to do is to save the google results webpage (the one with the numbers at the bottom), then click to go to the next google results webpage, save, go to next, save, etc (in other words I need to save all the webpages of the above mentioned urls)

all the above must be done within the web browser, because the website needs me to first authenticate via a web form

Navigation

[0] Message Index

[#] Next page