Welcome Guest.   Make a donation to an author on the site December 19, 2014, 05:32:35 PM  *

Please login or register.
Or did you miss your validation email?


Login with username and password (forgot your password?)
Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.


You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.
 
The N.A.N.Y. Challenge 2014! Download dozens of custom programs!
   
   Forum Home   Thread Marks Chat! Downloads Search Login Register  
Pages: [1]   Go Down
  Reply  |  New Topic  |  Print  
Author Topic: navigate webpage results automaticaly and save them  (Read 6035 times)
kalos
Member
**
Posts: 1,112

View Profile Give some DonationCredits to this forum member
« on: August 27, 2008, 10:20:02 PM »

hello

first, I am talking about webpages that have 1,2,3,next etc links (like google results)
I need a 'bot' that will click 'next' in a webpage, go to the next webpage, save it and then click next, go to the next, save it, etc

is there anything like this?

thanks

PS: in any browser, I have no specific preference
Logged
jgpaiva
Global Moderator
*****
Posts: 4,717



Artificial Idiocy

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #1 on: August 28, 2008, 03:19:51 PM »

I think you can use httrack for that. Just pass it the page's address and configure it to download all the pages linked by that one at a depth of '1'.
If the page has other links not related to the search, they will be downloaded too, but I suppose you could delete those manually or something. I think httrack can ignore domains, so if those other pages are all in the same domain (the domain of the original page), you could just ignore that one and you'd get only the interesting pages Wink
Logged

kalos
Member
**
Posts: 1,112

View Profile Give some DonationCredits to this forum member
« Reply #2 on: August 28, 2008, 05:43:23 PM »

the problem is that I need to do this inside the web browser, because the website needs authentication, which is not easy to achieve in webpage offline downloaders (it is not webpages in http://user:pass@domain.com format, but it requires web form authentication)
Logged
4wd
Supporting Member
**
Posts: 3,527



see users location on a map View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #3 on: August 28, 2008, 07:50:46 PM »

the problem is that I need to do this inside the web browser, because the website needs authentication, which is not easy to achieve in webpage offline downloaders (it is not webpages in http://user:pass@domain.com format, but it requires web form authentication)

Sounds like a job for either GreaseMonkey, AutoIt and AutoHK but unless you're willing to provide some details I don't think anyone will be able to help:

eg.

GreaseMonkey - you need to provide access to the site so as to be able to create a userscript to do the actions you want.
AutoIt/AutoHK - you might get away with providing a screenshot of the site so as to give reference to mouse movement/actions and/or key input.

I think these are the most likely automated options barring a dedicated program.

If the website is using a form for verification then it most likely sets a cookie and you could use a website downloader that can use the cookie.

Try FireFox with DownThemAll! - it can supposedly download all links on a page.
« Last Edit: August 28, 2008, 08:02:07 PM by 4wd » Logged

I do not need to control my anger ... people just need to stop pissing me off!
kalos
Member
**
Posts: 1,112

View Profile Give some DonationCredits to this forum member
« Reply #4 on: August 28, 2008, 11:49:55 PM »

unless you're willing to provide some details I don't think anyone will be able to help

let's say I search in google for 'something' and it returns a webpage that it displays the google results, where at the bottom there is 1,2,3,4,next

each of the number of the google results webpages has this url:
http://www.google.com/sea...?q=something&start=10
http://www.google.com/sea...?q=something&start=20
etc

what I want to do is to save the google results webpage (the one with the numbers at the bottom), then click to go to the next google results webpage, save, go to next, save, etc (in other words I need to save all the webpages of the above mentioned urls)

all the above must be done within the web browser, because the website needs me to first authenticate via a web form

Logged
lanux128
Global Moderator
*****
Posts: 6,136



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #5 on: August 29, 2008, 01:29:28 AM »

you can use Repagination to combine all the pages into one and then save. just a thought. smiley


https://addons.mozilla.or.../en-US/firefox/addon/2099
Logged

kalos
Member
**
Posts: 1,112

View Profile Give some DonationCredits to this forum member
« Reply #6 on: August 29, 2008, 05:51:48 AM »

very interesting!

you can do miracles with JAVAscript and greasemonkey, but unfortunately it's hard to code it and there are not many JS developers

I will test it asap, thanks
Logged
sri
Honorary Member
**
Posts: 673


see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #7 on: August 29, 2008, 06:01:20 AM »

Related:

https://addons.mozilla.or.../en-US/firefox/addon/4925
http://antipagination.googlepages.com/index.html
Logged

kalos
Member
**
Posts: 1,112

View Profile Give some DonationCredits to this forum member
« Reply #8 on: August 29, 2008, 07:49:44 AM »

it works, but for the 400+ webpages results that I need to save... it will crash the browser

a web navigation automate script or bot would be the ultimate solution

is there any?
Logged
lanux128
Global Moderator
*****
Posts: 6,136



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #9 on: August 29, 2008, 09:25:29 AM »

it works, but for the 400+ webpages results that I need to save... it will crash the browser

wow, that is a lot of pages. smiley there is one other add-on that i have in my bookmarks but haven't tried it before.


https://addons.mozilla.or...en-US/firefox/addon/3262/
Logged

lanux128
Global Moderator
*****
Posts: 6,136



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #10 on: August 29, 2008, 09:43:51 AM »

i totally forgot about this - iMacros for Firefox. smiley


http://www.iopus.com/imacros/firefox/
Logged

kalos
Member
**
Posts: 1,112

View Profile Give some DonationCredits to this forum member
« Reply #11 on: August 31, 2008, 09:41:42 PM »

unfortunately macros won't work, because when I try to save each webpage of the results, the name of the filename is the same

is there a way to auto-rename them?
Logged
cmpm
Charter Member
***
Posts: 2,025

View Profile Give some DonationCredits to this forum member
« Reply #12 on: August 31, 2008, 10:51:39 PM »

I'd think you would have to use Foxmarks to sync your bookmarks.
Then go to your Foxmarks web site where all your links are and work with them from there.
Of course you need Firefox also which I guess you have.

Would the addon, 'Download Them All', work?

Or you can use a download manager and the addon 'Copy all Links'.

 Copy and paste them into the manager, which ever one is built into Firefox, and there are a few. Which one to choose would depend on it's options that you need.
Logged
mwang
Supporting Member
**
Posts: 204

see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #13 on: September 01, 2008, 01:10:13 AM »

Scrapbook (https://addons.mozilla.org/en-US/firefox/addon/427), maybe?

If the sequential pages have some sort of numbering rule in their URL (most do, I think), then you could copy the starting URL, duplicate it as many times as required in an editor, change the numbering as required for each URL (with 400+ items, I would probably do this step in Excel or something similar), and ask Scrapbook to down them all in a folder.

I did a small test with one of the long thread on this forum:


If you can't or don't want to produce the URLs in advance, you can still do it with Scrapbook, but this time with the help of a Scrapbook Addon called AutoSave (http://amb.vis.ne.jp/mozilla/scrapbook/addons.php?lang=en#AutoSave) and iMacro mentioned above or something similar. I didn't try this approach though.
Logged
kalos
Member
**
Posts: 1,112

View Profile Give some DonationCredits to this forum member
« Reply #14 on: September 01, 2008, 11:43:06 PM »

thanks

these are interesting, but I wonder if it is possible the program to know when the webpage is 100% loaded and afterwards to save it (so that there will be no incomplete webpages saved)
Logged
mwang
Supporting Member
**
Posts: 204

see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #15 on: September 02, 2008, 12:31:22 AM »

If you use the first method I mentioned (giving Scrapbook a list of URLs to save), it saves the web pages in the background, meaning it doesn't load the pages into Firefox. There's a small pop up showing the progress:


It saves one page at a time, with a small delay (a couple of seconds) in between, so it won't overwhelm the server. You may safely ignore the progress dialog (which would take some time if you give it a long list) and continue to use Firefox.

When it's done, the progress dialog goes away and another small message box pup up from the lower-right corner telling you "capture completed".

Logged
kalos
Member
**
Posts: 1,112

View Profile Give some DonationCredits to this forum member
« Reply #16 on: September 08, 2008, 04:23:52 AM »

if the next webpage with results has an url that cannot be shown? eg. if to go there you click a button and the new url is not shown? then I cannot find the list of urls

is there any javascript bot that can auto-browse under specific commands, wait pages to load and then save them?
Logged
mwang
Supporting Member
**
Posts: 204

see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #17 on: September 08, 2008, 07:42:24 PM »

The auto saving part can be taken care by Scrapbook (with AutoSave plugin), as I mentioned above. There are other extensions that do this as well.

As to the auto clicking part, you'll probably need the help of iMicro (also mentioned above) or something like that. I've never tried it though, so can't help you there.
Logged
Paul Keith
Member
**
Posts: 1,982


see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #18 on: September 09, 2008, 12:27:07 AM »

Just out of curiosity, why do you need 400+ Google results pages?
Logged

<reserve space for the day DC can auto-generate your signature from your personal PopUp Wisdom quotes>
kalos
Member
**
Posts: 1,112

View Profile Give some DonationCredits to this forum member
« Reply #19 on: September 09, 2008, 05:24:08 AM »

Just out of curiosity, why do you need 400+ Google results pages?

it's not about google results, google was just for example
Logged
Paul Keith
Member
**
Posts: 1,982


see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #20 on: September 09, 2008, 09:09:36 AM »

Oh ok. Thanks for clarifying that.
Logged

<reserve space for the day DC can auto-generate your signature from your personal PopUp Wisdom quotes>
Pages: [1]   Go Up
  Reply  |  New Topic  |  Print  
 
Jump to:  
   Forum Home   Thread Marks Chat! Downloads Search Login Register  

DonationCoder.com | About Us
DonationCoder.com Forum | Powered by SMF
[ Page time: 0.048s | Server load: 0.08 ]