how to save all *.url links inside of an entire site?

[steve_rb]

I was trying for a long time to solve thois problem but until now I couldn't find any solution for this problem:

I want to find a way or software to go through all pages of entire website and find all links with *.rul format for example and save these links inside a text file for me. URL Snooper can do this but I have to go through all pages manually one by one and have to open all pages at least once. This is not possible for a site over thousands of pages. Anyidea how to do this?

 

[jgpaiva]

Using httrack, you can download the whole site at once, removing non-important parts (like images and such), keeping only the html pages.
Then, there are programs that using a regular expression or such, can pick all the links from those html files. One example of such program would be powergrep.

[steve_rb]

httrack is very complicated and seems download whole pages. The site I am dealing with is very big and probably thousands of pages and it is not possible to download all pages. I just want the software to surf all pages and copy certain links into a text file. I am sure there should be such a software around. I have to find it. If anyone has experience in this please comment.

 

[jgpaiva]

But wasn't your idea not to have to manually surf through each page?

httrack would allow you to download only the html files, which would not take much space (this page for exampe, takes 70-80kb).
Then, powergrep would retrive the links all at once from all the pages. All you'd have to do would be to give it a regular expression (or a normal boolean expression) to match on the html files.

[steve_rb]

I tried two days but couldn't get httrack to save pages I wanted. It saves every other pages but not the pages I want. I know this is too much to ask but If you have time please give this a try and see if you can make this work. I am after URLs in the following format for example from this site:

"http://gigapedia.org...3f16fbd84d2dfaa3.url"

Only digit part is changing for other links. I want these links to be written in a text file.

Site is "h**p://gigapedia.org"
when you enter this site use following ID to enter or you can register with your own ID and password:

ID: steve_rb
pass: ******

Then click on browse, and then Articles & Whitepapers for example, and then click on the first item on page 1, then click on links tab and you will see a few download links on this page and if you put mouse pointer on those links you will see a link in the above format at the bottom of your browser or you can right click on it and use "copy url" to copy it and pase into a text file. I want these links to be extracted inside a text file. If you didn't see any link please go back and choose second item and open links page.

 

[jgpaiva]

Ok, i now understand your problem.
Those pages need a login to be accessed, and apparently httrack doesn't support that. (i actually am surprised, i thought it did)
There should be similar programs out there that do support autentication.

Anyone?

[steve_rb]

No. I tried httrack and it works ok. After loging, ID and password goes to cookies and I think thats why httrack works ok. It saved a lot of pages and they are all ok. It just doesn't save links pages. It even save descriptions pages OK. This shows httrack enters inside the site and start saving html pages but I think maybe proper choice of parameters and options could make it to save links pages too. I tried for two days but couldn't do it. Some experts in httrack should help here.

 

[jgpaiva]

I see steve. It must be using the cokies of internet explorer.
It probably doesn't save those pages because they require you to click on that <link> tab, which appears to be javascipt.

I'm sorry but i see no simple way to retrieve those links