topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Tuesday March 19, 2024, 6:12 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: how to extract links from a site?  (Read 12821 times)

steve_rb

  • Participant
  • Joined in 2006
  • *
  • default avatar
  • Posts: 14
    • View Profile
    • Donate to Member
how to extract links from a site?
« on: April 29, 2007, 02:16 AM »
I am looking for a program to extract hiden links inside a site starting with let say for example "http://www.rapidshare.de/" . Is there any program to do this or anyone has any idea or comment?

Regards
Steve

zridling

  • Friend of the Site
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 3,299
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #1 on: April 29, 2007, 02:23 AM »
Check out fellow DC member Veign's donationware XSite software. I've used it to collect links from complex blog posts that I wanted to follow up on later.

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: how to extract links from a site?
« Reply #2 on: April 29, 2007, 05:35 AM »
do you just want to extract links on a single page, or on an entire site?
and you just want the links, not the entire site content?

there are some good tools for grabbing the entire content from a site (Teleport Pro, Offline Explorer).

and there are some good download managers (and probably firefox extensions) which can list all the links on a page from within your browser.

justice

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 1,898
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #3 on: April 29, 2007, 06:15 AM »
Also, Opera has a Link panel, which gives you all the links on a page, you can then search through them, but not save them all out it seems.

If you'd use a site link checker like Xeno then you'd get all the links in a site including which ones do no longer work.

steve_rb

  • Participant
  • Joined in 2006
  • *
  • default avatar
  • Posts: 14
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #4 on: April 29, 2007, 07:26 AM »
@ mouser
I want to extract links inside entire site not just one page. I also want hiden links (links you can not see directly but you can access via search field inside that site for example) to be extracted too.

@ zridling
Veign's site is down. I can't access it. Is you have Xsite program please send me via email ([email protected]).

Regards

steve_rb

  • Participant
  • Joined in 2006
  • *
  • default avatar
  • Posts: 14
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #5 on: April 29, 2007, 07:29 AM »
@ justice
Do you have any download link for Xenon?

 :D

justice

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 1,898
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #6 on: April 29, 2007, 02:16 PM »

zridling

  • Friend of the Site
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 3,299
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #7 on: April 30, 2007, 03:56 AM »
Sent XSite your way.

steve_rb

  • Participant
  • Joined in 2006
  • *
  • default avatar
  • Posts: 14
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #8 on: April 30, 2007, 07:23 AM »
@ zridling

thanks for the program but It only can extract links on the page I gave it to the program. It does not follow links. I want a program to follow all links and search all pages and extact all links in that pages. Simply all links in the entire site.

 ;)

steve_rb

  • Participant
  • Joined in 2006
  • *
  • default avatar
  • Posts: 14
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #9 on: May 01, 2007, 11:59 PM »

Xenu is great and could almost extract all links from a site but still not all hidden or redirected urls. Thanks justice for letting me know about this software. It solves my problem to some extend but I still need more power full software specially to show redirected links and follow all links untill to get to the bottom of them.

 :Thmbsup:

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: how to extract links from a site?
« Reply #10 on: May 02, 2007, 07:06 AM »
If you feel like playing around, you can try this web robot in VB.NET or C#.

You'd be surprised just how many really cool things are available for free at the CodeProject. :D

(OK - They take some work, but the basics are there.)
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

zridling

  • Friend of the Site
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 3,299
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #11 on: May 02, 2007, 08:47 AM »
Steve, as mouser mentioned, you're looking for something more powerful then. The most accurate I've found is Teleport Pro.

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: how to extract links from a site?
« Reply #12 on: May 02, 2007, 10:42 AM »
I've found Teleport Pro to be very valuable. You'll have a hard time finding something that can automatically surf all your porn for you! :D

Edit: Yep - I know what you're using it for Zaine! :D
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Darwin

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,984
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #13 on: May 02, 2007, 01:37 PM »
Justice  :-[ steve_rb - I'd edit out your e-mail address above, post-haste, if I were you ( I *HATE* spam, get too much of it as it is, and don't go looking for any more of it!  ;D)...
« Last Edit: May 02, 2007, 03:05 PM by Darwin »

ender

  • Participant
  • Joined in 2006
  • *
  • default avatar
  • Posts: 14
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #14 on: May 09, 2007, 02:39 PM »
I am looking for a program to extract hiden links inside a site starting with let say for example "http://www.rapidshare.de/" . Is there any program to do this or anyone has any idea or comment?

Regards
Steve

I am using either of these two programs, if I want to batch download files from rapidshare & other temp. hoster:
http://www.rapget.com/en/index.html
http://www.portablefreeware.com/?id=1111

http://translate.goo...dimonius.ru/dusd.php
http://www.portablefreeware.com/?id=1251

both still require you to manually put in the links, but after that, I can leave my computer on overnight, and the next day, all files are on my harddrive.


Tinman57

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 1,702
    • View Profile
    • Donate to Member
Re: how to extract links from a site?
« Reply #15 on: May 09, 2007, 09:45 PM »
  Something like ScanDL that Searches HTML for links and constructs a file list?

http://scandl.com/en/news.html