Home | Blog | Software | Reviews and Features | Forum | Help | Donate | About us
topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • December 08, 2016, 08:24:21 AM
  • Proudly celebrating 10 years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: download url  (Read 2917 times)

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,473
    • View Profile
    • Donate to Member
download url
« on: March 20, 2008, 08:28:16 AM »
hello

I need a program that will download urls from a list in a text file

it should just save the html (saving the complete webpage, ie. the html AND a folder with the files, should be optional)

also, it should use any cookies necessary from the web browser (IE) in order to authenticate

last, it should download every few seconds between each url, or with random time interval

do you know any program?
I already know wget, but I have problems with cookie authentication

thanks

GeekK

  • Participant
  • Joined in 2006
  • *
  • default avatar
  • Posts: 3
    • View Profile
    • Donate to Member
Re: download url
« Reply #1 on: March 20, 2008, 10:46:32 PM »
HTTrack Website Copier

- cope/paste text file in window:

- cookies: http://www.httrack.com/html/faq.html#Q1c (copy the cookies located into the Temporary Internet Files folder for IE into your project folder (or even the HTTrack folder))

- it can pause after chosen number of Bytes: http://www.httrack.c...html/step9_opt2.html

- faq's and manuals: http://www.httrack.com/html/index.html; has lot more options like flow-control



« Last Edit: March 20, 2008, 10:49:00 PM by GeekK »

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,473
    • View Profile
    • Donate to Member
Re: download url
« Reply #2 on: March 21, 2008, 05:12:37 AM »
thanks, but I really need an alternative of Httrack, I have hard time with this program and it is not as much customizable

I wuold use wget, but I cannot make it to automaticaly choose and use the appropriate cookie

blackcat

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 62
    • View Profile
    • Donate to Member
Re: download url
« Reply #3 on: March 21, 2008, 07:16:19 AM »
what about cURL? Have you tried that one?

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,473
    • View Profile
    • Donate to Member
Re: download url
« Reply #4 on: March 28, 2008, 10:13:16 PM »
I am trying to make curl work for this, with no success atm

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,473
    • View Profile
    • Donate to Member
Re: download url
« Reply #5 on: March 29, 2008, 09:54:25 PM »
is there a download manager that can:

1. download html (optionaly recursively)
2. download every x seconds (random interval would be best)

thanks