topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • April 20, 2018, 09:28 PM
  • Proudly celebrating 13 years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: grab urls  (Read 4256 times)

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 9,320
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #25 on: April 14, 2018, 03:39 PM »
I don't feel comfortable using WGET. I need a GUI so that I can see the progress etc.

Is there a GUI download manager that will allow me to download a list of urls, from a login website, delaying 5 seconds between each attempt? Ideally, it could scan the links of the url and download any files of specific extensions in those urls?



Downthemall might do it.  I haven't used it in forever.  It probably wouldn't do it as is- what you'd have to do is make a simple site with the links, and then open it in firefox.  All of this is, however, speculative, as I haven't tried it.

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,797
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #26 on: April 15, 2018, 12:08 AM »
I don't feel comfortable using WGET. I need a GUI so that I can see the progress etc.

Is there a GUI download manager that will allow me to download a list of urls, from a login website, delaying 5 seconds between each attempt? Ideally, it could scan the links of the url and download any files of specific extensions in those urls?

Downthemall might do it.

It was suggested, he doesn't like it or it's too complicated.

JDownloader as suggested by Deozaan could most likely handle it, (logging in and downloading), but if he baulks at DownThemAll he's going to have no hope with JDownloader.

I don't feel comfortable using WGET. I need a GUI so that I can see the progress etc.

It outputs it's progress to the CLI window.

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,584
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #27 on: April 19, 2018, 03:56 PM »
It's not that I don't like DownThemAll.
First, it doesn't work with latest Firefox.
Second, it doesn't have the option to download urls with delay between them, it will crash the server and get me banned!

tomos

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 11,146
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #28 on: April 19, 2018, 11:27 PM »
^ already addressed AFAICS.
And you say you can use a different browser - try PaleMoon or Firefox ESR
I gave Downthemall a try, but is there a way to make the downloads delay every x seconds so that I won't abuse the server???
You can configure concurrent downloads and downloads-per-server in DownThemAll - that really should be all you need to to avoid "abusing" anything :)
Tom

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,797
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #29 on: Today at 04:42 AM »
^ already addressed AFAICS.
And you say you can use a different browser - try PaleMoon or Firefox ESR
I gave Downthemall a try, but is there a way to make the downloads delay every x seconds so that I won't abuse the server???
You can configure concurrent downloads and downloads-per-server in DownThemAll - that really should be all you need to to avoid "abusing" anything :)


dTA! can't insert a delay between downloads, the next is fired off as soon as the previous finishes.

wget can insert a delay, (including a random delay).

Code: Text [Select]
  1. @echo off
  2. wget --save-cookies cookies.txt --keep-session-cookies --post-data 'user=foo&password=bar' --delete-after http://server.com/auth.php
  3. wget --load-cookies cookies.txt --secure-protocol="auto" --no-http-keep-alive -w 60 --random-wait --input-file="list.txt"