topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Thursday March 28, 2024, 5:43 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: grab urls  (Read 30946 times)

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
grab urls
« on: November 29, 2016, 05:33 PM »
Hello!

I have a list of URLs that I want to download from a website that has a form login.

How can I do that automatically in batch?

Is there a downloader and can the downloader use the cookie from my Firefox to authenticate?

thanks!

Target

  • Honorary Member
  • Joined in 2006
  • **
  • Posts: 1,832
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #1 on: November 29, 2016, 07:01 PM »
probably doesnt handle the login, but downthemall?

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #2 on: November 29, 2016, 07:34 PM »
probably doesnt handle the login, but downthemall?

Can I load a text file with the urls to that and it will use the cookie to authenticate and download?

Target

  • Honorary Member
  • Joined in 2006
  • **
  • Posts: 1,832
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #3 on: November 29, 2016, 07:45 PM »
probably doesnt handle the login, but downthemall?

Can I load a text file with the urls to that and it will use the cookie to authenticate and download?

have you looked at the tool at all?  did you actually read what I wrote?

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #4 on: November 29, 2016, 07:47 PM »
Well I saw the screen shots! I couldn't figure out

Target

  • Honorary Member
  • Joined in 2006
  • **
  • Posts: 1,832
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #5 on: November 29, 2016, 08:26 PM »
DownThemAll is all you can desire from a download manager: it features an advanced accelerator that increases speed up to 4x and it allows you to pause and resume downloads at any time.

DownThemAll is fast, reliable and easy-to-use! It lets you download all the links or images contained in a webpage and much more: you can refine your downloads by fully customizable criteria to get only what you really want! Be in full control over your downloads, dedicated speed and number of parallel connections at any time. Use Metalinks or add mirrors manually to download a file from different servers at the same time.

DownThemAll is open-source and freeware. No Adware, no Spyware, no hidden costs!

feature list...

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,641
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #6 on: November 29, 2016, 09:13 PM »
I have a list of URLs that I want to download from a website that has a form login.

How can I do that automatically in batch?

Is there a downloader and can the downloader use the cookie from my Firefox to authenticate?

wget

Login with wget and keep the cookie

Parameter to download files from a list:  -i list.txt

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Points: 1
  • Posts: 9,747
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: grab urls
« Reply #7 on: November 30, 2016, 02:27 AM »
This also sounds like something that could possibly be handled by JDownloader.

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,540
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: grab urls - try GetRight?
« Reply #8 on: November 30, 2016, 07:27 AM »
You could do worse than try GetRight - http://getright.com/ (from Headlight Software - http://headlightinc.com/).
I think it might have been a forerunner or led the way for meeting all/most downloading requirements, including batch downloading from a text list, passing logon credentials, etc. It works with major browsers too.
GetRight also includes the incredibly useful GetRight Browser, which enables the user to browse/view the directories of accessible download sites.

Some people (not me, you understand) might say that, with a bit of experimentation, it is surprising how much stuff one can find to download with GetRight that might not be apparently available to the casual/inexpert enquirer, by back-dooring weak/nonexistent access control and apparently locked directories ...    :o   - however, I couldn't possibly comment.
« Last Edit: November 30, 2016, 12:14 PM by IainB, Reason: Typo correction. »

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #9 on: December 02, 2016, 05:48 PM »
I gave Downthemall a try, but is there a way to make the downloads delay every x seconds so that I won't abuse the server???

Or any other solution, as easy as Downthemall that integrates seamlessly with my Firefox? (I can use other browsers/software)

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #10 on: December 09, 2016, 02:04 PM »
anyone please?

I need a download manager that will use my Firefox cookie / form authentication and that will download a list of urls I will feed to it delaying between each!

wget may work but I need to read the manual to make it work and I don't have the capacity to do tests!

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Points: 1
  • Posts: 9,747
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: grab urls
« Reply #11 on: December 09, 2016, 02:51 PM »
Did you try JDownloader or GetRight?

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #12 on: December 09, 2016, 03:05 PM »
I read online for JDownloader that it doesnt support delay between downloads or it needs a script or something.

I went to download GetRight but is it payware/spyware?

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #13 on: December 23, 2016, 03:43 PM »
Guys! I am really stuck!

JDownloader has no option do delay between downloads and GetRight is trialware with who knows limitations!

Isnt there any real free solution?

I need to load a text file of urls, grab the cookie from Firefox or other browser, and download the links with some delay one after the other!

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: grab urls
« Reply #14 on: December 27, 2016, 10:16 AM »
I gave Downthemall a try, but is there a way to make the downloads delay every x seconds so that I won't abuse the server???
You can configure concurrent downloads and downloads-per-server in DownThemAll - that really should be all you need to to avoid "abusing" anything :)
- carpe noctem

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
Batch downloading
« Reply #15 on: July 09, 2017, 08:06 AM »
Hello!

I want to download a list of urls in the following format:

http://domain.com/mo...e/view.php?id=500691
(with id changing)

How can I download the urls and have the download program to use cookies from my Firefox or maybe enter username/password in the login form when necessary?

Firefox stores cookies in an sqlite file, which makes it tricky, but I can use other browser instead?

Thanks!

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,641
    • View Profile
    • Donate to Member
Re: Batch downloading
« Reply #16 on: July 09, 2017, 08:56 AM »

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
Re: Batch downloading
« Reply #17 on: July 09, 2017, 09:14 AM »
thanks, but that stackoverflow thread is a mess
it is not clear if I need CURL to make wget work with POST authentication

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 11,186
    • View Profile
    • Donate to Member
Re: Batch downloading
« Reply #18 on: July 09, 2017, 10:08 AM »
thanks, but that stackoverflow thread is a mess
it is not clear if I need CURL to make wget work with POST authentication

curl and wget are two ways of doing the same thing.  Your answers are in the links 4wd gave on that other DC thread.

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,641
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #19 on: July 09, 2017, 03:35 PM »
Did threads just get merged or have I been into the cider too much?

thanks, but that stackoverflow thread is a mess

How exactly is the first answer in the thread a "mess"?

Based on the manual page:

# Log in to the server.  This can be done only once.                   
wget --save-cookies cookies.txt \
     --keep-session-cookies \
     --post-data 'user=foo&password=bar' \
     --delete-after \
     http://server.com/auth.php

# Now grab the page or pages we care about.
wget --load-cookies cookies.txt \
     http://server.com/in...eresting/article.php

Substitute -i list.txt for the URL in the second command, list.txt contains a list of URLs to download.

Plus further down the answers:
I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.

When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.
« Last Edit: July 10, 2017, 03:55 AM by 4wd »

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Points: 1
  • Posts: 9,747
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: grab urls
« Reply #20 on: July 09, 2017, 03:41 PM »
Did threads just get merged or have I been into the cider too much?

They got merged. You can tell by some of the pixels :P, also because some of the posts have different topics than others.

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,641
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #21 on: July 09, 2017, 03:51 PM »
Did threads just get merged or have I been into the cider too much?

They got merged. You can tell by some of the pixels :P, also because some of the posts have different topics than others.

Thank heavens, I've only had one cider today, a very nice Thatchers Vintage Cider  :D

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #22 on: April 08, 2018, 04:13 PM »
Did threads just get merged or have I been into the cider too much?

thanks, but that stackoverflow thread is a mess

How exactly is the first answer in the thread a "mess"?

Based on the manual page:

# Log in to the server.  This can be done only once.                   
wget --save-cookies cookies.txt \
     --keep-session-cookies \
     --post-data 'user=foo&password=bar' \
     --delete-after \
     http://server.com/auth.php

# Now grab the page or pages we care about.
wget --load-cookies cookies.txt \
     http://server.com/in...eresting/article.php

Substitute -i list.txt for the URL in the second command, list.txt contains a list of URLs to download.

Plus further down the answers:
I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.

When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.


Thanks but, the problem with that is that some cookies/form-logins expire. So when you go on downloading a file it displays the form to login instead.
That's why I am looking for a software that can detect that and use my login each time it is need.
Is there a way to do this?

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,641
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #23 on: April 08, 2018, 07:55 PM »
Thanks but, the problem with that is that some cookies/form-logins expire. So when you go on downloading a file it displays the form to login instead.
That's why I am looking for a software that can detect that and use my login each time it is need.
Is there a way to do this?

So you're saying that the cookies expire in the couple of seconds between sending the first command followed by the second command?

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,823
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #24 on: April 14, 2018, 03:22 PM »
I don't feel comfortable using WGET. I need a GUI so that I can see the progress etc.

Is there a GUI download manager that will allow me to download a list of urls, from a login website, delaying 5 seconds between each attempt? Ideally, it could scan the links of the url and download any files of specific extensions in those urls?