Author Topic: grab urls (Read 34459 times)

kalos · « **on:** November 29, 2016, 05:33 PM »

Hello!

I have a list of URLs that I want to download from a website that has a form login.

How can I do that automatically in batch?

Is there a downloader and can the downloader use the cookie from my Firefox to authenticate?

thanks!

Target · « **Reply #1 on:** November 29, 2016, 07:01 PM »

probably doesnt handle the login, but downthemall?

kalos · « **Reply #2 on:** November 29, 2016, 07:34 PM »

probably doesnt handle the login, but downthemall?
-Target (November 29, 2016, 07:01 PM)

Can I load a text file with the urls to that and it will use the cookie to authenticate and download?

Target · « **Reply #3 on:** November 29, 2016, 07:45 PM »

probably doesnt handle the login, but downthemall?
-Target (November 29, 2016, 07:01 PM)

Can I load a text file with the urls to that and it will use the cookie to authenticate and download?
-kalos (November 29, 2016, 07:34 PM)

have you looked at the tool at all? did you actually read what I wrote?

kalos · « **Reply #4 on:** November 29, 2016, 07:47 PM »

Well I saw the screen shots! I couldn't figure out

Target · « **Reply #5 on:** November 29, 2016, 08:26 PM »

[Select]

DownThemAll is all you can desire from a download manager: it features an advanced accelerator that increases speed up to 4x and it allows you to pause and resume downloads at any time.

DownThemAll is fast, reliable and easy-to-use! It lets you download all the links or images contained in a webpage and much more: you can refine your downloads by fully customizable criteria to get only what you really want! Be in full control over your downloads, dedicated speed and number of parallel connections at any time. Use Metalinks or add mirrors manually to download a file from different servers at the same time.

DownThemAll is open-source and freeware. No Adware, no Spyware, no hidden costs!

feature list...

4wd · « **Reply #6 on:** November 29, 2016, 09:13 PM »

I have a list of URLs that I want to download from a website that has a form login.

How can I do that automatically in batch?

Is there a downloader and can the downloader use the cookie from my Firefox to authenticate?
-kalos (November 29, 2016, 05:33 PM)

wget

Login with wget and keep the cookie

Parameter to download files from a list: -i list.txt

Deozaan · « **Reply #7 on:** November 30, 2016, 02:27 AM »

This also sounds like something that could possibly be handled by JDownloader.

IainB · « **Reply #8 on:** November 30, 2016, 07:27 AM »

You could do worse than try GetRight - http://getright.com/ (from Headlight Software - http://headlightinc.com/).
I think it might have been a forerunner or led the way for meeting all/most downloading requirements, including batch downloading from a text list, passing logon credentials, etc. It works with major browsers too.
GetRight also includes the incredibly useful GetRight Browser, which enables the user to browse/view the directories of accessible download sites.

Some people (not me, you understand) might say that, with a bit of experimentation, it is surprising how much stuff one can find to download with GetRight that might not be apparently available to the casual/inexpert enquirer, by back-dooring weak/nonexistent access control and apparently locked directories ...

- however, I couldn't possibly comment.

kalos · « **Reply #9 on:** December 02, 2016, 05:48 PM »

I gave Downthemall a try, but is there a way to make the downloads delay every x seconds so that I won't abuse the server???

Or any other solution, as easy as Downthemall that integrates seamlessly with my Firefox? (I can use other browsers/software)

kalos · « **Reply #10 on:** December 09, 2016, 02:04 PM »

anyone please?

I need a download manager that will use my Firefox cookie / form authentication and that will download a list of urls I will feed to it delaying between each!

wget may work but I need to read the manual to make it work and I don't have the capacity to do tests!

Deozaan · « **Reply #11 on:** December 09, 2016, 02:51 PM »

Did you try JDownloader or GetRight?

kalos · « **Reply #12 on:** December 09, 2016, 03:05 PM »

I read online for JDownloader that it doesnt support delay between downloads or it needs a script or something.

I went to download GetRight but is it payware/spyware?

kalos · « **Reply #13 on:** December 23, 2016, 03:43 PM »

Guys! I am really stuck!

JDownloader has no option do delay between downloads and GetRight is trialware with who knows limitations!

Isnt there any real free solution?

I need to load a text file of urls, grab the cookie from Firefox or other browser, and download the links with some delay one after the other!

f0dder · « **Reply #14 on:** December 27, 2016, 10:16 AM »

I gave Downthemall a try, but is there a way to make the downloads delay every x seconds so that I won't abuse the server???
-kalos (December 02, 2016, 05:48 PM)

You can configure concurrent downloads and downloads-per-server in DownThemAll - that really should be all you need to to avoid "abusing" anything

kalos · « **Reply #15 on:** July 09, 2017, 08:06 AM »

Hello!

I want to download a list of urls in the following format:

http://domain.com/mo...e/view.php?id=500691
(with id changing)

How can I download the urls and have the download program to use cookies from my Firefox or maybe enter username/password in the login form when necessary?

Firefox stores cookies in an sqlite file, which makes it tricky, but I can use other browser instead?

Thanks!

4wd · « **Reply #16 on:** July 09, 2017, 08:56 AM »

https://www.donation....msg404268#msg404268

kalos · « **Reply #17 on:** July 09, 2017, 09:14 AM »

thanks, but that stackoverflow thread is a mess
it is not clear if I need CURL to make wget work with POST authentication

wraith808 · « **Reply #18 on:** July 09, 2017, 10:08 AM »

thanks, but that stackoverflow thread is a mess
it is not clear if I need CURL to make wget work with POST authentication
-kalos (July 09, 2017, 09:14 AM)

curl and wget are two ways of doing the same thing. Your answers are in the links 4wd gave on that other DC thread.

4wd · « **Reply #19 on:** July 09, 2017, 03:35 PM »

Did threads just get merged or have I been into the cider too much?

thanks, but that stackoverflow thread is a mess
-kalos (July 09, 2017, 09:14 AM)

How exactly is the first answer in the thread a "mess"?

Based on the manual page:

# Log in to the server. This can be done only once.
wget --save-cookies cookies.txt \
--keep-session-cookies \
--post-data 'user=foo&password=bar' \
--delete-after \
http://server.com/auth.php

# Now grab the page or pages we care about.
wget --load-cookies cookies.txt \
http://server.com/in...eresting/article.php

Substitute -i list.txt for the URL in the second command, list.txt contains a list of URLs to download.

Plus further down the answers:

I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.

When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.

Deozaan · « **Reply #20 on:** July 09, 2017, 03:41 PM »

Did threads just get merged or have I been into the cider too much?
-4wd (July 09, 2017, 03:35 PM)

They got merged. You can tell by some of the pixels

, also because some of the posts have different topics than others.

4wd · « **Reply #21 on:** July 09, 2017, 03:51 PM »

Did threads just get merged or have I been into the cider too much?
-4wd (July 09, 2017, 03:35 PM)

They got merged. You can tell by some of the pixels , also because some of the posts have different topics than others.
-Deozaan (July 09, 2017, 03:41 PM)

Thank heavens, I've only had one cider today, a very nice Thatchers Vintage Cider

kalos · « **Reply #22 on:** April 08, 2018, 04:13 PM »

Did threads just get merged or have I been into the cider too much?

thanks, but that stackoverflow thread is a mess
-kalos (July 09, 2017, 09:14 AM)

How exactly is the first answer in the thread a "mess"?

Based on the manual page:

# Log in to the server. This can be done only once.
wget --save-cookies cookies.txt \
--keep-session-cookies \
--post-data 'user=foo&password=bar' \
--delete-after \
http://server.com/auth.php

# Now grab the page or pages we care about.
wget --load-cookies cookies.txt \
http://server.com/in...eresting/article.php

Substitute -i list.txt for the URL in the second command, list.txt contains a list of URLs to download.

Plus further down the answers:
I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.

When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.
-4wd (July 09, 2017, 03:35 PM)

Thanks but, the problem with that is that some cookies/form-logins expire. So when you go on downloading a file it displays the form to login instead.
That's why I am looking for a software that can detect that and use my login each time it is need.
Is there a way to do this?

4wd · « **Reply #23 on:** April 08, 2018, 07:55 PM »

Thanks but, the problem with that is that some cookies/form-logins expire. So when you go on downloading a file it displays the form to login instead.
That's why I am looking for a software that can detect that and use my login each time it is need.
Is there a way to do this?
-kalos (April 08, 2018, 04:13 PM)

So you're saying that the cookies expire in the couple of seconds between sending the first command followed by the second command?

kalos · « **Reply #24 on:** April 14, 2018, 03:22 PM »

I don't feel comfortable using WGET. I need a GUI so that I can see the progress etc.

Is there a GUI download manager that will allow me to download a list of urls, from a login website, delaying 5 seconds between each attempt? Ideally, it could scan the links of the url and download any files of specific extensions in those urls?