Topic: grab urls (Read 49,313 times)
probably doesnt handle the login, but downthemall?
probably doesnt handle the login, but downthemall?
Can I load a text file with the urls to that and it will use the cookie to authenticate and download?
DownThemAll is all you can desire from a download manager: it features an advanced accelerator that increases speed up to 4x and it allows you to pause and resume downloads at any time.
DownThemAll is fast, reliable and easy-to-use! It lets you download all the links or images contained in a webpage and much more: you can refine your downloads by fully customizable criteria to get only what you really want! Be in full control over your downloads, dedicated speed and number of parallel connections at any time. Use Metalinks or add mirrors manually to download a file from different servers at the same time.
DownThemAll is open-source and freeware. No Adware, no Spyware, no hidden costs!
I have a list of URLs that I want to download from a website that has a form login.
How can I do that automatically in batch?
Is there a downloader and can the downloader use the cookie from my Firefox to authenticate?
- however, I couldn't possibly comment. 
I gave Downthemall a try, but is there a way to make the downloads delay every x seconds so that I won't abuse the server???You can configure concurrent downloads and downloads-per-server in DownThemAll - that really should be all you need to to avoid "abusing" anything
- carpe noctem
thanks, but that stackoverflow thread is a mess
it is not clear if I need CURL to make wget work with POST authentication
thanks, but that stackoverflow thread is a mess
Based on the manual page:
# Log in to the server. This can be done only once.
wget --save-cookies cookies.txt \
--keep-session-cookies \
--post-data 'user=foo&password=bar' \
--delete-after \
http://server.com/auth.php
# Now grab the page or pages we care about.
wget --load-cookies cookies.txt \
http://server.com/interesting/article.php
I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.
When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.
Did threads just get merged or have I been into the cider too much?
, also because some of the posts have different topics than others.Did threads just get merged or have I been into the cider too much?
They got merged. You can tell by some of the pixels
Did threads just get merged or have I been into the cider too much?thanks, but that stackoverflow thread is a mess
How exactly is the first answer in the thread a "mess"?QuoteBased on the manual page:
# Log in to the server. This can be done only once.
wget --save-cookies cookies.txt \
--keep-session-cookies \
--post-data 'user=foo&password=bar' \
--delete-after \
http://server.com/auth.php
# Now grab the page or pages we care about.
wget --load-cookies cookies.txt \
http://server.com/interesting/article.php
Substitute -i list.txt for the URL in the second command, list.txt contains a list of URLs to download.
Plus further down the answers:QuoteI had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.
When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.
Thanks but, the problem with that is that some cookies/form-logins expire. So when you go on downloading a file it displays the form to login instead.
That's why I am looking for a software that can detect that and use my login each time it is need.
Is there a way to do this?
Original address: https://www.donationcoder.com/forum/index.php?topic=43236.0