Main Area and Open Discussion > Living Room

grab urls

<< < (4/16) > >>

kalos:
Hello!

I want to download a list of urls in the following format:

http://domain.com/mod/resource/view.php?id=500691
(with id changing)

How can I download the urls and have the download program to use cookies from my Firefox or maybe enter username/password in the login form when necessary?

Firefox stores cookies in an sqlite file, which makes it tricky, but I can use other browser instead?

Thanks!

4wd:
https://www.donationcoder.com/forum/index.php?topic=43236.msg404268#msg404268

kalos:
thanks, but that stackoverflow thread is a mess
it is not clear if I need CURL to make wget work with POST authentication

wraith808:
thanks, but that stackoverflow thread is a mess
it is not clear if I need CURL to make wget work with POST authentication
-kalos (July 09, 2017, 09:14 AM)
--- End quote ---

curl and wget are two ways of doing the same thing.  Your answers are in the links 4wd gave on that other DC thread.

4wd:
Did threads just get merged or have I been into the cider too much?

thanks, but that stackoverflow thread is a mess-kalos (July 09, 2017, 09:14 AM)
--- End quote ---

How exactly is the first answer in the thread a "mess"?

Based on the manual page:

# Log in to the server.  This can be done only once.                   
wget --save-cookies cookies.txt \
     --keep-session-cookies \
     --post-data 'user=foo&password=bar' \
     --delete-after \
     http://server.com/auth.php

# Now grab the page or pages we care about.
wget --load-cookies cookies.txt \
     http://server.com/interesting/article.php
--- End quote ---

Substitute -i list.txt for the URL in the second command, list.txt contains a list of URLs to download.

Plus further down the answers:
I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.

When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.
--- End quote ---

Navigation

[0] Message Index

[#] Next page

[*] Previous page