topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • November 16, 2018, 03:12 AM
  • Proudly celebrating 13 years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: grab urls  (Read 10178 times)

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 9,814
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #25 on: April 14, 2018, 03:39 PM »
I don't feel comfortable using WGET. I need a GUI so that I can see the progress etc.

Is there a GUI download manager that will allow me to download a list of urls, from a login website, delaying 5 seconds between each attempt? Ideally, it could scan the links of the url and download any files of specific extensions in those urls?



Downthemall might do it.  I haven't used it in forever.  It probably wouldn't do it as is- what you'd have to do is make a simple site with the links, and then open it in firefox.  All of this is, however, speculative, as I haven't tried it.

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,006
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #26 on: April 15, 2018, 12:08 AM »
I don't feel comfortable using WGET. I need a GUI so that I can see the progress etc.

Is there a GUI download manager that will allow me to download a list of urls, from a login website, delaying 5 seconds between each attempt? Ideally, it could scan the links of the url and download any files of specific extensions in those urls?

Downthemall might do it.

It was suggested, he doesn't like it or it's too complicated.

JDownloader as suggested by Deozaan could most likely handle it, (logging in and downloading), but if he baulks at DownThemAll he's going to have no hope with JDownloader.

I don't feel comfortable using WGET. I need a GUI so that I can see the progress etc.

It outputs it's progress to the CLI window.

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,685
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #27 on: April 19, 2018, 03:56 PM »
It's not that I don't like DownThemAll.
First, it doesn't work with latest Firefox.
Second, it doesn't have the option to download urls with delay between them, it will crash the server and get me banned!

tomos

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 11,347
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #28 on: April 19, 2018, 11:27 PM »
^ already addressed AFAICS.
And you say you can use a different browser - try PaleMoon or Firefox ESR
I gave Downthemall a try, but is there a way to make the downloads delay every x seconds so that I won't abuse the server???
You can configure concurrent downloads and downloads-per-server in DownThemAll - that really should be all you need to to avoid "abusing" anything :)
Tom

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,006
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #29 on: April 20, 2018, 04:42 AM »
^ already addressed AFAICS.
And you say you can use a different browser - try PaleMoon or Firefox ESR
I gave Downthemall a try, but is there a way to make the downloads delay every x seconds so that I won't abuse the server???
You can configure concurrent downloads and downloads-per-server in DownThemAll - that really should be all you need to to avoid "abusing" anything :)


dTA! can't insert a delay between downloads, the next is fired off as soon as the previous finishes.

wget can insert a delay, (including a random delay).

Code: Text [Select]
  1. @echo off
  2. wget --save-cookies cookies.txt --keep-session-cookies --post-data 'user=foo&password=bar' --delete-after http://server.com/auth.php
  3. wget --load-cookies cookies.txt --secure-protocol="auto" --no-http-keep-alive -w 60 --random-wait --input-file="list.txt"

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,685
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #30 on: April 22, 2018, 12:19 PM »
I installed Jdowloader2 and this horrible software changed my Firefox search engine to horrible bing.com!
I wonder how can I revert it back and what other 'alterations' it made to my system??

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,006
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #31 on: April 22, 2018, 08:52 PM »
I installed Jdowloader2 and this horrible I failed to read any of the installation requests and blindly clicked buttons so the software changed my Firefox search engine to horrible bing.com!

FTFY

 :D

EDIT: While it might seem I'm making fun of you above it is only indirectly because the point I am making, (and has been made in other threads on this site regarding unwanted things happening during installation of some software), is I have installed JDownloader2 half a dozen times across various machines and not once has it ever changed the search engine in any browser I use, therefore you are doing something wrong.
« Last Edit: April 22, 2018, 10:25 PM by 4wd »

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,685
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #32 on: April 23, 2018, 06:05 PM »
I am very disappointed that some people here suggested me JDownloader.

After installing that horrible software, it hijacked my Firefox, changed homepage and search engine, and I believe it did more harm as now my google searches behave strange.
As you can see in the video below, after one second of loading the google results, some strange links appear!
<a href="https:// ufile.io/k18fj"></a>

What horrendous piece of software is that and why do you really suggest it?
« Last Edit: April 23, 2018, 09:02 PM by wraith808, Reason: Modified URI so it\'s not clickable. »

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 9,814
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #33 on: April 23, 2018, 09:02 PM »
^ You posted that twice after 4wd said it doesn't do that and he installed it himself?

Stephen66515

  • Animated Giffer in Chief
  • Honorary Member
  • Joined in 2010
  • **
  • Posts: 3,457
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #34 on: April 23, 2018, 09:56 PM »
I just downloaded JDownloader and it instantly got killed by my system with a "Severe" threat level, because of: Trojan:Win32/Occamy.C

I was only doing this so I could poke through the installation and have a looksie what's going on but now I'm curious if this is a FP or otherwise?  I got the download from http://jdownloader.org/download/index which redirected me to a mega.nz link in order to download "Install JDownloader.rar".

**edit**

I've googled to check if anybody else has had this show up with JDownloader and found nothing...which has me even more curious now.

jdown.png
jdown2.png

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 9,814
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #35 on: April 23, 2018, 10:31 PM »
The URL was not detected by virustotal...

https://www.virustot...7d43e36f5d/detection

Stephen66515

  • Animated Giffer in Chief
  • Honorary Member
  • Joined in 2010
  • **
  • Posts: 3,457
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #36 on: April 23, 2018, 10:35 PM »
The URL was not detected by virustotal...

https://www.virustot...7d43e36f5d/detection

That just seems to have scanned the Mega.nz website rather than the RAR that came from there.

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 9,814
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #37 on: April 23, 2018, 11:03 PM »
It scans through the url doesn't it?

Just in case, I downloaded it, and scanned the file (Install JDownloader.rar) with Malwarebytes.  Nothing.

I then uploaded that to virustotal.

https://www.virustot...211b7d581f/detection

There are a few, but I think it's a heuristic failure or something, looking at the results.  It does have adware in it, apparently, which might as 4wd put it, cause the symptoms stated if installed.

I don't care enough to go further than that.
« Last Edit: April 23, 2018, 11:12 PM by wraith808 »

Stephen66515

  • Animated Giffer in Chief
  • Honorary Member
  • Joined in 2010
  • **
  • Posts: 3,457
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #38 on: April 23, 2018, 11:07 PM »
It scans through the url doesn't it?

I clicked the link and it just seems to be scanning Mega as a page, rather than the download - I could be wrong but that's how it looks.  I kinda wanna download the file again and let it through, then upload it to be scanned after...but then again, I don't want my computer to implode...sooooooooooooooo.......I won't do it haha...it's not worth setting up a secure environment to do it...I'm really not THAT bothered, just slightly curious seeing as 4wd has recommended it above.

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,006
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #39 on: April 24, 2018, 12:04 AM »
I just downloaded JDownloader and it instantly got killed by my system with a "Severe" threat level, because of: Trojan:Win32/Occamy.C
-Stephen66515 (April 23, 2018, 09:56 PM)

Downloaded, extracted, nothing according to MBAM and Defender:

2018-04-24 14_49_09-K__.pnggrab urls

@Stephen: Interested to know how you got that particular result since none of the engines on VirusTotal report it as such: Results

The comments against it on VirusTotal suggest that it will ask you to install "offers" or "toolbars" ... never had it do that but maybe it is based on what it sees as the default browser.  Only installed browsers I have are Vivaldi and Edge, everything else is portable.


But anyway, RevoUninstalled my existing installation, (had a RevoUninstaller log for it to really clean up), and re-installed.
NOTE: I should point out, if there is a button that says Advanced or Custom Installation, use it - that's where "extras" are normally hidden.

2018-04-24 14_56_31-.pnggrab urls2018-04-24 14_56_39-.pnggrab urls2018-04-24 14_56_44-.pnggrab urls2018-04-24 15_02_01-.pnggrab urls

It required 4 separate internet connections for the installation files:

2018-04-24 15_00_23-Windows Firewall Control - Rules Panel.pnggrab urls

During the installation, MBAM and Defender reported nothing, when it was finished it opened a page in the browser, (Vivaldi), saying thanks for the install - only thing bad to say about it was it stuck an icon on the Desktop after I told it not to.

Scanning of the temp files after installation with MBAM showed the following:

2018-04-24 15_10_26-Malwarebytes Anti-Malware.png
2018-04-24 15_09_39-Malwarebytes Anti-Malware Home (Premium) 2.2.1.1043.pnggrab urls

Not a threat, just an annoyance ... besides the fact that it actually didn't install anything.

PS:
... seeing as 4wd has recommended it above.
-Stephen66515 (April 23, 2018, 11:07 PM)

Didn't recommend it, I reinterated that @Deozaan suggested it and mentioned it could most likely do what he wanted, I'll keep with my suggestion of using wget even though I do regularly use JDownloader.
« Last Edit: April 24, 2018, 01:30 AM by 4wd »

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Points: 1
  • Posts: 8,563
    • View Profile
    • The Blog of Deozaan
    • Donate to Member
Re: grab urls
« Reply #40 on: April 24, 2018, 12:50 AM »
I downloaded from your link and Windows Defender immediately flagged it with the same results you got, Stephen.

https://www.microsof...;threatid=2147726780

EDIT: Something is really fishy about the site. The download links redirect you to all sorts of places before finally ending up on Mega. And supposedly JDownloader is Open Source but I couldn't find the source code.
« Last Edit: April 24, 2018, 01:03 AM by Deozaan »

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,006
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #41 on: April 24, 2018, 12:55 AM »
I downloaded from your link and Windows Defender immediately flagged it with the same results you got, Stephen.

https://www.microsof...;threatid=2147726780

That is completely weird then, Defender is up to date - it's also strange that Microsoft report something completely different via VirusTotal, (PUA:Win32/Creprote - Potentially Unwanted Application, I'm guessing), for both the RAR and EXE.

And supposedly JDownloader is Open Source but I couldn't find the source code.

It seems to be available via SVN repository, (old version at GitHub).

Something is really fishy about the site. The download links redirect you to all sorts of places before finally ending up on Mega.

It appears to be their own CDN, (there's ~3 fetch.jdcdn.org links it redirects through):

Code: Text [Select]
  1. Whois query for jdcdn.org...
  2. Results returned from whois.publicinterestregistry.net:
  3. Domain Name: JDCDN.ORG
  4. Registrant Name: Thomas Rechenmacher
  5. Registrant Organization: AppWork GmbH

AppWork is the company that wrote JDownloader.
« Last Edit: April 24, 2018, 08:46 AM by 4wd »

Stephen66515

  • Animated Giffer in Chief
  • Honorary Member
  • Joined in 2010
  • **
  • Posts: 3,457
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #42 on: April 24, 2018, 10:09 AM »
I downloaded from your link and Windows Defender immediately flagged it with the same results you got, Stephen.

https://www.microsof...;threatid=2147726780

EDIT: Something is really fishy about the site. The download links redirect you to all sorts of places before finally ending up on Mega. And supposedly JDownloader is Open Source but I couldn't find the source code.

Yeah that's what struck me as odd - The CDN redirect didn't concern me that much but then being bounced around until finally landing on mega (why use a CDN if you're just gonna host your download elsewhere?) - The download was flagged and stopped by Chrome by Windows Defender (images above).

The whole thing just strikes me as very very odd indeed.

Note: In case the developer of JDownloader pops by and reads this, I'm not suggesting that the software IS bad/malware, I'm simply pointing out that it FEELS that way (again, the report from Defender could certainly be a false positive, and it wouldn't be the first time that has happened).

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Points: 1
  • Posts: 8,563
    • View Profile
    • The Blog of Deozaan
    • Donate to Member
Re: grab urls
« Reply #43 on: April 24, 2018, 01:20 PM »
Something is really fishy about the site. The download links redirect you to all sorts of places before finally ending up on Mega.

It appears to be their own CDN, (there's ~3 fetch.jdcdn.org links it redirects through):

Code: Text [Select]
  1. Whois query for jdcdn.org...
  2. Results returned from whois.publicinterestregistry.net:
  3. Domain Name: JDCDN.ORG
  4. Registrant Name: Thomas Rechenmacher
  5. Registrant Organization: AppWork GmbH

AppWork is the company that wrote JDownloader.

I figured jdcdn was JDownloader CDN, but the final redirect before ending up on Mega was some fishy looking firsturl.de (archived link) site. It looked like one of those sites that forces you to wait 30 seconds viewing ads before allowing you to get to the real link. Except it didn't force me to wait and just redirected me to Mega after a second or so. Seems it's just a URL shortener/tracker link.

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,685
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #44 on: April 24, 2018, 01:38 PM »
I just downloaded JDownloader and it instantly got killed by my system with a "Severe" threat level, because of: Trojan:Win32/Occamy.C

I was only doing this so I could poke through the installation and have a looksie what's going on but now I'm curious if this is a FP or otherwise?  I got the download from http://jdownloader.org/download/index which redirected me to a mega.nz link in order to download "Install JDownloader.rar".

**edit**

I've googled to check if anybody else has had this show up with JDownloader and found nothing...which has me even more curious now.

[ Invalid Attachment ]
[ Invalid Attachment ]
-Stephen66515 (April 23, 2018, 09:56 PM)

Interesting, what system do you have?
To be honest, I may did a mistake and installed the toolbars the installer offers at the begining.
But isn't there a way to clean my system? I still get the fake google results.
I have tried everything RogueKiller, Zemana, MalwareBytes, AdwCleaner, JRT, with no luck.
« Last Edit: April 24, 2018, 02:38 PM by kalos »

tomos

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 11,347
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #45 on: April 24, 2018, 02:51 PM »
I installed Jdowloader2 and this horrible software changed my Firefox search engine to horrible bing.com!
I wonder how can I revert it back and what other 'alterations' it made to my system??
Were you not change (back) the default search engine and home page yourself (?)

And can you elaborate on the 'fake google results'?
Tom

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,006
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #46 on: April 24, 2018, 03:38 PM »
The whole thing just strikes me as very very odd indeed.
-Stephen66515 (April 24, 2018, 10:09 AM)

What I find odd is across six different systems using the same AV you're getting three different results on the same file:

Stephen: Trojan (Chrome)
Deozaan: Trojan
Wraith: Clean
Kalos: Clean (Firefox)
4wd: Clean (Vivaldi)
VirusTotal: PUA

I'm assuming that both Wraith and kalos didn't get a hit by Defender because neither said the installer was quarantined.
I'm also assuming that kalos used Firefox since he never mentions anything else.

Now I'm going to guess that Deozaan uses Chrome and Wraith, Not Chrome ?

VirusTotal, yeah well, its results change dependent on wind direction and time by the looks, only have to look at the results for the keylogger a couple of days ago, started off at 14/64, then 22/65, then 26/66, now who knows what they are.  That and the overly scary (mis)classifications used by the majority of AV manufacturers based on what something looks like, not what it does ... its results are something I take with a grain of salt, (my opinion only).
« Last Edit: April 24, 2018, 03:44 PM by 4wd »

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,685
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #47 on: April 24, 2018, 03:41 PM »
I installed Jdowloader2 and this horrible software changed my Firefox search engine to horrible bing.com!
I wonder how can I revert it back and what other 'alterations' it made to my system??
Were you not change (back) the default search engine and home page yourself (?)

And can you elaborate on the 'fake google results'?

I did change back the homepage and search engine, but who knows what other damage it has caused.
About fake google results, I did post a video earlier:
<a href="https://ufile.io/k18fj"></a>

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 9,814
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #48 on: April 24, 2018, 03:43 PM »
I installed Jdowloader2 and this horrible software changed my Firefox search engine to horrible bing.com!
I wonder how can I revert it back and what other 'alterations' it made to my system??
Were you not change (back) the default search engine and home page yourself (?)

And can you elaborate on the 'fake google results'?


I've seen it before on infected systems.  They hijack any searches to do it on another platform and make it look pretty similar to google.  Changing the search engine doesn't change this; it's an extension/plugin/BHO.

I installed Jdowloader2 and this horrible software changed my Firefox search engine to horrible bing.com!
I wonder how can I revert it back and what other 'alterations' it made to my system??
Were you not change (back) the default search engine and home page yourself (?)

And can you elaborate on the 'fake google results'?

I did post a video earlier:
<a href="https://ufile.io/k18fj"></a>

I changed that in the prior post to not be link, because archive.is shows it linking to some sketchy sites.  I'll leave it, but you might want to look at using a more reputable site to host your downloads- perhaps even just attach it here?

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,685
    • View Profile
    • Donate to Member
Re: grab urls
« Reply #49 on: April 24, 2018, 04:21 PM »
I installed Jdowloader2 and this horrible software changed my Firefox search engine to horrible bing.com!
I wonder how can I revert it back and what other 'alterations' it made to my system??
Were you not change (back) the default search engine and home page yourself (?)

And can you elaborate on the 'fake google results'?


I've seen it before on infected systems.  They hijack any searches to do it on another platform and make it look pretty similar to google.  Changing the search engine doesn't change this; it's an extension/plugin/BHO.

I installed Jdowloader2 and this horrible software changed my Firefox search engine to horrible bing.com!
I wonder how can I revert it back and what other 'alterations' it made to my system??
Were you not change (back) the default search engine and home page yourself (?)

And can you elaborate on the 'fake google results'?

I did post a video earlier:
<a href="https://ufile.io/k18fj"></a>

I changed that in the prior post to not be link, because archive.is shows it linking to some sketchy sites.  I'll leave it, but you might want to look at using a more reputable site to host your downloads- perhaps even just attach it here?

Sure, here it is.
EDIT: oh it doesn;t let me upload flv! :(