Author Topic: SOLVED: URL Extractor (Read 18351 times)

magician62 · « **on:** August 26, 2011, 05:46 AM »

Most URL extractors I have found will take an HTML page and extract all URLs, but this is not what I am trying to do.
I am not looking for a downloader program, the URL are for later processing.

Scenario On Youtube, you are presented with thumbnails of other videos, each linking to their page.

What I want to be able to do is drag an image from Youtube onto "Program-X" and extract the URL
I want to be able to repeat this action so as to build a list of URL's for later processing, with each entry on a new line.
The list should be saveable in .txt format
Another optional would be the ability to copy all the URL for pasting elsewhere.

At present the steps are:-

Create .txt file
Open text file
Select thumbnail
Right click Thumbnail
Copy Shortcut
Select .txt file
Paste URL (either right click, paste or CTRL-V)
Enter (to move to next line for next entry)

The new idea should bring this down to
drag-drop
Save .txt file

TIA

rjbull · « **Reply #1 on:** August 26, 2011, 05:55 AM »

I am not looking for a downloader program, the URL are for later processing.
-magician62 (August 26, 2011, 05:46 AM)

Don't overlook downloaders... I think Internet Download Manager (IDM) can do what you want. You can clip an area on a Web page, then go Tasks > Add batch download from clipboard. It gives you a screen where everything is listed, with checkboxes to mark those you do/don't want. Those required are put into a queue, which you can start at any later time you want.

magician62 · « **Reply #2 on:** August 26, 2011, 08:41 AM »

Thanks for the reply. I am a bit concerned about using IDM, as I already run Getright (another paid program), which performs much of the same function, but for what I am trying to do, still seems to adds a number of extra steps.

The purpose of the util, is not necessarily to download the files at the link, just to extract the URL's

I did find Skwire's ClipTrap, which goes part-way there. though it doesn't have drag drop addition of new links.

bob99 · « **Reply #3 on:** August 26, 2011, 10:38 AM »

I am not looking for a downloader program, the URL are for later processing.
-magician62 (August 26, 2011, 05:46 AM)
Don't overlook downloaders... I think Internet Download Manager (IDM) can do what you want. You can clip an area on a Web page, then go Tasks > Add batch download from clipboard. It gives you a screen where everything is listed, with checkboxes to mark those you do/don't want. Those required are put into a queue, which you can start at any later time you want.
-rjbull (August 26, 2011, 05:55 AM)

I currently have Downloadhelper, FlashGot, DownThemAll, Scrapbook, Screengrab and some other add-ons in Firefox. At one time or another one of them has been able to do something that one of the others couldn't, or was easier with. Thinking of trying IDM and maybe now Getright.
Could I ask your thoughts on whether IDM or Getright could replace the add-ons I have?
I'd like to try and streamline the number of programs I use if it's even feasible.

Thanks,

magician62 · « **Reply #4 on:** August 26, 2011, 11:00 AM »

From what I can tell IDM may have slightly different additional features to Getright, ( I have used getright for about 10 years, though it may have been longer, currently use the Pro version). As to if either would suitably replace what you currently use, I will leave to others to answer. Unfortunately the last time I installed Firefox, it screwed my system, and I had to spend a couple of days sorting it. The closest it gets now is a stick

skwire · « **Reply #5 on:** August 26, 2011, 12:21 PM »

If nobody gets to this within a few days, I'll take care of it.

skwire · « **Reply #6 on:** August 26, 2011, 12:27 PM »

Actually, now that I think about it, WordPad can accept a dropped URL and extract the text. Yes, you would have to hit enter to get a new line but that's not so bad, right? Let me know if that will be an acceptable solution.

magician62 · « **Reply #7 on:** August 26, 2011, 01:06 PM »

I have used Word in the past. Usually create 1000's of lines, the copying teh result to a notepad. Interesting idea of using Wordpad, not tried that, unfortunately it didnt work. Though I now wonder if there are system issues. On dragging the imagelink to wordpad, my cursor changed to a red bordered circle with diagonal line (Prohibited Icon)! Same happened in Word. Will investigate further, as this could be why things are not working

skwire · « **Reply #8 on:** August 26, 2011, 01:12 PM »

FWIW, I just drag-n-drop tested WordPad with both a text link and an image link from both Internet Explorer and Firefox. All four tests were successful. This was on Windows XP SP3.

magician62 · « **Reply #9 on:** August 26, 2011, 01:20 PM »

I am thinking its Win 7 UAC. It used to work, jsut turned UAC off, and will try again

magician62 · « **Reply #10 on:** August 26, 2011, 01:48 PM »

I can confirm it was the UAC causing the drag drop issue.

A utility that processes the links to a text file may still be benefficial.

First it would eliminate the need for enter, after each line, or preparing empty lines first.
Second it would eliminate moving hand to mouse to perform each enter operation.

Outcome, a lot faster processing. Simpler for those who tend to work one handed, i.e mostly/only mouse

skwire · « **Reply #11 on:** August 26, 2011, 02:32 PM »

I can confirm it was the UAC causing the drag drop issue.
-magician62 (August 26, 2011, 01:48 PM)

Nice, thanks for tracking this down.

A utility that processes the links to a text file may still be benefficial.

First it would eliminate the need for enter, after each line, or preparing empty lines first.
Second it would eliminate moving hand to mouse to perform each enter operation.

Outcome, a lot faster processing. Simpler for those who tend to work one handed, i.e mostly/only mouse
-magician62 (August 26, 2011, 01:48 PM)

Fair enough, I'll write it for you. However, it will embed the RichEdit control that WordPad uses so you will need to leave UAC off. Cool?

magician62 · « **Reply #12 on:** August 26, 2011, 02:39 PM »

No Problems, with the UAC off. Its a great concept, but sometimes a real pain. I have a great utility, that must have it turned off, otherwise I can't use it, and unfortunately the developer is no longer with us to update it.

skwire · « **Reply #13 on:** August 26, 2011, 02:48 PM »

Great, it'll be a few days, but I have this on my ToDo (unless somebody else does it first).

rjbull · « **Reply #14 on:** August 26, 2011, 02:54 PM »

I currently have Downloadhelper, FlashGot, DownThemAll, Scrapbook, Screengrab and some other add-ons in Firefox. At one time or another one of them has been able to do something that one of the others couldn't, or was easier with. Thinking of trying IDM and maybe now Getright.
Could I ask your thoughts on whether IDM or Getright could replace the add-ons I have?
I'd like to try and streamline the number of programs I use if it's even feasible.
-bob99 (August 26, 2011, 10:38 AM)

Can't help much, I'm afraid. I haven't used any of those FF extensions, nor Getright. There's also ReGet, which I seem to remember had some feature that IDM didn't, but I'm happy with IDM now. I have used Free Download Manager (FDM), which supported proxy.pac files before IDM did and worked well, but not used it for a while.

IDM can download entire Web sites for offline browsing, but it doesn't replace utilities for clipping text or images from the Web, in the usual sense. You need something else for that. For that job I've used Evernote, Ultra Recall (a bit) and briefly looked at a few other things. In a way, Evernote sort-of answers the OP, because it does clip URLs. However, mine is the last desktop version, 2.2, and it's now gone cloud, which I don't want. CintaNotes is a good freestanding Web clipper as long as you only want text. Mouser's Clipboard Help+Spell (CHS) is a clipboard enhancer that now saves the URLs of Web clips (and also now clips images, despite the home page not having been updated to say so). But, neither CintaNotes nor CHS understand and save the URLs within a page, only the URL of the page itself. Another clipboard enhancer, ClipCache, does, but I've found recent versions somewhat fragile on my system. I'd guess ClipMate would be certain to, as the Big Daddy of the genre, but I don't have it installed at present.

Sorry, not much help.

rjbull · « **Reply #15 on:** August 26, 2011, 03:23 PM »

I want to be able to repeat this action so as to build a list of URL's for later processing, with each entry on a new line.
-magician62 (August 26, 2011, 05:46 AM)

magician62, do you use Firefox? Because if you do, here are some extensions that might help:

Copy Links
Copy multiple URLs in anchors to clipboard.
Copy Link URL
Copy the URLs of the selected links to clipboard.
Copy Link Text
Right-click a link to copy its text; also optionally offers ability to right-click drop-down menu items, buttons, and other form controls to get their text or values
CoLT
CoLT makes it easy to copy either a hyperlink's text or both the link and the link's text (in a format you specify). Two handy context-menu items make this possible, and don't add clutter; the items are only visible when right-clicking a link!

Try going to the Mozilla Add-Ons Page and searching "copy links" or "copy link."

skwire · « **Reply #16 on:** August 26, 2011, 10:43 PM »

magician62: Did you want this to save the list between sessions or just give a button to save off the list when you want to? Or both?

magician62 · « **Reply #17 on:** August 27, 2011, 10:12 AM »

A new list each session is fine, though I can see there may be a benefit to some for it being maintained between sessions of use. Either way would work for my needs.

rjbull
As mentioned earlier FF screwed my system some time back, so is rarely used, and lives on a stick

skwire · « **Reply #18 on:** August 27, 2011, 11:12 AM »

A new list each session is fine, though I can see there may be a benefit to some for it being maintained between sessions of use. Either way would work for my needs.
-magician62 (August 27, 2011, 10:12 AM)

I'll probably add three buttons: Clear, Save, and Copy

rjbull · « **Reply #19 on:** August 28, 2011, 11:09 AM »

rjbull
As mentioned earlier FF screwed my system some time back, so is rarely used, and lives on a stick
-magician62 (August 27, 2011, 10:12 AM)

Sorry, I overlooked that. Do you use Opera? Sanity check - I haven't tried this - but there's MakeLinks for Opera:

MakeLinks is an extension for the web browser Opera that will search for all web addresses in text form and display an in-page popup of clickable links. Optionally, there is a mode for automatic detection of text web addresses on the entire page.

skwire · « **Reply #20 on:** September 01, 2011, 12:16 AM »

I've done some work on this and the RichEdit control in the library I'm using only seems to work 100% with Firefox. With Internet Explorer, it only accepts links dragged from the icon in the address bar; links dragged from the actual page do not work. Furthermore, I can't make it work with Opera at all. Hell, I have a tough time just getting a link to drag from Opera to anywhere. So, with the Firefox limitation in mind, did you still want me to pursue this?

magician62 · « **Reply #21 on:** September 01, 2011, 06:35 AM »

Many thanks for trying, unfortunately IE is the browser of choice. I will live with the current method of drag to Wordpad, copy all lines from wordpad to notepad. Maybe teh future will provide other options.

skwire · « **Reply #22 on:** September 01, 2011, 07:34 AM »

copy all lines from wordpad to notepad
-magician62 (September 01, 2011, 06:35 AM)

Why are you doing this? Wordpad can save in straight plain-text on its own.

magician62 · « **Reply #23 on:** September 01, 2011, 09:35 AM »

Good point ! I forgot that as I tend to use Word, and the default save is .doc

skwire · « **Reply #24 on:** September 01, 2011, 11:34 AM »

Do you mind if I mark this thread as solved?