Author Topic: Latest Web Link Captor Release - v1.11.01 - Dec 17, 2013 (Read 83126 times)

mouser · « **on:** December 11, 2011, 04:20 AM »

Web Link Captor is my new application, which was created as part of the NANY 2012 event at DonationCoder.com

Web Link Captor is a program that takes a list of plaintext items, and builds an output document containing a list of url-linked results based on web searches for the items. It's a way to quickly build a clickable list of items from just their names, suitable for posting on your blog or a forum, etc.

Download latest version from Official Web Page: HERE (download now available to the public).

Watch videos of the program in action: Part 1 and Part 2.

Features:

Friendly GUI lets you build flexible sets of processing steps to parse plaintext input lists and perform search and output functions.
You can save and load projects and script configurations for easy re-use.
Uses a set of standalone utility scripts that can be run without the windows front end GUI, and can be easily modified and expanded.
The included backend scripts are in Python (but other languages can be used)
Uses caching to avoid repeating web searches unnecessarily.
Comes with output formatters for HTML, BBCode, Markdown/Wikipedia
Comes with searching engines for google, and bing/opensearch (which can handle amazon, wikipedia, and many others)
Flexible scoring system lets you interactively or programmatically tweak your results to get the best list of links.

Version History

v1.11.01 - December 17, 2013

[Feature] Scripts now create different temporary filenames in temp directory to make it much easier to debug where a search or manipulation script went wrong.
[Documentation] Added help information for how to narrow a search to a specific site in google and how to add text to input.
[BugFix] Excerpts in result grid that contained newlines were messing up the result grid (and potentially the results).
[Feature] Option to check for updates at startup.

v1.10.01 - April 17, 2012

[BugFix] Double-quotes in search strings were not being handled properly and resulting in 0 results

v1.08.01 - December 24, 2011

[Feature] Now packaging with a minimal python addon included

v1.07.01 - December 23, 2011

[Feature] New script combobox is hierarchical.
[Feature] Script Tool files now specified hierarchical path and unique identifier which will make it easier to change tool labels without losing track of scripts.
[Feature] Better error reporting on missing scripts.

v1.06.01 - December 22, 2011

[MinorFeature] Added plaintext output script
[BugFix] Fixed bug where blank labels could cause crash
[Minor Feature] New option in output to remove labels from items
[MinorFeatures] Minor improvements

v1.05.01 - December 15, 2011

[MajorFeature] You can now download Python2 addon pack for the program, rather than using a standard python installation.
[MinorFeatures] Lots of small user interface improvements.

v1.03.01 - December 13, 2011

[MajorFeature] Embedded Python 2.7 DLL mode option -- MUCH faster
[Feature] Improved all python scripting invocation features and options and error reporting
[MinorFeature] Added more gui options including ability to hide help panels
[MinorFeature] Small improvements to tools (ability to show readme links)

v1.02.01 - December 13, 2011

[MajorFeature] You can now use an embedded DLL python interpretter option
[Feature] Cache controls from options
[Feature] Includes additional search scripts from ewemoa (thanks) [note they require some additional module installations]
[MinorFeature] Reports elapsed time for operations
[MinorFeature] Scripts can now include readme files and link to them from their tool descriptions.
[MinorFeature] Some reorganization of script files
[MinorFeature] Hover over links shows url in statusbar of web preview.

v1.01.05 - December 11, 2011

Regular expression script improvements -- should be faster and may work on Python 2.6 now

v1.01.04 - December 11, 2011

Tools/Scripts improvements
Minor GUI tweaks

You can read the original NANY2012 discussion that led to this program here.

mouser · « **Reply #1 on:** December 11, 2011, 04:24 AM »

The program will be officially released as part of NANY 2012, on January 1st, 2012.

BUT If you would like to try it early, just send me a personal message via the forum or email me ([email protected]) and I will send you a link.

mouser · « **Reply #2 on:** December 11, 2011, 04:45 AM »

As a super simple example of the kinds of things that WLC can do, I put in a list of a few of my favorite bands, and click "Run" to get the following output, ready to paste in a forum post:

Input:

Cat Power - The Greatest
Bikini Kill - Reject All American
The Doo Wop Box - Volume one
Dexy's Midnight Runners - Greatest Hits

Output (raw):

[Select]

[url=http://www.amazon.com/Greatest-Cat-Power/dp/B000C0X3ZC]Cat Power[/url]
[url=http://www.amazon.com/Reject-All-American-Bikini-Kill/dp/B00000373N]Bikini Kill[/url]
[url=http://www.amazon.com/Doo-Wop-Box-Various-Artists/dp/B00000333M]The Doo Wop Box[/url]
[url=http://www.amazon.com/Too-Rye-Ay-Dexys-Midnight-Runners/dp/B00005Y1ZF]Dexy's Midnight Runners[/url]

Output:

And here's output from a different kind of search and output on those same bands:

Cat Power:

Bikini Kill:

The Doo Wop Box:

Dexy's Midnight Runners:

mouser · « **Reply #3 on:** December 11, 2011, 04:57 AM »

One of the major goals of the program is to provide a flexible way to rank and filter search results.

This can be done interactively using the result grid, by using pattern matching to modify scores, by adding terms and options to web searches, and by using external processing scripts to apply heuristics.

In such a way one can hope to filter out results that are undesirable and favor sites that are more likely to have good results.

I'm hoping that people will write and share additional such filter and search scripts with us to improve the program. The included scripts are written in python and it's (relatively) easy to add your own new ones to the system.

mouser · « **Reply #4 on:** December 11, 2011, 04:59 AM »

I hope the program is useful to people. There is a lot that could still be added to it. I will be posting a screencast video tutorial in the next day or so.

And lastly.. If you'd like to see this program continue to be developed, please consider making a donation -- It would be very appreciated. Thank you

CleverCat · « **Reply #5 on:** December 11, 2011, 05:25 AM »

This looks very promising - can't wait to try it!

kyrathaba · « **Reply #6 on:** December 11, 2011, 06:39 AM »

That is a really useful application!

mouser · « **Reply #7 on:** December 11, 2011, 10:50 AM »

I've uploaded two video screencasts of the program in action:

Video 1 is four minutes and just shows the program in action:

Screenshot - 12_24_2011 , 8_05_58 AM.png

.

Video 2 is ten minutes long and goes into more detail:

Screenshot - 12_11_2011 , 10_48_38 AM.png

.

Ath · « **Reply #8 on:** December 11, 2011, 01:08 PM »

Just watched the video, and I'm very impressed

Great program.

Had to use the DC hosted version, as the screen-text in the youtube release was nearly unreadable for me

IainB · « **Reply #9 on:** December 11, 2011, 03:46 PM »

+1 from me for what Ath wrote.
Quite elegant!

Just curious:
On the input: would it be possible to direct the proggy to recognise and pick up the search terms from CHS, if they were already in the CHS database?
On the searched sources: would it be possible to have the proggy also search CHS for matches? (I think this would imply that the data to be matched was in the Clip data, and the related URL reference would be a string (or strings) in the Notes column/field or the Clip data.)

mouser · « **Reply #10 on:** December 11, 2011, 03:49 PM »

Leaving aside the specific issue of getting data from CHS.. Let me say that it would absolutely be possible to have the program "search" for "results" not on the web, but in a local database or text file catalog, etc. and there is no reason that the results would necessarily have to be urls.

And in a more general sense, the codebase for this program is really all about instantiating chains of parameterized processing scripts that work on (text) files.. and the code may have additional uses in creating different programs that do different things but use the same paradigm of letting user assemble and configure chains of batch scripts to run.

Deozaan · « **Reply #11 on:** December 11, 2011, 09:48 PM »

Watch the original full size and much higher quality video from the DC server:
-mouser (December 11, 2011, 10:50 AM)

Great screencast. But I'm curious why YouTube couldn't handle the full size/high quality video? Was it a limitation or a choice to have lower quality on YouTube?

WLC looks really awesome! It makes me wish I wrote more lists that needed links.

mouser · « **Reply #12 on:** December 11, 2011, 10:06 PM »

Was it a limitation or a choice to have lower quality on YouTube?

certainly wasn't by my choice.. i also don't know why it shows it in bad quality. the original file i 800x600.

ewemoa · « **Reply #13 on:** December 12, 2011, 06:03 AM »

I've uploaded a long (sorry) video demonstration of the program in action:

Watch on youtube:
...
-mouser (December 11, 2011, 10:50 AM)

I found that it was easier to see the text in the 480p version.

Also, it's at about 8:40 that one gets to see the first generated web preview. May be that will make 20 minutes seem like less

ewemoa · « **Reply #14 on:** December 12, 2011, 06:28 AM »

In the output preview section, is there a recommended way to transfer the output elsewhere? With the appropriate tab selected Control+A, Control+C, and Control+V at the destination?

mouser · « **Reply #15 on:** December 12, 2011, 06:49 AM »

In the output preview section, is there a recommended way to transfer the output elsewhere? With the appropriate tab selected Control+A, Control+C, and Control+V at the destination?

yeah i suppose i really should have a Copy To Clipboard button shouldn't I?

IainB · « **Reply #16 on:** December 12, 2011, 03:32 PM »

In the output preview section, is there a recommended way to transfer the output elsewhere? With the appropriate tab selected Control+A, Control+C, and Control+V at the destination?

yeah i suppose i really should have a Copy To Clipboard button shouldn't I?
-mouser (December 12, 2011, 06:49 AM)

Maybe also consider implementing a design concept of structured integration with CHS, rather than ad hoc extensions?
So you might get, for example (from this thread so far), options to:

INPUT: recognise and pick up the search terms from CHS, if they were already in the CHS database.
PROCESS: include CHS database records (of some definable type) in the search for matches.
OUTPUT: output results to CHS.

That could save a lot of duplication of effort and mucking about when using the WLC (assuming that you already had the relevant data in CHS and wanted the output in there too.)

On another point, I would be very interested in ß testing WLC if you were opening it up for "...additional uses in creating different programs that do different things but use the same paradigm of letting user assemble and configure chains of batch scripts to run" (as you had intimated above).

mahesh2k · « **Reply #17 on:** December 12, 2011, 04:02 PM »

I see one more use of WLC : Capture the URLs from various web pages into clipboards and later add it to WLC for uploading these urls into delicious or similar other sites. This saves time, instead of using browser widgets and bookmark link, this can process large chunk of URLS. I hope this could be possible with extensions.

IainB · « **Reply #18 on:** December 12, 2011, 04:28 PM »

In terms of "other uses", I'm not sure, but I wondered if this app. might be relevant: Boxie - Fix Music File Names (from id3 tags)
I wanted to take a look at their website but Malwarbytes tells me it is a dodgy URL, so I didn't.

mouser · « **Reply #19 on:** December 12, 2011, 04:36 PM »

Well one very very natural extension (simplification really) of WLC would be as a combination GUI and extensible script collection for performing intricate text manipulation tasks. It wouldn't be able to DO things you couldn't do using a scripting language, but it could make creating, configuring, and previewing chains of operations much easier to work with.

mouser · « **Reply #20 on:** December 12, 2011, 05:29 PM »

Ive replaced the video with one half as long.

mouser · « **Reply #21 on:** December 22, 2011, 10:48 PM »

Version History

v1.07.01 - December 23, 2011

[Feature] New script combobox is hierarchical.
[Feature] Script Tool files now specified hierarchical path and unique identifier which will make it easier to change tool labels without losing track of scripts.
[Feature] Better error reporting on missing scripts.

v1.06.01 - December 22, 2011

[MinorFeature] Added plaintext output script
[BugFix] Fixed bug where blank labels could cause crash
[Minor Feature] New option in output to remove labels from items
[MinorFeatures] Minor improvements

v1.05.01 - December 15, 2011

[MajorFeature] You can now download Python2 addon pack for the program, rather than using a standard python installation.
[MinorFeatures] Lots of small user interface improvements.

v1.03.01 - December 13, 2011

[MajorFeature] Embedded Python 2.7 DLL mode option -- MUCH faster
[Feature] Improved all python scripting invocation features and options and error reporting
[MinorFeature] Added more gui options including ability to hide help panels
[MinorFeature] Small improvements to tools (ability to show readme links)

v1.02.01 - December 13, 2011

[MajorFeature] You can now use an embedded DLL python interpretter option
[Feature] Cache controls from options
[Feature] Includes additional search scripts from ewemoa (thanks) [note they require some additional module installations]
[MinorFeature] Reports elapsed time for operations
[MinorFeature] Scripts can now include readme files and link to them from their tool descriptions.
[MinorFeature] Some reorganization of script files
[MinorFeature] Hover over links shows url in statusbar of web preview.

v1.01.05 - December 11, 2011

Regular expression script improvements -- should be faster and may work on Python 2.6 now

v1.01.04 - December 11, 2011

Tools/Scripts improvements
Minor GUI tweaks

Note:
If you are upgrading your old script configurations will not be found, as the identifiers for them have changed. Simply double click on each script entry and reselect it to fix.

mouser · « **Reply #22 on:** December 24, 2011, 03:19 AM »

v1.08.01 - December 24, 2011

[Feature] Now packaging with a minimal python addon included

mouser · « **Reply #23 on:** December 30, 2011, 03:36 PM »

Web Link Captor download is now open to the public on the program web page listed above.
Let me know how you like it.

SFdude · « **Reply #24 on:** January 08, 2012, 07:28 PM »

Web Link Captor: great and useful!

Quick question:

Does WLC in any way, search and/or display
any of its web results
(for ex: in the "Web Preview Tab"),
using the MS Internet Explorer browser,
or any MS-IE tools?

I'm concerned about using the IE Browser, any version,
because of security concerns.
(I never use IE, only Firefox & Chrome in XP-SP3 f/browsing...that just me).

Thanks for your answer!
SFdude