FARR plugin: FarrWebMetaSearch 1.6.0

[phitsc]

Description
FarrWebMetaSearch is a generic web retrieval FARR plugin that show results in list mode. It is therefore targeted at web sites delivering list-like results. Results are selected using regular expressions. FarrWebMetaSearch searches live as you type!



Screencast
http://donationcoder...h/webmetasearch.html
YouTube: http://www.youtube.c.../watch?v=pxKcvQ-uHjQ

Download & Install
Download the FarrWebMetaSearch installer from here then double-click FarrWebMetaSearchSetup.exe to start the installation. I've tested the installer on both Windows XP and Windows 7 32-bit. If you'd rather install the plugin manually, you can download it from here, then unzip it into the FARR plugins directory (usually C:\Program Files\FindAndRunRobot\Plugins) and restart FARR.

Basic usage
Type

wms [search] [+subsearch] [search term]

to use a pre-configured search.

Typing +, - or / after the search name will display available subsearches. [+subsearch] is optional. It currently has to follow right after the search name.

If [search] is configured as a web search, [search term] will be searched live as you type. If [search] is configured as a feed, [search term] will filter the found results. In this case [search term] is optional.

Tip: Type wms ? and hit enter to open the help file.

Searches
The following searches are currently available:

• Amazon
• Amazon.de
• CDDB
• C++ Reference
• Engadget (feed)
• Google
• IMDB
• Jamendo
• Kindleforum (feed)
• MSDN
• OneLook
• Play.com
• Tages Anzeiger Front (feed)
• Twitter

Version history
1.6.0 - 9. July 2010

• can now search local files (see help file)
• New / updated searches:
  
  • google (by Alien282)
  • fixed MSDN search

1.5.1 - 11. February 2010

• implemented status bar icons
• implemented two stage regex
• replacing many more character entity references. now supports all the French characters as well.
• New / updated searches:
  
  • amazon and amazonde (much improved!)
  • imdb (improved)

1.5.0 - 5. February 2010

• implemented sortOrder parameter
• implemented isHidden parameter
• implemented context submenus
• now supports more than 9 context menu items
• now clears context menus for subsearches
• subsearches / abbreviations are now case-insensitive
• now prints the number of listed searches in the status bar
• updated help file
• New / updated searches:
  
  • onelook (by Perry Mowbray)
  • timeanddate (by Perry Mowbray)
  • autohotkey (ewemoa)
  • cheatsheet
  • amazon (fixed typo)

1.4.2 - 29. January 2010

• Now filters for subsearch abbreviations too
• Added some logging capability. Type 'wms !' to enable / disable logging.
• Added %PLUGINALIAS% variable
• Updated help file
• New / updated searches:
  
  • onelook (by Perry Mowbray) - with more subsearches, abbreviations and help file!

1.4.1 - 26. January 2010

• subsearch filtering now matches at the start (no longer anywhere in the string)
• show abbreviation of subsearches (if they have one). e.g. wms amazon +books (b)
• Shift + Enter launches search result in FARR's internal HTML view
• replace more special characters in FARR output (  and °)
• fixed problem with subsearches having isFeed=true

1.4.0 - 24. January 2010

• Shift + Enter shows info about search (using FARR's internal HTML view)
• Subsearches can now have abbreviations ( [Subsearch|Abb] )
• Searches can now define context menu items (see play search for an example)
• Fixed 'feeds not showing subsearches on entering +'
• new / updated searches:
  
  • onelook (by Perry Mowbray) - added four new subsearches
  • jamendo (by Perry Mowbray)
  • cppreference - replaced with a different one providing better information
  • play - added 'buy item' and 'show basket' context menu items

Previous releases:

Spoiler

1.3.0 - 15. January 2010

• search term (%SEARCHTERM%) and search URL (%SEARCHURL%) are now available as variables in the resultPattern, farrPath, farrCaption and farrGroup parameters
• search term (%SEARCHTERM%) is now available as a variable in the searchUrl parameter. If searchUrl does not contain the %SEARCHTERM% variable, the search term is appended at the end (as it was until now)
• properly handle and report server errors
• show down-arrow icon on status bar to switch to FARR's extended view (which doesn't handle html tags properly though)
• new / updated searches:
  
  • onelook (by Perry Mowbray) - added name subsearch

1.2.0 - 14. January 2010

• Fixed bug with search parameters defined as empty string
• Updated help file Search config files
• new / updated searches:
  
  • onelook (by Perry Mowbray)
  • kindleforum (feed)
  • play.com

1.1.0 - 9. January 2010

• Type wms ? + Enter to show help file
• Added about list items
• Updated help file (still needs better treatment of .config files)

1.0.0 - 8. January 2010

• Changed search behaviour for news feeds
• Improved amazon.com search. Now gives more relevant results when searching for 'neural control' (and other titles with more than one author )
• replace more special characters in FARR output, e.g. © and ®
• resultPattern can now contain the %SEARCHTERM% variable, which will be replaced by the entered search terms
• added new searches:
  
  • onelook (thanks Perry Mowbray!)
  • engadget (feed)
  • twitter
• changed to TiddlyWiki help file format (although it's far from complete)

0.9.1 - 31. December 2009

• use FARR's launching functionality to open URLs

0.9.0 - 29. December 2009

• First release

Credits
Thanks!

• to Carroll (hamradio on donationcoder.com forums) for designing the icons
• to Perry Mowbray for his extremely cool and powerful onelook search (and for beta testing )
• to everyone who gave me feedback on the NANY 2010 event

<a href="https://www.donation...ex.php?topic=20951.0">
<img src="https://www.donation...0/nanyrib_2010_t.png">
</a><br />
<a href="https://www.donationcoder.com/nany2010/">NANY 2010</a>

[phitsc]

I have improved the help file somewhat. .config files still need a much better description though. To open the help file type

wms ? and hit enter

You'll notice that it's a TiddlyWiki file. While that obviously has quite some overhead I find it very convenient to edit.

Perry: %SEARCHTERM% in the resultPattern will now be replaced by the search terms entered
lanux128: I've added twitter. Please check it out and tell me what you think

I have now changed the behaviour for web feeds. As there is no live searching on feeds FarrWebMetaSearch treats them a bit differently. Instead of doing live searching, the search terms typed will now narrow down the results to match the search terms. Note that you'll have to wait until the results are displayed after typing the search name. You can try like this:

'wms engadget ' (currently needs the space), then wait for the results and type a search term

The .config file needs isFeed=true to distinguish it from regular searches.

[lanux128]

lanux128: I've added twitter. Please check it out and tell me what you think

-phitsc (January 09, 2010, 03:47 PM)

thanks phitsc, it works very well. one step closer to twitter nirvana.

FARR plugin: FarrWebMetaSearch 1.6.0

[Perry Mowbray]

I can't wait to try...

I had a quick bash at twitter, but mine needed more thought. With Twitter it'd be good to set a context menu as well, eg to poster, original, etc.

One other quick question: I wondered about a two stage RegEx. I'm working with a page that is very difficult to produce the results in one RegEx, and I wondered about the 1st Regex producing the input for the second?

Edit: I'm wondering if it'd be better to have this more open ended so that multiple RegEx's can be defined, each using the result of the previous?? IE Not just 2.

I'm in the process of changing the onelook search... I'll post a new one later  

PS: I love your badge!!!  

[Perry Mowbray]

I'm trying the %SEARCHTERM% functionality, and I'm not sure it's working 

When I use it I get zero results, when there should be results.

Any ideas?

[phitsc]

Concerning %SEARCHTERM% not working:
Although I haven't made a search using it, I've tested it in the debugger and it worked as expected. Nevertheless, I'll make you a version that prints out the regex used in DebugView so you can check if it's replaced as you would expect.

Concerning your multi-stage regex idea:
I'll implement that so you can check if it helps with this difficult to parse page.

Concerning the context menu:
A very good idea!

[phitsc]

Perry, please download a special version of FarrWebMetaSearch from here. The only difference to the current official release is that it will print out the current search term and the regex used in the debug output which you can view with the DebugView tool.

[Perry Mowbray]

Cool, I'll have a go when I get home.

Concerning %SEACHTERM%, the only reason that I'm confused is when the seach was (%SEACHTERM%) you'd expect to get some matches, but I'm not 

[Perry Mowbray]

Perry, please download a special version of FarrWebMetaSearch from here. The only difference to the current official release is that it will print out the current search term and the regex used in the debug output which you can view with the DebugView tool.

-phitsc (January 12, 2010, 02:13 PM)

Couldn't wait till I got home: this one works fine! 

Here is an update to the onelook search: onelook.zip (0.68 kB - downloaded 531 times.)

I think you are not clearing the farrGroup definition when loading a subSearch though? If you check this updated subSearch you'll notice that +Phrase and +Reverse both keep the original farrGroup definition.

Also, do you think it would be helpful to be able to add the subsearch switch anywhere? In the onelook search for example I wouldn't mind being able to change from Phrase to quick then to dictionary...

[phitsc]

Here is an update to the onelook search:  (see attachment in previous post)

-Perry Mowbray (January 12, 2010, 04:21 PM)

Thanks. Will be in the next release!

I think you are not clearing the farrGroup definition when loading a subSearch though? If you check this updated subSearch you'll notice that +Phrase and +Reverse both keep the original farrGroup definition.

-Perry Mowbray (January 12, 2010, 04:21 PM)

Thanks for reporting this one. Will be fixed in the next release.

Also, do you think it would be helpful to be able to add the subsearch switch anywhere? In the onelook search for example I wouldn't mind being able to change from Phrase to quick then to dictionary...

-Perry Mowbray (January 12, 2010, 04:21 PM)

Indeed I think it would. And I have put it on my todo list. The advantage of having the switch at the front is that I don't have to start searching until actual search terms have been entered. If I allow the switch to be anywhere, adding/changing the switch either triggers a research for every character entered, or I have to detect while typing that the thing being entered is actually a switch and not part of the search term. But I'll definitely implement this somehow.

[phitsc]

New version fixes the bug reported by Perry, updates Perry's onelook search and adds play.com search and kindleforum feed.

[Perry Mowbray]

Looks great! 

The only thing I'm (and maybe others) is missing (apart from multiple stage RegEx searches) is the ability to add the farrPath from somewhere other than the RegEx match. For example, either allow the substitution of %SEACHTERM% so it can be constructed in the search definition, or have something like %SEARCHURL% which would be the url that WMS uses to get the data.

Also: Added a Name subSearch to OneLook -- onelook.zip (0.75 kB - downloaded 479 times.)

[Perry Mowbray]

Did I say "only"?  

I'd also like to be able to construct the searchUrl, not just add the SearchTerm to the end of the searchUrl.

I'd be happy to be able to do something like:

searchUrl=http://www.somewhere.com/?q=%SEARCHTERM%&t=quick&p=slow

Maybe with the functionality that if no %SEARCHTERM% is present current placement (at the end) is used?

I was just about to also ask about descriptions for each subSearch: but you've already done that  

Also: it'd be much better if you can switch to small icon view when there are greater than 10 matches, or if mouser can override the Plugin with FARR setting??

[phitsc]

New version adds some of your requests (well, multi stage regex searches not yet ). I haven't tested the variable replacements very well. So if something doesn't work as you'd expect please contact me before searching for an error at your end too long.

I've added the possibility to switch to FARR's extended list mode (with the small icons). If a search returns more items than what you've set as max. items in normal mode a small icon is shown in the status bar (on the far right). Clicking it will switch to extended mode. You'll note that this mode doesn't handle html tags. It would be nice if it would at least just discard them. You might have requested that the plugin switches to that mode automatically when there are more than 'max. items in normal mode' items. I don't like that though, so I'd have to make it an option (but there's no way to configure any options yet, so I'd have to add that too).

[Perry Mowbray]

No, everything looks like it's working fine.

Here's an update for onelook: onelook.zip (2.31 kB - downloaded 498 times.)

So Onelook contains:

• Quick Definition (Default)
• Reverse - return words based on a concept. Use <pattern>: words where pattern can contain ? and * wildcards
• New: general phrase search - returns phrases that contain the word
• New: common phrase search - returns only common words/phrases
• New: match all - returns word / phrase matches of a pattern (use ? and * as wildcards)
• New: match common - returns common word / phrase matches of a pattern(use ? and * as wildcards)
• Dicts - Returns links to dictionaries that contain the word
• Name - Returns name information about the word

I'm also playing with OneLookType onelooktype.zip (2.51 kB - downloaded 518 times.):
OneLook has the ability to look within certain dictionary types. At the moment the results are only sorted with the requested dictionary matches on top; once two-stage RegEx parsing is enabled we can make this a little more clear. The dictionary types that can be searched are:

• Art
• Business
• Computing
• Medicine
• Miscellaneous
• Religion
• Science
• Slang
• Sports
• Technology

I'd love to have some quick way of navigating between one search, say looking for a word and then searching for the quick definition. Originally I was thinking of being able to have the switches at the end, but maybe there's a better way?

[phitsc]

Very cool Perry! Thanks for that.

I've started to implement the possibility to show basic info about searches. Pressing Shift + Enter on a search without search terms will show info about the respective search in FARR's internal HTML view mode.

Concerning a quick way of navigating between subsearches: making it possible to add a switch at the end would certainly be an option. It has two drawbacks though: it will trigger a re-search (which I might be able to prevent) and it won't be possible to show a list of subsearches for the user to select. However, I had these two ideas to help with the problem:
1. add a statusbar icon for each subsearch. pressing an icon would switch to the respective subsearch with the current search terms.
2. introduce subsearch abbreviations so that a user doesn't have to type the whole subsearch name (because there's no auto completion when used at the end instead of at the beginning)

what do you think?

[Perry Mowbray]

If we start by thinking what would work for this scenario that then refine so that it's useful for all??

For this type of search what would be nice is to re-use part of the result for different subSearches.

So if I had a reverse search on "Play with words" I get a hundred results+ (words). What would be neat is to be able to get that result ($1 from the RegEx) and use that as the search term for a new search. That way if I wasn't sure about the meaning of a word I could select it and use it to do a new search.

This was why I was wondering about a context menu definition: you could add text for menu and the subSearch to use

• "Definition of $1" >> [Quick]
• "Dictionary listings with $1" >> [Dicts]
• "Phrases with $1" >> [Prase]
• etc

Is search history a possibility? I'm wondering about navigating backward from the definition lookup mentioned before. That would make good status bar icons.

subSearch abbreviations is a good idea, you'll notice I used [Dicts], but +Di would be even better 

 I'm not sure how that fits with the whole plugin??

[phitsc]

If we start by thinking what would work for this scenario that then refine so that it's useful for all??

For this type of search what would be nice is to re-use part of the result for different subSearches.

So if I had a reverse search on "Play with words" I get a hundred results+ (words). What would be neat is to be able to get that result ($1 from the RegEx) and use that as the search term for a new search. That way if I wasn't sure about the meaning of a word I could select it and use it to do a new search.

This was why I was wondering about a context menu definition: you could add text for menu and the subSearch to use

• "Definition of $1" >> [Quick]
• "Dictionary listings with $1" >> [Dicts]
• "Phrases with $1" >> [Prase]
• etc

-Perry Mowbray (January 18, 2010, 06:03 PM)

Ok, I'll try to do that next.

Is search history a possibility? I'm wondering about navigating backward from the definition lookup mentioned before. That would make good status bar icons.

-Perry Mowbray (January 18, 2010, 06:03 PM)

Can you please elaborate?

subSearch abbreviations is a good idea, you'll notice I used [Dicts], but +Di would be even better 

-Perry Mowbray (January 18, 2010, 06:03 PM)

I've implemented abbreviations. They will be specified like so: [Dicts|Di]

[Perry Mowbray]

History: I mean something like this scenario:
1. Reverse look up -> list of words
2. Quick search a word from the result list -> list of definitions
3. Not the definition I want, Navigate back 1 in history -> #1 list of words
4. Quick search a different word -> list of definitions
5. Happy with definition, get Dictionary List for word -> dictionary listing
6. Not happy with dictionary list, Navigate back 2 in history -> #3 list of words

Abbreviations: will they be either/or?

subSearch switching: Just a note -- I've just realised that I used $1 in each of the examples... it may be $1 for one subSearch and $3 for another: if you get my drift 

[phitsc]

Abbreviations: either/or, yes.

[Perry Mowbray]

While we're thinking future things  I was playing with a RegEx from Lannux and could have used a RegEx Replace function to remove some text first... but that would have to come after multi-stage RegEx I'd think?

[Perry Mowbray]

Here's a search for Jamendo:


jamendo.zip (7.6 kB - downloaded 539 times.)

What is Jamendo?
Jamendo is a community of free, legal and unlimited music published under Creative Commons licenses.
Share your music, download your favorite artists!

wms jamendo (with no search string) will display a list of albums in popularity rank:
FARR plugin: FarrWebMetaSearch 1.6.0

Note:

• FARR doesn't seem to be parsing the HTML Character codes correctly (hence the &nbsp;)
• WebMetaSearch has a bit of a functional anomaly: It does not display the subSearches when you type "+" when the search is defined as a Feed. Therefore you'll need to remember the subsearches:
  
  • +Album: Search by album name
  • +Artist: Search by artist name
  • +Member: Search by member name
  • +Tags: Search by tag
  • +Playlist: Search by playlist name

If your native language is not English you should probably change the "/en" to an appropriate language  (although I've had no reason to test this)

[Perry Mowbray]

I've started to implement the possibility to show basic info about searches. Pressing Shift + Enter on a search without search terms will show info about the respective search in FARR's internal HTML view mode.

-phitsc (January 18, 2010, 04:24 PM)

Is this working? I can't seem to get it to trigger on my machine??

Here's a new onelook: onelook.zip (0.89 kB - downloaded 526 times.)

This adds the expand functionality where onelook will return phrases that spell out the search term. Onelook suggests "nasa":

[phitsc]

Check out the new release: there's subsearch abbreviations, context menu support, custom search help file support, Shift + Enter to launch result in FARR's internal web browser, new and updated searches and more...

And then there's Perry's OneLook search. What can I say: it's just amazing!

Actually, I think wms onelook has become my main use of FARR.

[Perry Mowbray]

Here is a new version of OneLook: onelook.zip (6.03 kB - downloaded 497 times.)

this one tweaks some of the subSearches:

• I changed match to patternMatch so there's no mistaking the requirement of a pattern (it'll fail if it just looks up a word)
• I've added Context Menus that make it easier to look up definitions of a word that you've searched on, and back again (handy if you're like me and you dig around a concept for the right word)
• Tweaked patternMatch so that concepts can be added after the pattern, in the <pattern>: <description of concept> format.
• Added a new Crossword subSearch: like patternMatch but more restrictive on what is returned.
• Added OneLook's Word of the Day: This lists the 5 top words matched today and then the 5 top words matched the last hour.

I've also added a Copy menu item which should make it easier to copy the results back into another application.

And here is a first cut at a Time and Weather search from TimeAndDate.com: timeanddate.zip (6.69 kB - downloaded 524 times.)

There's no help file yet, so just quickly:

• wms timeanddate will return the list of current times in Country - State - City format. Entering a search term will filter the results.
  
  • Sun RMB Context Menu: Will load information about Sun times for the selected location.
• wms timeanddate +w will return the current weather conditions of various cities around the world. Entering a search term will filter the results.
  
  • Forecast RMB Context Menu: Will load the 14 day forecast for this location in the following format:
    <date>: <conditions>
    <low temp> :: <high temp>
  • Current Conditions RMB Context Menu: Will load the following detailed informaton for the selected location:
    
    • Temperature
    • Comfort Level
    • Barometer
    • Humidity
    • Visibility
    • Wind

Caution:
There are some subSearches in this WMS TimeAndDate search that will crash FARR (+current, +forecast, +sun) -- they really need to be hidden: hence the next section 

Requested New Functionality to WMS 

• subSearch sorting: I've added a parameter to the subSearches though it can be named anything. OneLook suffers from not having the ability to sort the subSearches.
• subSearch hiding: Some of the subSearches in TimeAndDate should only be used from the context menu as typing the search criteria in doesn't make sense (and crashes FARR at the moment)
• Context Menu:
  
  • I seem to have hit the limit with the context menu. Is that a WMS restriction or FARR?
  • The context menus don't seem to be being cleared for each subSearch. You can see this in the OneLook search: the submenus for the default search are carried over to all of the subSearches. I think it would be preferable to start each subSearch fresh.
  • Can we have subMenus? I'd like to put all the dictionary type searches into a subMenu (Art, Computer, etc)
• Two-stage RegEx: Those dictionary type searches could really do with a two-stage RegEx Match where the second RegEx operates on the result of the first. I also wondered if a RegEx Replace was possible (would help cleaning up some pages).
• Search Navigation: It would be helpful to have an ability to backwards and forwards in the search results. Sometimes you can not get back to where you came from with the search results, and a back button would be helpful. Something like this:
  
  • wms timeanddate +w: cities displayed
  • type "syd": filter to "Sydney"
  • select forecast context menu: 14 day forecast displayed
  • at this point I would like to go back one step to the cities filtered on "syd" to access the current conditions context menu.