Author Topic: NANY 2010 Final Release: FarrWebMetaSearch (Read 72944 times)

phitsc · « **on:** December 11, 2009, 09:51 AM »

NANY 2010 Entry Information

Application Name	FarrWebMetaSearch
Version	0.9.1
Short Description	A FARR plugin used to query results from a web page and display them in list form.
Supported OSes	MS Windows
Web Page	http://farrwms.objecttechnology.com
Download Link	download
System Requirements	FindAndRunRobot (FARR)
Version History	0.9.1 - preliminary - use FARR's launching functionality to open URLs 0.9.0 - preliminary - first release
Author	phitsc
Credits	Thanks to hamradio for designing the plugin icon!
Screencast	http://donationcoder...h/webmetasearch.html YouTube: http://www.youtube.c.../watch?v=pxKcvQ-uHjQ

Description
FarrWebMetaSearch is another generic web retrieval FARR plugin. Unlike ewemoas Webgrab this plugin will show results in list mode. It is therefore targeted at web sites delivering list-like results.

Configuration and usage of the plugin will be as follows:

searches are configured via a text configuration file, one for each 'search'. the file will specify search url and a regular expression to capture the result.
search terms will be specified through the FARR input box

See implementation notes further down for further clarification.

Features
It just does what it says in the description.

Planned Features

Fix icon copyright problem
Fix launching problem
Add proper help file
Add a way for users to contribute / get searches
Add logging/notification in case a configuration files is invalid
Improve usage for news feeds
Make original search string available as a variable
Add many more searches
~~Either make plugin asynchronious or implement some way to delay searching~~
~~Add a description for each search to xml files and FARR searches list~~
~~Implement filtering of searches~~
~~Implement auto-completion of searches with Enter key (well, maybe)~~
~~Add a way to specify search options (for web sites that support it)~~
~~Make parsing of searches files more robust~~

Screencast

Screenshots

	FarrWebMetaSearch	Web Site
MSDN	NANY 2010 Final Release: FarrWebMetaSearch	NANY 2010 Final Release: FarrWebMetaSearch
CDDB	NANY 2010 Final Release: FarrWebMetaSearch	NANY 2010 Final Release: FarrWebMetaSearch

and a teaser screencast.

Implementation Notes

FarrWebMetaSearch reads all configuration files in its searches subdirectory when it's being loaded. When the plugin alias (currently wms) is entered, it will list all configured searches. To use a search one would then enter, e.g.:

wms msdn PathAppend

to find MSDN entries for the PathAppend API function.

Search files currently look like this (this is the cddb.conf file):
description=Microsoft Developer Network searchUrl=http://social.msdn.microsoft.com/Search/en-GB?query= resultPattern=<a onmousedown=\".*\" href=\".*\">(.*)</a>\r\n<br/>\r\n<div class=\"ResultDescription\">(.*)</div>\r\n<div class=\"ResultUrl\">(.*)</div> farrCaption=$1 farrGroup=$2 farrPath=$3
The regex sub-expressions can (should, actually) be used to form the results.

mouser · « **Reply #1 on:** December 11, 2009, 10:50 AM »

Sounds wonderful!
Reminds me of this FARR plugin as well which deserves more attention and users:
https://www.donation...ex.php?topic=16006.0

phitsc · « **Reply #2 on:** December 13, 2009, 03:45 PM »

I've added some screen shots.

mouser · « **Reply #3 on:** December 13, 2009, 03:51 PM »

whoa now this is looking good.

phitsc · « **Reply #4 on:** December 13, 2009, 04:52 PM »

updated implementation notes and added a todo list

mouser · « **Reply #5 on:** December 13, 2009, 07:06 PM »

it doesnt show results live as we type does it?

phitsc · « **Reply #6 on:** December 14, 2009, 02:42 AM »

it doesnt show results live as we type does it?
-mouser (December 13, 2009, 07:06 PM)

It's meant to. But fetching something from the web obviously takes time. So currently the plugin is best suited to touch typists

mouser · « **Reply #7 on:** December 14, 2009, 02:56 AM »

NICE

CWuestefeld · « **Reply #8 on:** December 15, 2009, 11:30 AM »

Cool.

Request: put in a search file for IMDB.

phitsc · « **Reply #9 on:** December 15, 2009, 03:27 PM »

I've implemented the most important features now, especially asynchronious searching. I'm going to try and add some more searches now with focus on stability. I'm going to start with IMDB then.

mouser · « **Reply #10 on:** December 15, 2009, 04:06 PM »

imdb live search is already provided in google plus addon by the way, not that it's a bad idea to add it.
now adding a live search of amazon and amazon books.. DROOOOOOL......

i'd give my left arm for that.

TucknDar · « **Reply #11 on:** December 15, 2009, 04:14 PM »

Really looking forward to this one, looks incredibly useful!

Amazon, play.com, eBay, yellow pages...

mouser · « **Reply #12 on:** December 15, 2009, 05:02 PM »

ebay and amazon, that's like the holy grail of instant search

phitsc · « **Reply #13 on:** December 16, 2009, 02:49 AM »

Amazon will be next on my list. I've ordered myself a Kindle for Christmas

phitsc · « **Reply #14 on:** December 17, 2009, 02:10 PM »

I've changed the configuration files from xml to ini-style (see implementation notes in first post). Makes it much easier to handle the (already complex) regular expressions, as you don't have to encode < and > etc.. And then xml would be overkill for this anyway.

Capturing IMDB's output turned out to be a hard nut to crack. I'm afraid I can't offer anything satisfactory.

mouser · « **Reply #15 on:** December 17, 2009, 03:07 PM »

live search of some of the dictionary website would be awesome as well, while we are building a wishlist

(since imdb is already supported as live search in google plus addon for farr it's not high on my priority list)

Perry Mowbray · « **Reply #16 on:** December 17, 2009, 07:02 PM »

Dictonary sites would have to be my most used

TucknDar · « **Reply #17 on:** December 18, 2009, 01:51 AM »

Dictonary sites would have to be my most used
-Perry Mowbray (December 17, 2009, 07:02 PM)

Actually, you missed an 'i' there, it's 'Dictionary'

Perry Mowbray · « **Reply #18 on:** December 18, 2009, 01:59 AM »

Dictonary sites would have to be my most used
-Perry Mowbray (December 17, 2009, 07:02 PM)
Actually, you missed an 'i' there, it's 'Dictionary'

-TucknDar (December 18, 2009, 01:51 AM)

Cool: you know I staired at it for ages but I couldn't see it!! I often don't have an eye for detal...

TucknDar · « **Reply #19 on:** December 18, 2009, 02:02 AM »

Dictonary sites would have to be my most used
-Perry Mowbray (December 17, 2009, 07:02 PM)
Actually, you missed an 'i' there, it's 'Dictionary'

-TucknDar (December 18, 2009, 01:51 AM)

Cool: you know I staired at it for ages but I couldn't see it!! I often don't have an eye for detal...
-Perry Mowbray (December 18, 2009, 01:59 AM)

cranioscopical · « **Reply #20 on:** December 18, 2009, 10:35 AM »

I staired at it for ages but I couldn't see it!!
-Perry Mowbray (December 18, 2009, 01:59 AM)

Tread softly or someone will try to take a riser out of you

phitsc · « **Reply #21 on:** December 21, 2009, 02:14 PM »

I've recorded a short teaser screen cast.

TucknDar · « **Reply #22 on:** December 21, 2009, 02:26 PM »

I've recorded a short teaser screen cast.
-phitsc (December 21, 2009, 02:14 PM)

mouser · « **Reply #23 on:** December 21, 2009, 02:46 PM »

nice!

phitsc · « **Reply #24 on:** December 23, 2009, 08:19 AM »

just realised FarrWebMetaSearch could be (mis)used to read news feeds. because they are xml they are actually simpler to split up with regular expressions than 'normal' web pages.

here's an example:

description=Tages Anzeiger Front searchUrl=http://www.tagesanzeiger.ch/rss.html? resultPattern=<item>\s*<id>\d*</id>\s*<title>(.*)</title>\s*<description><!\[CDATA\[(.*)\]\]></description>\s*<category>.*</category>\s*<link>(.*)</link>\s*<guid.*</guid>\s*<pubDate>(.*)</pubDate>\s*</item> farrCaption=$1 ($4) farrGroup=$2 farrPath=$3

will give you this:

NANY 2010 Final Release: FarrWebMetaSearch