I Need, I Pay

[maloner]

Here's the deal, Ladies and Gentlemen.

I would like to commission someone to create an internet search "delegator" - a search spider that can interact and command the other search engines of the WWW (Google, Yahoo, etc, etc) to search all databases in the entire WWW using keywords input to the delegator, and come up with an "answer".

To better explain, though it's difficult:-

You write a program that orders other people's search engines to do our search work (whatever that work is). Most internet databases have a search function built in - libraries, museums, universities, governments. So you write a spider that finds their search engines, inputs your keywords, and asks them to search. This way, we harness the power of thousands of engines, working in unison, for our purposes. Kind of like parallel processing, or a kind of metasystem.

I appreciate that coding a search spider may be a menial task, depending on one's expertise and experience. But, like everything else, it's only easy if you know how to do it !

So, I'd be quite willing to pay someone to do this for me.

I guess the questions are.....who will do it ? How do I determine who to pay if more than one person does the job ? Should I run a competition and pay the winner ? Who will judge ? How much would be a reasonable payment ?

Your thoughts and ideas would be appreciated.

[Veign]

1) It will be very expensive to do what you want
2) If the site takes off your connections with the other search engines will begin to fail.  Other search engines have strict rules against scraping their data and the correct channels to use their data (through an API) have daily limits.

So you need to ask yourself why you would be doing this.  If its just for your personal use then are you willing to pay the hefty cost for the project, if its for the 'public' then understand it will fail at some point and there will be nothing for you to do to get it to work.

[app103]

Try Copernic Agent.

[parkint]

To locate someone, using a competition, go to Sitepoint Marketplace or Rent-A-Coder.

Best of luck.

[Veign]

Try Copernic Agent.

-app103 (July 16, 2010, 11:16 AM)

That will work for personal use but not if this will be a public website - Google, and others, will block if they feel there data is being scraped for redistribution on a website.

[Veign]

To locate someone, using a competition, go to Sitepoint Marketplace or Rent-A-Coder.

Best of luck.

-parkint (July 16, 2010, 12:27 PM)

You'll find many that will take the job (probably under $500) but it will be a complete waste of money (for a public website).

There are lots of dishonest developers out there who will happily take your money for projects that legally can't be done, or can't be done because of limitation imposed by supporting companies (the search engines in this case).

[daddydave]

This is called a metasearch engine. There was kind of a golden age of metasearches, maybe the 80's or 90's, and an actual site metasearch.com, although if you go there now, it just redirects your search terms to Google. So think about who is the Google of metasearch nowadays, and I think the answer is harder to answer today than it has ever been. Surfwax tries to do this but the results don't cut it for me. Perhaps metasearch fizzled out because most people figured out it's more efficient to figure out which search engine is best, and just use that instead of collecting all those results from different search engines. But that is just one man's opinion.

On the other hand, if you're offering the equivalent of the X Prize to create the Google of Metasearch, for that much money someone, or more likely a team of someones, might come up with something truly excellent.

But what I really think is that a lot of people tried already, and that the results you get back are not going to be as useful as you think they are, unless you are capable of recruiting Google-class talent to tackle the problem.

[maloner]

Thanks for all your informed comments to date, Gents.
Just to add further....this project is not for public consumption; it is for my own personal and private use.
The search results would be for my own research purposes only.

[wetsmellydog]

Actually it is very easy to do yourself without a lot of coding.

A quick search of "metasearch engines" turned up this gem Pandia Powersearch. It states "Pandia Powersearch is our all-in-one list of search engines and directories. Search the Web using the search form above, or select one of the categories below to find the best Internet search tools."

It also has this blurb;

List of search engines and directories

The databases used by search engines are made by "robots" or "spiders" that automatically map the Web by following the links between sites. The directories, on the other hand, contain sites that are hand picked by human beings. Still, most search sites present search engine as well as directory results. Metasearch engines gather results from several search engines and directories. They are not that good at advanced Boolean searching, but may provide you with a broad sweep of the Web.

Traditional search engines list normal web pages and some other file types that are interlinked to from such webpages. The database sites, however, give you access to data that are not part of the Web as such. See also sites focusing on the hidden Web below.

So if you educate yourself on some of the hidden features of search engine commands like Google's commands found here http://www.seowarrio...et/google-search-co/ (just one of many sites with the info), you can find almost anything without the crap!

Lots of luck! 

[Veign]

Pandia Powersearch uses Google's results, by default, with lots of ads.  Its a junk website that doesn't do what the OP asked.

[maloner]

Thanks, wetsmellydog. I will invest some time in following up what you say.
Appreciate your advice as well as the input of everyone else in this thread.
All comments are acknowledged and valued.

[wetsmellydog]

Pandia Powersearch uses Google's results, by default, with lots of ads.  Its a junk website that doesn't do what the OP asked.

-Veign (July 17, 2010, 01:56 PM)

I use Firefox with adblock so I do not get any advertising. You can change the default search engine or use it as a metasearch engine which is what the requester was asking about.
There is even a tutorial on the site on how to make best use of search engines.

A tool is only as good as the operator.

BTW, it was the first site that came up while doing a cursory search and looked interesting. I mentioned that further investigation would be prudent to find one that was a good fit.
I was also suggesting that the tools were possibly available to do what was needed without having to pay for it or at least have an educated idea on what you would be paying for.

[MilesAhead]

You can get some of this effect using Firefox Search Sidebar, but you'd need to come up with a mechanism to glean info from the result tabs.