ATTENTION: You are viewing a page formatted for mobile devices; to view the full web page, click HERE.

Main Area and Open Discussion > General Software Discussion

Idea request: "Site discovery tool"

<< < (2/3) > >>

Sometime back, I saw this site on Searchbots ( The idea is that you list some topics and build a searchbot which goes off and searches around for content. Over time, it gathers more content and then aggregates them. Every day, it returns fresh content.

Pubsub ( was another search engine that could return results based on current recently indexed news content. For finding those high quality, yet hard to locate sites, I suggest directories over search engines. Complete Planet ( has a listing of several specialist directories on the internet.

I wrote an article about searching the internet about 2 years ago, but I failed to maintain my site and it was taken down. You can find its last version on the WaybackMachine :

Some of the stuff is a bit outdated, but much of it can still be used.

Thanks icekin, I can see you spent a lot of time thinking about these things  :Thmbsup:

Have you heard of StumbleUpon??? That sounds like exactly what you're looking for...

Google alerts certainly cannot do the job - they illustrate everything that is wrong about google's approach when you go for more topical/targeted information.

I have heard of stumble upon, but... I want the bits that arent popular enough in topics "light" enough to make it through filters like stumble upon... we're talking professional topics, and not in web dev or something popular like that, and stumble upon stumbles a bit there...

Or in a way I am looking for a way to automate what we are doing now, which equates to trawling numerous blogs and forums, subscribe to newsfeeds, blogrools, press release services, do regular deep searches, watch 100 companies... to find tidbits of information, pages, white papers etc. which are in the dark/deep web. The opposite of what digg/technorati/stumble upon do, which tries to fload some things to the surface because they are fun/cool/intriguing to a large number of people.

Some bits I know how to improve, using watchers, feed readers, search tools - but especially when it comes to the search tool I would like something more clever than the usual search. Something that remembers what i have already found before and doesnt show it again unless it has changed a lot. Something that can blacklist things and ignore things and make it easier for me to see stuff that i havent seen before...

I guess it's a "i wish i still had time to code" moment :S

You might want to give Copernic Agent a try. Professional edition uses multiple search engines to find results, specialized search engines based on the type of info you are looking for (you can even do patent and ftp searches with it), tracks results, etc. This was THE way of finding info back when there wasn't a Google to use.

You could combine it with something like Newzie, which has features for alerting for certain keywords in rss feeds, which it can do at every feed update. It can also act as a site watcher, notifying you of not only changes on a site, but I believe it can notify of changes that trigger the same keywords alerts you have set up for feeds. (Newzie is pretty powerful!)


[0] Message Index

[#] Next page

[*] Previous page

Go to full version