Desktop Forum Reader

[kurtisnelson]

I am wondering if anyone knows of a program that will in some form crawl a forum and download the posts kind of like a newsgroup client. I frequent many forums, but get frustrated at how long each individual page load takes and the inefficiency of browsing them in a browser. Does such a program exist? If not, does any one have an idea on how to create one?

[KenR]

I don't know if it's what you're looking for, but you might take a look at http://www.newzcrawler.com/.

Ken

[kurtisnelson]

More what I mean is something that will crawl a forum and browse it similar to a newsgroup, text based and all local files. Something that would work well on a slow connection and could connect to the internet, update, then disconnect to read. Every page view on a forum for me tends to involve having to redownload headers and images over and over.

[tranglos]

Every page view on a forum for me tends to involve having to redownload headers and images over and over.

-kurtisnelson (February 09, 2007, 12:47 PM)

Images shouldn't be redownloaded, since the browser would normally pull them from the cache. Or you could disable downloading images, though it may make some sites unnavigable.

A generic application like you describe is not really possible, I think, since each forum has a different layout and slightly different functionality. There is no common "protocol" to follow. A minor change in the layout of a forum would require updating the application. Also, if a forum depends on JavaScript, for example, that application would have to have JavaScript built in, etc. It would have to support authentication (cookies), encrypted connections... In effect, it would almost have to be a fully-featured web browser, with only a different interface and the added storage functionality.

What you may want to try, instead, is a program that will download a whole website (or parts of it) to disk. Years ago I used a program called Teleport Pro for this, but there are probably newer offerings available now. Reget Deluxe can also download websites recursively, and I know there's at least one freeware app, but can't recall its name. This is worth a try, although this solution works best with static websites, where pages don't change between views. With dynamically generated sites, such as discussion forums, recursive download may take a long time (or, if URLs change dynamically, may never stop, because the downloader keeps seeing "new" links every time it grabs a page - I've seen it happen often with Teleport Pro).

Also, without cookies, you'll be downloading content as seen by a user who is not logged-in, so no Reply links, for example.

[urlwolf]

hmm maybe what you want is website watcher by aignes:
http://www.aignes.com/

I think it's a nice tool. I'd buy it, but I'm waiting for a DC discount . A bit obsolete since most sites can be read with an RSS reader. Still, nice functionality to track changes of static (or slowly updated) pages.

Another solution: subscribe to the RSS feed, if the forum has one. Then use a reader. I like greatnews. This forum for example has plenty of RSS functionality. THat is probably the best solution.

[kurtisnelson]

What if there is some kind of a server side plugin that allows a newsreader to connect? See, the solution of using a generic crawler does not help much, as I primarily care about not having to load continuously all the nav stuff and headers and footers. Are there any AJAXy forum systems that would load the content in the background allowing quick nav? RSS feeds tend to not contain all the threading type layout making them hard to read in mass.

[tranglos]

I don't know if there's anything server side, but there may be something helpful on the browser side. For Firefox, there is an extension called Fasterfox, which loads all links in the background. The improvement is perceptible on simpler pages, like Google's search results, but on large pages with lots of content and links, the chance Fasterfox preloads the link you're eventually going to click is rather small. It would help if you could configure the links you want preloaded, e.g. grab only links containing words such as "next", "page", or "more" etc., but Fasterfox doesn't support that.

A long time ago I was using a http proxy called Naviscope, which had exactly that feature, and it speeded up page loading very nicely. (Its basic feature was scrubbing ads.) It was freeware and it performed flawlessly, but it got abandoned after a while, and I had to drop it when it became necessary to run an actual firewall. Perhaps there are other proxies now that have the preload feature.

[2stepsback]

What if there is some kind of a server side plugin that allows a newsreader to connect?

-kurtisnelson (February 09, 2007, 09:22 PM)

Sounds to me like a feature request for a plugin for all popular forum software out there

Or, as you said, something along the lines of Flock or LiveWriter or the Mozilla Amazon Browser - this last one (MAB) does exactly that with Amazon.com product listings - which are pretty much as dynamic as dynamic can get - it's a desktop program since it is a mozilla extension with an XUL-based GUI of its own.

Someone good at making Firefox extensions can see the source of MAB and build this extension for PHPBB, PunBB, SMF, IPB. Those IMO, are the popular (90%) forum applications.

Still one step ahead, a generic scriptable XML-based (easiest to "script") forum reader on the lines on WinHTTrack script.

I don't have the time or skill (XUL/xpi) else I would have thought seriously of writing this kind of thing. I'm thoroughly convinced of the usefullness of the idea.

Meanwhile as a simple workaround for bandwidth-handling, use Opera browser. It is sleek, fast, completely configurable as far as downloading images / Javascript / Flash goes. And it caches practically *everything*. So much that I've often been shown a page that's a couple of days old because I configured it to "never check latest version" of the pages. Consequently, I used to wonder what happened to the activity of the site. Till I realised that I had to press F5

-2stepsback