Home | Blog | Software | Reviews and Features | Forum | Help | Donate | About us
topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • December 08, 2016, 10:20:09 AM
  • Proudly celebrating 10 years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Desktop Forum Reader  (Read 4130 times)

kurtisnelson

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4
    • View Profile
    • Donate to Member
Desktop Forum Reader
« on: February 08, 2007, 08:22:51 PM »
I am wondering if anyone knows of a program that will in some form crawl a forum and download the posts kind of like a newsgroup client. I frequent many forums, but get frustrated at how long each individual page load takes and the inefficiency of browsing them in a browser. Does such a program exist? If not, does any one have an idea on how to create one?

KenR

  • Super
  • Blogger
  • Joined in 2006
  • ***
  • Posts: 826
    • View Profile
    • Donate to Member
Re: Desktop Forum Reader
« Reply #1 on: February 09, 2007, 08:04:03 AM »
I don't know if it's what you're looking for, but you might take a look at http://www.newzcrawler.com/.

Ken
Kenneth P. Reeder, Ph.D.
Clinical Psychologist
Jacksonville, North Carolina  28546

kurtisnelson

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4
    • View Profile
    • Donate to Member
Re: Desktop Forum Reader
« Reply #2 on: February 09, 2007, 12:47:32 PM »
More what I mean is something that will crawl a forum and browse it similar to a newsgroup, text based and all local files. Something that would work well on a slow connection and could connect to the internet, update, then disconnect to read. Every page view on a forum for me tends to involve having to redownload headers and images over and over.

tranglos

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 1,079
    • View Profile
    • Donate to Member
Re: Desktop Forum Reader
« Reply #3 on: February 09, 2007, 07:30:21 PM »
Every page view on a forum for me tends to involve having to redownload headers and images over and over.

Images shouldn't be redownloaded, since the browser would normally pull them from the cache. Or you could disable downloading images, though it may make some sites unnavigable.

A generic application like you describe is not really possible, I think, since each forum has a different layout and slightly different functionality. There is no common "protocol" to follow. A minor change in the layout of a forum would require updating the application. Also, if a forum depends on JavaScript, for example, that application would have to have JavaScript built in, etc. It would have to support authentication (cookies), encrypted connections... In effect, it would almost have to be a fully-featured web browser, with only a different interface and the added storage functionality.

What you may want to try, instead, is a program that will download a whole website (or parts of it) to disk. Years ago I used a program called Teleport Pro for this, but there are probably newer offerings available now. Reget Deluxe can also download websites recursively, and I know there's at least one freeware app, but can't recall its name. This is worth a try, although this solution works best with static websites, where pages don't change between views. With dynamically generated sites, such as discussion forums, recursive download may take a long time (or, if URLs change dynamically, may never stop, because the downloader keeps seeing "new" links every time it grabs a page - I've seen it happen often with Teleport Pro).

Also, without cookies, you'll be downloading content as seen by a user who is not logged-in, so no Reply links, for example.


urlwolf

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 1,797
    • View Profile
    • Donate to Member
Re: Desktop Forum Reader
« Reply #4 on: February 09, 2007, 08:38:44 PM »
hmm maybe what you want is website watcher by aignes:
http://www.aignes.com/

I think it's a nice tool. I'd buy it, but I'm waiting for a DC discount :). A bit obsolete since most sites can be read with an RSS reader. Still, nice functionality to track changes of static (or slowly updated) pages.

Another solution: subscribe to the RSS feed, if the forum has one. Then use a reader. I like greatnews. This forum for example has plenty of RSS functionality. THat is probably the best solution.

kurtisnelson

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4
    • View Profile
    • Donate to Member
Re: Desktop Forum Reader
« Reply #5 on: February 09, 2007, 09:22:51 PM »
What if there is some kind of a server side plugin that allows a newsreader to connect? See, the solution of using a generic crawler does not help much, as I primarily care about not having to load continuously all the nav stuff and headers and footers. Are there any AJAXy forum systems that would load the content in the background allowing quick nav? RSS feeds tend to not contain all the threading type layout making them hard to read in mass.

tranglos

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 1,079
    • View Profile
    • Donate to Member
Re: Desktop Forum Reader
« Reply #6 on: February 10, 2007, 08:24:11 AM »
I don't know if there's anything server side, but there may be something helpful on the browser side. For Firefox, there is an extension called Fasterfox, which loads all links in the background. The improvement is perceptible on simpler pages, like Google's search results, but on large pages with lots of content and links, the chance Fasterfox preloads the link you're eventually going to click is rather small. It would help if you could configure the links you want preloaded, e.g. grab only links containing words such as "next", "page", or "more" etc., but Fasterfox doesn't support that.

A long time ago I was using a http proxy called Naviscope, which had exactly that feature, and it speeded up page loading very nicely. (Its basic feature was scrubbing ads.) It was freeware and it performed flawlessly, but it got abandoned after a while, and I had to drop it when it became necessary to run an actual firewall. Perhaps there are other proxies now that have the preload feature.

2stepsback

  • Supporting Member
  • Joined in 2007
  • **
  • default avatar
  • Posts: 110
    • View Profile
    • My "2stepsback" blog - Updated!
    • Read more about this member.
    • Donate to Member
Re: Desktop Forum Reader
« Reply #7 on: March 17, 2007, 02:17:11 AM »
What if there is some kind of a server side plugin that allows a newsreader to connect?
Sounds to me like a feature request for a plugin for all popular forum software out there :)

Or, as you said, something along the lines of Flock or LiveWriter or the Mozilla Amazon Browser - this last one (MAB) does exactly that with Amazon.com product listings - which are pretty much as dynamic as dynamic can get - it's a desktop program since it is a mozilla extension with an XUL-based GUI of its own.

Someone good at making Firefox extensions can see the source of MAB and build this extension for PHPBB, PunBB, SMF, IPB. Those IMO, are the popular (90%) forum applications.

Still one step ahead, a generic scriptable XML-based (easiest to "script") forum reader on the lines on WinHTTrack script.

I don't have the time or skill (XUL/xpi) else I would have thought seriously of writing this kind of thing. I'm thoroughly convinced of the usefullness of the idea.

Meanwhile as a simple workaround for bandwidth-handling, use Opera browser. It is sleek, fast, completely configurable as far as downloading images / Javascript / Flash goes. And it caches practically *everything*. So much that I've often been shown a page that's a couple of days old because I configured it to "never check latest version" of the pages. Consequently, I used to wonder what happened to the activity of the site. Till I realised that I had to press F5 :) :) :)

-2stepsback
An apple a day keeps the doctor away. A good deed a day keeps the Devil away.
See http://www.codinghor...archives/000735.html
------------
<a href="http://www.w3schools.com/">W3Schools</a> - A collection of free HTML, CSS, JavaScript, DHTML, XML, XHTML, WAP, ASP, SQL tutorials with lots of working examples and source code.