Organizing Reviews

[Renegade]

I'm splitting this from a previous topic...

You know I'd love to see a more organized process on this forum where a small group of people could agree to do an organized, exhaustive, frenetic testing of programs and try to come up with some consensus of best tools for a task.

In other words, it would be nice if we could take a thread like this and have a few people just go out and try every help file maker they can find and winnow down the list of top candidates.  Not only would that be helpful but we could then go and try to get some discounts on these, as well as provide some guidance to the non-top authors on how they could improve their programs.

-mouser (August 18, 2007, 12:19 AM)

Just one consideration...

I'd like to throw in "not-testing"... For these kinds of things there are people that use the software in real-world production and can contribute that real-world experience.

One of the problems I find with a lot of reviews is that they are little more than "feature lists" with no real commentary on how the application truly performs in real life.

I simply don't have the time to go out and download, install, then test a bunch of software, but I can certainly contribute for those applications where I really know them very well from real world experience.

I say "real world experience" because there's a VERY big difference between running a simple test for 5 minutes, and using an application on a daily basis.

Of course feature lists are important, but adding in something about the performance would take a review from being a "review" to being "authoritative".

Ok - That's all. Let the ideas start flying on how to get this organized... Perhaps we should start with a list of goals. I'll go first:

GOALS:

1) Make the reviews authoritative and reliable for readers
2) Exhaustive reviews of individual pieces of software
3) Exhaustive comparisons between pieces of software
4) Relatively easy to contribute to the process in a structured way, perhaps "templated" - this would require a metadata type structure
5) Not sure... Please continue...

Cheers!

[mitzevo]

The official reviews follow these guide lines pretty much, as for the mini reviews, they are not as strict, and do not have "Exhaustive" testing methods. That's why they are called "mini-reviews", because they are for DC members to write about software which they like and/or think other people will like and find useful etc.

If you want more authority, the official reviews are what you want. It takes a lot of time to review "stuff", especially the way you have outlined above.. The official reviews have been idle for since 2004/2005, I think first of all they should be updated for 2008 (I have mentioned this many times in IRC and a few times on the forum) and then probably updated at 2010 or some thing..

[Grorgy]

I like both sorts of reviews.  The detailed review of certain types of applications that renegade is after are excellent but, and it’s a big but, if the reviews posted weres limited to those that were extensive, then the number of reviews would decline significantly, and the chance of anyone who isnt technically minded to write one almost disappears.  If im looking for something or just looking then being able to get a list together of possibles and probables from people who just like their latest purchase/find whatever and the comments that then flow and other programs that are in that category and relative merits is both entertaining and informative and open for quick updates and new players appearing on the scene.    tooo many words i'll stop now   

But just to finish lets have both 

[Perry Mowbray]

A while ago I floated something not dissimilar to Mouser where DC members could add their rating for various aspects of a particular piece of software.

Mouser was concerned that it was too easy for unethical people to overrate their products, or products they like, and if that happened then it certainly isn't a great idea, or not a bad idea too easily ruined.

But I still like the idea of having a rating system, or as Renegade said a "Non-Testing" review. I thought that aggregating all of DC's opinions about applications would be a great addition to the site. Sort of like Wacoopa but not just usage. It'd work hand-in-hand with the other reviews too, and actually add more value to them.

Anyway Renegade, in answer to your suggestion: I agree!

[Darwin]

+1 for me too. I like your idea, too, Perry, but I agree with mouser. Renegade's is the best solution - still "ripe" for exploitation but at least anyone unethical to do so is going to have to write a thorough review of an app to get it posted. My guess is anyone tempted will be put off by the effort involved and move on...

[Renegade]

So how about this as a meager beginning for a review:

1) Post about a very narrow category of software to review, e.g. Help file authoring tools
2) Include a few examples (continue with this throughout up to step 6)
3) Define the major features / categories
4) Define minor features / categories within the majors
5) Define "extras" or unique features
6) Build complete list. List has these columns:
Feature Name
Yes/No
Rating
Comments
(Columns could be tables as well, e.g. A comments table)
7) Volunteer to review a piece of software (at anytime up to here)
Post review for later aggregation

Well, that's a brief outline of a methodology.

I'm up for reviewing H&M...

[Perry Mowbray]

Well, that's a brief outline of a methodology.

-Renegade (August 18, 2007, 08:46 PM)

I like it!

But I wonder if a Wiki may not fit the bill better? If you start H&M can anyone else add to your Index/Introduction page?

I'd also consider grouping the features into Classes such as the standard Functionality, Usability, etc and average out to give overview ratings across software groups.

I really like this idea.

[Darwin]

A donationcoder review wiki - I like it! There would not doubt be issues, but this would a very interesting experiment 

[Perry Mowbray]

A donationcoder review wiki - I like it! There would not doubt be issues, but this would a very interesting experiment 

-Darwin (August 19, 2007, 07:34 PM)

I'd imagine that it wouldn't be a full Wiki as we'd want to limit edit rights to members or certain members, but apart from very granular Forum permissions, I can't see how we could maintain a living page as the Introduction/Index to the software categories?

And that's where I think the benefit of DC doing something like this is: we have a large number of expert/well informed/opinionated  members, that if we were able to produce an aggregation of their usage/opinions/mini reviews into something larger it would be a fantastic resource.

It's the living nature of something like this that appeals to me: software changes, versions change, etc, etc and not always for the better. What's the point of reading a comparative review that recommended Software X over Software Y if Software Y's development has overtaken X's?

The review is out of date. Where as, if we had an Introductory Page that listed Feature Rankings for the reviewed versions then the comparison would take care of itself. Having the ability to have multiple reviews of the one software by different people would be wonderful 

[mouser]

I have given a great deal of thought to creating a nextgen review system, with a hierarchy of editors and using wiki-like technology.  I'd love to work on it but we are talking about a very big project and i haven't yet found a way to either fund it and i dont have the time to work on such a large project without funding.

[Perry Mowbray]

I have given a great deal of thought to creating a nextgen review system, with a hierarchy of editors and using wiki-like technology.  I'd love to work on it but we are talking about a very big project and i haven't yet found a way to either fund it and i dont have the time to work on such a large project without funding.

-mouser (August 19, 2007, 08:19 PM)

What about working up a Proof of Concept on DC then asking DC Members who'd like to throw in? I imagine that it's not only financial but more time issues as well?

Is this what you meant once when you said "a big review site", because if it is I think you undersold your idea 

[Renegade]

I like the idea of a wiki.

As for funding, that's a hard question.  A few bucks from a few people is hardly enough to fund a project. AdSense? Yahoo Publisher's Network? Open up the review request to authors for a fee? (e.g. Donate $100.00 or whatever to have your application included in a review.)

Not sure how that problem could get solved best. A commercial project like this would bear significant costs...

[Darwin]

I'm a bit naive about the financial side of the internet... WRT wikis, is the expense due to bandwidth, hardware, software, or personnel requirements? Or a combination of all of the above? If it's personel, could the cost be defrayed by using and abusing the good members of this community in a volunteer capacity?

Just curious because I like the idea (of the wiki - the use and abuse of dc's membership is just a possible peripheral benefit!).

[Perry Mowbray]

I'm curious to know what sort of funding models you'd think would be appropriate and how far you got down that track.