Author Topic: ZillaRank: An improved search for smf forum (Read 274,890 times)

Wordzilla · « **Reply #25 on:** July 08, 2007, 09:33 AM »

To search DonationCoder.com forum directly from within your browser, check out app103's plugins:

http://www.donationcoder.com/Software/Other/ff/

Wordzilla · « **Reply #26 on:** July 12, 2007, 11:23 PM »

Forum search gets another update, which deals exclusively with relevance ranking (as I'm confident that previous updates already fixed missing results and excessive noise).

Currently the following factors are calculated to work out how relevant is a topic/post to your search query:

Original SMF Factors:

Number of matching messages within a topic (the more replies to a topic in which same keywords can be found, the more likely this topic is relevant)
Age of last matching message (we love latest news, how recent is this message?)
Topic length (number of replies in a topic. How interested are people in this discussion?)
Matching subject (bull's eye!)
First message match (sets the theme of the topic)
Sticky topic (moderator's "sponsored link") (for now we are ignoring this factor)

Per-message Keyword Analysis:

Rarer keywords get higher points, cliched ones lower. And how concentrated are the keywords in a message? Suppose we just posted a super-lengthy newsletter on the forum, with truckloads of keywords -- there's a great chance that it'll match our query, however it shouldn't be well-ranked coz keywords are diluted.

Penalty for messages written with "Spartan brevity":

They get awarded for high keyword concentration but there won't be much to read. We don't like them.

Active board ranking heuristic:

Most likely you are more interested in relevant topics within the board you are currently browsing, as such we give "local" matching topics an edge over others to catch you eye.

Quote from: jgpaiva on July 04, 2007, 07:17 PM

First suggestion:
Ignore the "MOVED: " threads, since those are a repetition of the real thread.

Hmmm I haven't added any routine to check for MOVED posts at this time because:

1. Not so many "MOVED: " threads out there, it's unlikely to be efficient to add this check for every matching topic
2. They get penalized for "not saying much"

I hope this update will help you get to your targets in the shortest time possible.

I have no doubt there's still much to be done when it comes to relevance ranking. so...

mouser · « **Reply #27 on:** July 12, 2007, 11:29 PM »

awesome

cranioscopical · « **Reply #28 on:** July 13, 2007, 04:38 AM »

It's an odd but pleasant sensation to find the search feature useful

mouser · « **Reply #29 on:** July 14, 2007, 02:26 AM »

ZillaRank has a new update with a very cool new feature:
You can now sort by relevance or date from the search results page by clicking on columns.

Very very useful stuff - congratulations to wordzilla for his amazing coding work on this!

(boo to simplemachines.org for not caring, what is going on with them? )

Wordzilla · « **Reply #30 on:** July 19, 2007, 10:30 AM »

ty mouser

Our forum search got another update just now, changes:

1. Performance tweak to speed up queries that contain short keywords

Queries like fd-isr used to take approximately 10 seconds to return results, now 2-3 seconds.

At the moment it takes 2-3 seconds for DC search to return results (in most cases), and in the worst case (an insanely stupid query which only contains a bunch of 2 or 3-letter words, with intent to kill the server

) it should not have you wait for over 8 seconds.

2. Accepts url search parameter and is now fully compatible with DC search browser plugins

SMF search wasn't designed to receive search terms from URL so there were problems when, e.g., the user searched DC from browser search plugin with query that contained " quote or ' apostrophe ("it's worth it"). Now the problem is no more.

3. Direct search link

Every time you do DC search, at the bottom of each search results page you'll see "Static link for reproducing this search: here", which basically allows copying/pasting the url of the search query directly, without having to elaborate on the how-to -- "search xxx (after the semi-colon) here on DonationCoder, with quotes, blah blah...".

jgpaiva · « **Reply #31 on:** July 19, 2007, 11:19 AM »

Quote from: Wordzilla on July 19, 2007, 10:30 AM

3. Direct search link

That's a very important add!!!
Thanks a lot for your work in this, anderson!!

Now i never google search to find stuff on DC, your work really has made wonders

jgpaiva · « **Reply #32 on:** July 19, 2007, 10:00 PM »

Here's another suggestion: add "by:" filter, similar to google's "site:" filter, so that we can make a search which searches only the posts by someone without having to go to the advanced search

Wordzilla · « **Reply #33 on:** July 20, 2007, 12:57 AM »

Quote from: jgpaiva on July 19, 2007, 11:19 AM

Now i never google search to find stuff on DC, your work really has made wonders

Does that mean I have to fix SEO for DC as well?

Yes let there be no misunderstanding -- it's gonna be my next project!

Wordzilla · « **Reply #34 on:** July 20, 2007, 08:43 AM »

Quote from: jgpaiva on July 19, 2007, 10:00 PM

Here's another suggestion: add "by:" filter, similar to google's "site:" filter, so that we can make a search which searches only the posts by someone without having to go to the advanced search

Good idea, and shouldnt be v hard to implement. Lemme c what I can do.

Wordzilla · « **Reply #35 on:** July 20, 2007, 09:39 AM »

Quote

You can now sort by relevance or date from the search results page by clicking on columns.

Now we have added yet another sortable column: Replies. (number of total replies in the thread, could be an indicator of the popularity of that thread)

So now it's easy to sort results by relevance, reply count, or date, in either ascending or descending (default) order. To some users these functions can be particularly handy -- only on DC, you don't see these on other SMF forums!

I'd like to thank mouser here for his brilliant ideas.

[attachthumb=#]

cranioscopical · « **Reply #36 on:** July 20, 2007, 02:20 PM »

Quote from: Wordzilla on July 20, 2007, 09:39 AM

Now we have added yet another sortable column: Replies.

Very nice, too!

Wordzilla · « **Reply #37 on:** July 21, 2007, 06:50 AM »

Quote from: cranioscopical on July 20, 2007, 02:20 PM

Very nice, too!

TY!

Update: version string search support added.

Now we are able to search specific version of apps on the forum properly (the function used to return excessive noise in many cases).

examples:

2.00.108
smf 1.1

Wordzilla · « **Reply #38 on:** July 27, 2007, 08:31 PM »

DC search got another update yesterday to improve unsatisfactory (IMO

) ranking algorithm.

Special Thanks to jgpaiva for reporting cases where topics were not ranked fairly and for helping us testing candidate scripts.

Among other optimizations, a few new content-based factors have been introduced to penalize/award matching topics individually to give more accurate (relative) relevancy.

For search query "donation", results generated by original SMF 1.1.3 search script:

And results generated by our working DC search:

Well, I'm sure there's is still plenty of room for search improvement -- I'm happy to look into problems to work out updates in the short time possible, if needed, anytime. So do not hesitate to post feedback/bug report here!! TIA

mouser · « **Reply #39 on:** July 27, 2007, 09:59 PM »

The search is so amazing now, a pure joy to use..
The old timers will now be able to tell the young whipper snappers about the old days when search was horrible, and no one will believe them.

Wordzilla · « **Reply #40 on:** July 27, 2007, 10:05 PM »

Quote from: f0dder on July 05, 2007, 09:09 AM

I think a drop-down solution might be nice, with "prefer hits from this section", "only search this section" and "just search entire board, dammit!" choices...

Yes!! Perhaps we need a dropdown box like that on SMF forums:

Our quick search dropdown box will have the following choices:

1. Community (search entire Forum)
2. This board (search within this board)
3. This topic (search within this topic)
4. Help Desk (search our help desk entries)
5. Members (find a member)
6. Entire Site (google site search)

What do you think? Any ideas?

mouser · « **Reply #41 on:** July 27, 2007, 10:11 PM »

I've generally been against this idea, but that's not bad..
i might delete #2 and call #1 "Entire Forum"

jgpaiva · « **Reply #42 on:** July 27, 2007, 10:13 PM »

I personally always make a search in the full site, since i use the "dcs" search alias in opera, thus can't be a big assistence.
I can see the usefulness of that dropbox, though.

Lashiec · « **Reply #43 on:** July 27, 2007, 10:18 PM »

Um, good idea, Wordzilla. I won't be as radical as mouser, removing the second option, mostly because as the board grows, people will want to refine the search (unless the mod performance really escalates well as the size grows). The drop-down box will remove one step, the one for accessing the advanced search. Besides, the possible choices mentioned are well thought

Also, one of my crazy ideas, that I briefly discussed with Wordzilla: an alias for searching the forum with the same options as the drop-down box

Wordzilla · « **Reply #44 on:** July 27, 2007, 10:32 PM »

I suspect there's need for search within a single topic:

We have humongous topics here, like:

General brainstorming for Note-taking software
with 570 replies!

Latest FARR v2.00.135 ALPHA PREVIEW Release - July 17, 2007
with 783 replies!

You know when you do search, even when there are multiple posts matching your query within a topic, only the latest (yes, just one) matching post will be shown (quoted) on search results page:

[attachimg=#]

(Only the latest matching post is quoted)

And as a matter of fact, there are two matching posts in that topic, and the second one (not shown in search results) might be more interesting:

http://www.donationcoder.com/forum/index.php?action=search2;topic=3824.0;local=1;search=allsnap

Now without search box support for searching within a single topic, you literally have to go through more than a dozen topic pages (if there are more than 200 replies) to find the rest of posts that also matched your query!

p.s. our new DC search (engine) supports single topic search straightforwardly, we are actually deciding on whether to add user interface for it.

jgpaiva · « **Reply #45 on:** July 27, 2007, 10:38 PM »

You're absolutelly right.
I think that box should be added, and it could be handy even for people who always use the whole forum search, to search for something more relevant in one single topic.

AbteriX · « **Reply #46 on:** July 27, 2007, 11:30 PM »

Pretty cool, thanks Wordzilla

i too have two suggestions to improve the search at least i would found useful:

AbteriX · « **Reply #47 on:** July 27, 2007, 11:38 PM »

First: i like to find all post no matter what subject from the last three days to see what happens.
f.ex. i was online but had no time to read all ..... and next i went online i don't see the last posts any more.

Second: i like to see the search result in an compressed form... the subject lines only.
Sometimes this would help to find the wanted thread, especially for the issue mentioned above.

I know there is the way via "Forum > Recent Posts" ... but i guess find Recent Posts by search function is more native?

mouser · « **Reply #48 on:** July 27, 2007, 11:42 PM »

semi-related:
maybe we should have quick links on the advanced search page for:

Show recent topics.
Show unread posts (since last visit).
Show recent posts (full text display).
Show new replies to your posts (unreplied topics).

AbteriX · « **Reply #49 on:** July 28, 2007, 12:08 AM »

And maybe search for "hole word" only as default:===> 'Fun'
With an hint to use asterisks ===> '*fun*' for fuzzy search.
Or better an check box?

Able to use RegEx would be handy of course ==> '\bFun??' ===> 'Fun' , 'funny',....

Maybe "Case sensitive" -search too? I mean no.