Author Topic: ZillaRank: An improved search for smf forum (Read 274,890 times)

mouser · « **on:** July 04, 2007, 06:48 PM »

As many of you know, we often joke that the SMF search function on the forum sucks, and seems to miss returning search results that we know are there.

Well, our own Wordzilla has been immersing himself in the SMF code, and has begun totally revamping the search code used by the forum.

He's just gotten started, but we need your help to:

test it and see if you can find any posts that should be returned during a search but aren't -- this is most important request -- try to find searches that are missing results that should have been found
make other suggestions on what you'd like to see improved (one thing already planned by wordzilla is to be able to sort results easily by date, author, relevancy).

We look forward to your help in bughunting!

NOTE: SEE ALSO THIS THREAD: http://www.donationcoder.com/forum/index.php?topic=9134.0

jgpaiva · « **Reply #1 on:** July 04, 2007, 07:17 PM »

I'm glad to know this is working in the forum!!!

First suggestion:
Ignore the "MOVED: " threads, since those are a repetition of the real thread.

nudone · « **Reply #2 on:** July 04, 2007, 07:22 PM »

fantastic.

i'm sure wordy will make a brilliant job of it.

Wordzilla · « **Reply #3 on:** July 04, 2007, 07:27 PM »

Good idea Joao, maybe I should give all "MOVED: " threads extra low relevance score.

I'm hoping that this first update will be able to address these 2 issues:

1. Missing results

2. Excessive noise/irrelevant info in results

The next update to forum search will (hopefully) feature a better relevance ranking system. We dont need having 10-pages of results - IMO just a dozen most relevant topics on topmost would suffice, for most people.

Lashiec · « **Reply #4 on:** July 04, 2007, 10:12 PM »

So this is the the SMF search you were going to put your hands on

Excellent! Now all the other forums (especially those using phpBB) would be envious of the improved search function here

f0dder · « **Reply #5 on:** July 04, 2007, 11:09 PM »

If this turns out well, are you going to contribute this back to the SMF people?

mouser · « **Reply #6 on:** July 04, 2007, 11:21 PM »

Quote

If this turns out well, are you going to contribute this back to the SMF people?

Of course. What a ridiculous question! we're not savages here!

lanux128 · « **Reply #7 on:** July 05, 2007, 02:13 AM »

so how do i get involved? can i just start typing the queries now?

mouser · « **Reply #8 on:** July 05, 2007, 02:23 AM »

yep, just start searching for all those posts you couldnt find before

mouser · « **Reply #9 on:** July 05, 2007, 02:41 AM »

Here's a really good idea come up with on the irc channel with hamradio:

Right now if you search using the editbox at top of the forum, while you are in a forum section, it will ONLY search in that forum section.

This is very disorienting and most people don't realize it.

What if ZillaRank changed this behavior and always searched the whole board, BUT (here is the clever part), gave a higher score bonus to results from the section you are in at the time of search.

lanux128 · « **Reply #10 on:** July 05, 2007, 03:52 AM »

Quote from: mouser on July 05, 2007, 02:41 AM

What if ZillaRank changed this behavior and always searched the whole board, BUT (here is the clever part), gave a higher score bonus to results from the section you are in at the time of search.

that would be cool! i fully agree..

f0dder · « **Reply #11 on:** July 05, 2007, 09:09 AM »

Quote from: mouser on July 04, 2007, 11:21 PM

Quote
If this turns out well, are you going to contribute this back to the SMF people?

Of course. What a ridiculous question! we're not savages here!

Well, I meant more in the sense "and do you reckon they might fix up their default behaviour?". Doing the contribution itself is mostly a no-brainer, I even think their license requiresa it?

Quote from: mouser on July 05, 2007, 02:41 AM

Right now if you search using the editbox at top of the forum, while you are in a forum section, it will ONLY search in that forum section.

This is very disorienting and most people don't realize it.

What if ZillaRank changed this behavior and always searched the whole board, BUT (here is the clever part), gave a higher score bonus to results from the section you are in at the time of search.

It's very disorienting indeed!

I think a drop-down solution might be nice, with "prefer hits from this section", "only search this section" and "just search entire board, dammit!" choices...

Wordzilla · « **Reply #12 on:** July 05, 2007, 10:28 AM »

Quote

I think a drop-down solution might be nice, with "prefer hits from this section", "only search this section" and "just search entire board, dammit!" choices...

Yup I believe this is more practical. An update to the script will be ready soon. By default it searches the entire forum unless user specifizes board/topic.

Wordzilla · « **Reply #13 on:** July 05, 2007, 05:50 PM »

Update: v1.2.1 is now live on DC forum

So what's new?

1. Searches entire forum by default, but is also capable of search within board/topic

Quote

Right now if you search using the editbox at top of the forum, while you are in a forum section, it will ONLY search in that forum section.

Now you no longer have to go back to forum index to search entire site, with new our DC smf search.

Quote

I think a drop-down solution might be nice, with "prefer hits from this section", "only search this section" and "just search entire board, dammit!" choices...

It's fairly easy for us to modify the current search boxes to better support searching within board/topic, if desired.

2. Active board ranking adjustment (thanks to mouser)

Quote

What if ZillaRank changed this behavior and always searched the whole board, BUT (here is the clever part), gave a higher score bonus to results from the section you are in at the time of search.

The script now raises the ranking of all matching topics within the forum you are browsing, while displaying all matching topics in the entire forum.

By applying a multiplier to existing smf relevance score, it distorts the ranking just a bit.

It only taking a boolean value change to turn off this feature.

Unlike standard smf search, DC search does not ignore words with 3 letters or less. So any keyword counts!

mouser · « **Reply #14 on:** July 05, 2007, 05:54 PM »

the new active board scoring heuristic is great.
(note for testers, if you do the same exact search twice in a row you wont see the effect because of smf caching, so make sure if you test this you conduct a search for some other term in between tests).

Wordzilla · « **Reply #15 on:** July 05, 2007, 06:24 PM »

Also numbers are now acceptable as keywords, try:

20% discount, smf 2.0, etc.

Wordzilla · « **Reply #16 on:** July 05, 2007, 08:53 PM »

Update: v1.2.2 is live now

thanks to our testers (esp. cranioscopical and Lashiec), a bug involving fulltext search engine and '-' dash handling has been fixed.

Thanks!!

Wordzilla · « **Reply #17 on:** July 06, 2007, 06:33 PM »

Update: v1.4.0 is live now

A few minor bugs have been fixed.

And much improved search performance for:

1. Phrases

2. Words that are prefixed and/or suffixed

We are looking forward to releasing this script soon. Join us in bughunting and plz do lemme know if you run into anything search-related that confuses/bothers you!

nudone · « **Reply #18 on:** July 06, 2007, 08:10 PM »

fantastic!

mouser · « **Reply #19 on:** July 07, 2007, 04:04 AM »

I hearby offer a prize to anyone who can now find a search on this forum that misses a result it should return.

lanux128 · « **Reply #20 on:** July 07, 2007, 05:57 AM »

Quote from: mouser on July 07, 2007, 04:04 AM

I hearby offer a prize to anyone who can now find a search on this forum that misses a result it should return.

which begets the question: what is the prize?

btw, good job, WordZilla.. now to undo the habit of searching from Fx's search plugins..

mouser · « **Reply #21 on:** July 07, 2007, 06:07 AM »

Quote

what is the prize?

that depends on if i have to give it out. if not, it's a million dollars. if so, it's a piece of gum and a kind word.

taichimaster · « **Reply #22 on:** July 07, 2007, 06:21 AM »

Quote from: mouser on July 07, 2007, 06:07 AM

Quote
what is the prize?
that depends on if i have to give it out. if not, it's a million dollars. if so, it's a piece of gum and a kind word.

cranioscopical · « **Reply #23 on:** July 08, 2007, 01:22 AM »

Quote

hearby offer a prize to anyone who can now find a search on this forum that misses a result it should return.

I searched for the message telling me that I'd won all of the prizes in this month's draw, but it couldn't be found!

Wordzilla · « **Reply #24 on:** July 08, 2007, 09:27 AM »

DC SMF Search Mod is out!

http://www.donationcoder.com/forum/index.php?topic=9134.msg67388#msg67388

Quote

Improvements to existing SMF code

Significant reduction of missing results (now very rare)
Less noise and therefore reasonably better relevance
Enhanced phrase search
Intelligent handling for ‘ apostrophes and – dashes
Supports short keywords (3 characters or less)
Supports stopwords (clichés)
Selectively expanding the search by finding keyword inflections

Features unique to this mod

Active board ranking heuristic (based on mouser’s suggestion)

Now u can even search phrases with ' apostrophe(s) in them, e.g. "it's worth it", "mouser's"

At last I no longer have to send my little bot to crawl DC forum twice-monthly!

I hope other SMFers will find it useful as well.