topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Saturday October 12, 2024, 11:20 pm
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: ZillaRank: An improved search for smf forum  (Read 235089 times)

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,913
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
ZillaRank: An improved search for smf forum
« on: July 04, 2007, 01:48 PM »
As many of you know, we often joke that the SMF search function on the forum sucks, and seems to miss returning search results that we know are there.

Well, our own Wordzilla has been immersing himself in the SMF code, and has begun totally revamping the search code used by the forum.

He's just gotten started, but we need your help to:
  • test it and see if you can find any posts that should be returned during a search but aren't -- this is most important request -- try to find searches that are missing results that should have been found
  • make other suggestions on what you'd like to see improved (one thing already planned by wordzilla is to be able to sort results easily by date, author, relevancy).

We look forward to your help in bughunting!


NOTE: SEE ALSO THIS THREAD: https://www.donation...dex.php?topic=9134.0
« Last Edit: August 01, 2007, 10:24 AM by mouser »

jgpaiva

  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 4,727
    • View Profile
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #1 on: July 04, 2007, 02:17 PM »
I'm glad to know this is working in the forum!!!

First suggestion:
Ignore the "MOVED: " threads, since those are a repetition of the real thread.

nudone

  • Cody's Creator
  • Columnist
  • Joined in 2005
  • ***
  • Posts: 4,119
    • View Profile
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #2 on: July 04, 2007, 02:22 PM »
fantastic.

i'm sure wordy will make a brilliant job of it.

Wordzilla

  • Forum Search Daemon
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 470
  • Two there should be; no more, no less.
    • View Profile
    • FreeThesaurus.net - The Free Online Synonym Finder
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #3 on: July 04, 2007, 02:27 PM »
Good idea Joao, maybe I should give all "MOVED: " threads extra low relevance score.

I'm hoping that this first update will be able to address these 2 issues:

1. Missing results

2. Excessive noise/irrelevant info in results


The next update to forum search will (hopefully) feature a better relevance ranking system. We dont need having 10-pages of results - IMO just a dozen most relevant topics on topmost would suffice, for most people.  :)

Lashiec

  • Member
  • Joined in 2006
  • **
  • Posts: 2,374
    • View Profile
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #4 on: July 04, 2007, 05:12 PM »
So this is the the SMF search you were going to put your hands on ;)

Excellent! Now all the other forums (especially those using phpBB) would be envious of the improved search function here

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #5 on: July 04, 2007, 06:09 PM »
If this turns out well, are you going to contribute this back to the SMF people?
- carpe noctem

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,913
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #6 on: July 04, 2007, 06:21 PM »
If this turns out well, are you going to contribute this back to the SMF people?

Of course.  What a ridiculous question! we're not savages here!  :P

lanux128

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 6,277
    • View Profile
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #7 on: July 04, 2007, 09:13 PM »
so how do i get involved? can i just start typing the queries now? :)

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,913
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #8 on: July 04, 2007, 09:23 PM »
yep, just start searching for all those posts you couldnt find before :)

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,913
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #9 on: July 04, 2007, 09:41 PM »
Here's a really good idea come up with on the irc channel with hamradio:

Right now if you search using the editbox at top of the forum, while you are in a forum section, it will ONLY search in that forum section.

This is very disorienting and most people don't realize it.

What if ZillaRank changed this behavior and always searched the whole board, BUT (here is the clever part), gave a higher score bonus to results from the section you are in at the time of search.

lanux128

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 6,277
    • View Profile
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #10 on: July 04, 2007, 10:52 PM »
What if ZillaRank changed this behavior and always searched the whole board, BUT (here is the clever part), gave a higher score bonus to results from the section you are in at the time of search.
that would be cool! i fully agree.. :Thmbsup:

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #11 on: July 05, 2007, 04:09 AM »
If this turns out well, are you going to contribute this back to the SMF people?

Of course.  What a ridiculous question! we're not savages here!  :P
Well, I meant more in the sense "and do you reckon they might fix up their default behaviour?". Doing the contribution itself is mostly a no-brainer, I even think their license requiresa it? :)

Right now if you search using the editbox at top of the forum, while you are in a forum section, it will ONLY search in that forum section.

This is very disorienting and most people don't realize it.

What if ZillaRank changed this behavior and always searched the whole board, BUT (here is the clever part), gave a higher score bonus to results from the section you are in at the time of search.
It's very disorienting indeed!

I think a drop-down solution might be nice, with "prefer hits from this section", "only search this section" and "just search entire board, dammit!" choices...
- carpe noctem

Wordzilla

  • Forum Search Daemon
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 470
  • Two there should be; no more, no less.
    • View Profile
    • FreeThesaurus.net - The Free Online Synonym Finder
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #12 on: July 05, 2007, 05:28 AM »
I think a drop-down solution might be nice, with "prefer hits from this section", "only search this section" and "just search entire board, dammit!" choices...

Yup I believe this is more practical. An update to the script will be ready soon. By default it searches the entire forum unless user specifizes board/topic.

Wordzilla

  • Forum Search Daemon
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 470
  • Two there should be; no more, no less.
    • View Profile
    • FreeThesaurus.net - The Free Online Synonym Finder
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #13 on: July 05, 2007, 12:50 PM »
Update: v1.2.1 is now live on DC forum

So what's new?

1. Searches entire forum by default, but is also capable of search within board/topic

Right now if you search using the editbox at top of the forum, while you are in a forum section, it will ONLY search in that forum section.

Now you no longer have to go back to forum index to search entire site, with new our DC smf search.

I think a drop-down solution might be nice, with "prefer hits from this section", "only search this section" and "just search entire board, dammit!" choices...

It's fairly easy for us to modify the current search boxes to better support searching within board/topic, if desired.


2. Active board ranking adjustment (thanks to mouser)

What if ZillaRank changed this behavior and always searched the whole board, BUT (here is the clever part), gave a higher score bonus to results from the section you are in at the time of search.

The script now raises the ranking of all matching topics within the forum you are browsing, while displaying all matching topics in the entire forum.

By applying a multiplier to existing smf relevance score, it distorts the ranking just a bit.

It only taking a boolean value change to turn off this feature.


Unlike standard smf search, DC search does not ignore words with 3 letters or less. So any keyword counts!  :)

:feedback:

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,913
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #14 on: July 05, 2007, 12:54 PM »
the new active board scoring heuristic is great.
(note for testers, if you do the same exact search twice in a row you wont see the effect because of smf caching, so make sure if you test this you conduct a search for some other term in between tests).

Wordzilla

  • Forum Search Daemon
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 470
  • Two there should be; no more, no less.
    • View Profile
    • FreeThesaurus.net - The Free Online Synonym Finder
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #15 on: July 05, 2007, 01:24 PM »
Also numbers are now acceptable as keywords, try:

20% discount, smf 2.0, etc.

Wordzilla

  • Forum Search Daemon
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 470
  • Two there should be; no more, no less.
    • View Profile
    • FreeThesaurus.net - The Free Online Synonym Finder
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #16 on: July 05, 2007, 03:53 PM »
Update: v1.2.2 is live now

thanks to our testers (esp. cranioscopical and Lashiec), a bug involving fulltext search engine and '-' dash handling has been fixed. 

Thanks!! :Thmbsup:

Wordzilla

  • Forum Search Daemon
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 470
  • Two there should be; no more, no less.
    • View Profile
    • FreeThesaurus.net - The Free Online Synonym Finder
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #17 on: July 06, 2007, 01:33 PM »
Update: v1.4.0 is live now

A few minor bugs have been fixed.

And much improved search performance for:

1. Phrases

2. Words that are prefixed and/or suffixed


We are looking forward to releasing this script soon. Join us in bughunting and plz do lemme know if you run into anything search-related that confuses/bothers you!

:feedback:

nudone

  • Cody's Creator
  • Columnist
  • Joined in 2005
  • ***
  • Posts: 4,119
    • View Profile
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #18 on: July 06, 2007, 03:10 PM »
fantastic!

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,913
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #19 on: July 06, 2007, 11:04 PM »
I hearby offer a prize to anyone who can now find a search on this forum that misses a result it should return.

lanux128

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 6,277
    • View Profile
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #20 on: July 07, 2007, 12:57 AM »
I hearby offer a prize to anyone who can now find a search on this forum that misses a result it should return.
which begets the question: what is the prize? :) btw, good job, WordZilla.. now to undo the habit of searching from Fx's search plugins..

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,913
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #21 on: July 07, 2007, 01:07 AM »
what is the prize?
that depends on if i have to give it out.  if not, it's a million dollars.  if so, it's a piece of gum and a kind word.

taichimaster

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 160
    • View Profile
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #22 on: July 07, 2007, 01:21 AM »
what is the prize?
that depends on if i have to give it out.  if not, it's a million dollars.  if so, it's a piece of gum and a kind word.

 ;D ;D ;D ;D

cranioscopical

  • Friend of the Site
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,776
    • View Profile
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #23 on: July 07, 2007, 08:22 PM »
hearby offer a prize to anyone who can now find a search on this forum that misses a result it should return.

I searched for the message telling me that I'd won all of the prizes in this month's draw, but it couldn't be found!

Wordzilla

  • Forum Search Daemon
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 470
  • Two there should be; no more, no less.
    • View Profile
    • FreeThesaurus.net - The Free Online Synonym Finder
    • Read more about this member.
    • Donate to Member
Re: ZillaRank: An improved search for smf forum
« Reply #24 on: July 08, 2007, 04:27 AM »
DC SMF Search Mod is out!

https://www.donation...34.msg67388#msg67388


Improvements to existing SMF code

  • Significant reduction of missing results (now very rare)
  • Less noise and therefore reasonably better relevance
  • Enhanced phrase search
  • Intelligent handling for ‘ apostrophes and – dashes
  • Supports short keywords (3 characters or less)
  • Supports stopwords (clichés)
  • Selectively expanding the search by finding keyword inflections


Features unique to this mod

Active board ranking heuristic (based on mouser’s suggestion)


Now u can even search phrases with ' apostrophe(s) in them, e.g. "it's worth it", "mouser's"

At last I no longer have to send my little bot to crawl DC forum twice-monthly! ;D

I hope other SMFers will find it useful as well.