DC SMF Search Mod - an improvement search for smf forum (codename Zillarank)

[Wordzilla]

Overview

DC SMF Search is a mod to the existing smf forum search function.

It has been powering forum search on DonationCoder.com since July 04 2007, so you may want to give it try on our forum.

For illustrated demo, visit: http://www.donationcoder.com/forum/index.php?topic=9434.0


Features

1. Significant reduction of missing results (now very rare)
2. Less noise and therefore reasonably better relevance
3. Hugely improved ranking algorithm (when using FullText index)
4. Enhanced phrase search
5. Intelligent handling for '(apostrophe), .(period), and –(dash)
6. Supports short keywords (3 characters or less)
7. Supports stopwords (clichés)
8. Active board ranking heuristic (based on mouser’s suggestion)
9. Expand/foldable result summary
10. Viewing all matching posts in a topic with one click
11. Ability to sort search results live
12. Quick dropdown search box


IMPORTANT CONFIGURATION INFO

Works best with Search Settings (in admin panel):

1. Use FullText index (Search index -> FullText index)
2. Leave "Force the use of a search index" UNchecked
3. Leave "match whole words only" UNchecked
4. Maximum results to show: 0  (0: no limit)


Compatibility and Installation

Tested on SMF version 1.1.3. To install, use package manager in admin panel - Packages.


Download

http://custom.simplemachines.org/mods/index.php?mod=887


Author's personal opinion

DC SMF Search does not consume noticeably more resource than standard smf search.


NOTE: SEE ALSO THIS THREAD: http://www.donationcoder.com/forum/index.php?topic=9091 --mouser

[mouser]

One other note that wordzilla forgot to mention -- he is going to be improving this mod in upcoming weeks, so:

• make suggestions for other improvements you'd like to see
• please report any cases where it doesn't find something it should
• this is a beta, and new improved versions of this mod will be released regularly in next month so people who don't like the idea of re-installing this may want to wait until it becomes more stable

[Wordzilla]

Yeah u were right mouser, just squashed a tiny bug 

1.0.0.1  Fixed a minor bug concerning phrase exclusion, e.g. heat wave -"newsletter for"

btw, you don't have to go through steps to install/upgrade the mod, just copy the search.php in the package and replace the original/older one in Sources folder

[Wordzilla]

New version uploaded (see attachment of the first message)

1.0.0.2  Fixed a missing results bug when query contains ' apostrophe


We are looking forwarding to submitting this mod to SimpleMachines.org soon so that more SMFers get to use it -- your help is needed! 

[mouser]

A little background to this mod and why it has been temporarily removed from download for now:

Many of you know that the smf built in search functions are flawed -- users have been complaining about it not finding posts for a while.  After failing to get smf people to fix the search after almost a year of begging and pointing out the problems, wordzilla dove right in and spent some real time fixing them and improving them, and was looking forward to improving them even more.

I was so proud of the work that I went and posted to the smf forum and invited people to try it if they wanted;  after asking for a fix for so long and then sharing one i really thought there would be some genuine thanks and appreciation.  Instead we got a pretty rude note saying that the zip file with the fix contained a modified version of one of the files, which is illegal according to their licensing, and we need to remove it and only include the "diff" file or the issue would be "elevated").  For people like me who dont use the package installer, the rule against not including a patched file is harmfull, and the attitude of the smf post was just plain offensive.

So in protest i've asked wordzilla to remove his mod, and I will report on how this issue develops.  I hope it's just a little glitch that is cleared up and that once the smf developers see what's happening they will change their approach.  I'll post a follow up later.  If it stands, I will remove our other inline image mods as well and we will wash our hands of the matter.

[Wordzilla]

The reason I took extra step to include the standalone modified search.php was:

For some unknown reason, the original package will NOT uninstall (even thou it installs without a damn problem) in SMF package manager. As such if the user continues with the uninstallation process, his search.php will be ruined.

And what make it even worse, he cannot reinstall the package right away to fix search.php -- now he has to:

1. Delete the original package
2. Download updated package
3. Restore original search.php from official smf installation
4. Install updated package

Do we hate supporters of our mod? What exactly is the point of troubling them with all these extra time-wasting steps?

If they have the already modified standalone search.php in the mod, applying/upgrading DC search to their own SMF forum couldnt be easier -- Copy + Replace, that's it!

[jgpaiva]

which is illegal according to their licensing, and we need to remove it and only include the "diff" file or the issue would be "elevated". 

That does sound like something coming from someone asking not to be helped.
I understand that they have their own rules, and they could just be asking you to follow them, but that's definitelly not the correct way to do things.
I agree with you, mouser. Better leave them with nothing.
On the other hand... Isn't there some kind of clause about people who make mods of SMF being obliged to distribute them?

[nudone]

i confess i don't really understand what this problem implies - does it mean DC is allowed to use wordy's brilliant search mod or not?

[Wordzilla]

i confess i don't really understand what this problem implies - does it mean DC is allowed to use wordy's brilliant search mod or not?

No worries nudone, nobody was trying to stop us from using our own modified script. 

Simply put it was because I included the ultimate modified script in my package so that ppl could just copy/replace to upgrade their search script, without going through the smf mod installer (a forum component).

SMF ppl claimed that the distribution of this directly replacement for their original script was "redistribution without written permission" and was again their licensing terms.

I only included the direct replacement script in the package because it was not possible for users to uninstall my package and as such it'd make it difficult for them to upgrade their search script next time. Now what could get easier than direct copy/replace?

Anyways, I've been working for 5 hrs to finally get a working uninstallable package out. SMF installer is hard to play with, sometimes it works, sometimes it just doesn't -- for no apparent reason.

[nudone]

ah, i see. thanks for the explanation, wordzilla.

[mouser]

I wasn't very clear --
basically what happened is that when we released the modification, instead of a thank you we got someone an official smf person complaining that due to a tiny technicality we were violating their license and we should correct it within a few days or face legal action.  in fairness the correction we were being asked to make was minor, but the only attitude left me with a really bad taste in my mouth, and it's a non-sensical rule that just makes it harder for people to share their mods, so that's why i had such a negative reaction.

[cranioscopical]

instead of a thank you we got someone an official smf person complaining that due to a tiny technicality we were violating their license and we should correct it within a few days or face legal action...  left me with a really bad taste in my mouth.

Small-minded people abound who are concerned with the letter of the law and not its spirit. I can well understand (and share) the initial flare of...annoyance. Aggravating though such people may be, it would be a pity to deny the larger community of SMF users the chance to benefit from the good work that's just been completed here. Might it be worth trying to make contact with someone at SMF who does enjoy a modicum of common sense, in order to see if a sensible distribution can be negotiated and 'sanctioned'? Don't take your ball and go home, that puts you in their league. Count to ten and try again?

[mouser]

I'm still hoping someone at smf will wake up.
Here was my initial response and maybe it will shed some light on my feelings:

from smf reply to wordzilla posting his modification for others:

UPDATE: It has been agreed that including the full Search.php file is redistribution, which you have not been given the permission to do. As the package already includes the modifications in the xml file, just remove the Search.php file and you'll be compliant. If we don't see this fixed by Friday, we will be forced to escalate the issue (and we really don't want to have to do it, it is a waste of time for everyone).

And my reply:

I understand that when big corporations get involved in things that things get more confusing, but this response to us trying to share an improvement that we finally gave up waiting for smf to fix and wrote ourselves, after i've been asking smf to pay some attention to for almost a year, is mind boggling to me.

My inclination at this point is to just take down this mod, and the other inline image mod we did that people seem to like, and just never mind.

I understand you are just doing your job but I'd really like the smf coders to take a reconsider if this is the approach they want to take with these things, it's just the wrong attitude in my opinion.  We included a mod, and then the patched version of one file.  It's just silly to object to that.  For people like me who do not use package mods and modify the files themselves, stopping people from including a modified file is insane.

If i sound bit peeved it's because after all of this time asking people to fix the bugs in smf search, what i really expected was a "cool, thanks for working on this and sharing it!".

We'll remove our mods and maybe this can spark a discussion on simplemachines about changing your approach to mod writers, if not, que sera sera.  Please understand I'm not mad at you Motoko-chan, I'm sure you are just doing your job.  That's why my protest is against your job instructions and why i want someone who makes the rules to read this post and think about this approach.

[jgpaiva]

Very good post, mouser.
I also support your points.

[mouser]

You'll probably sense that a good deal of my anger about this stems from the fact that it's been almost a year since i've been trying to get someone at smf to fix their broken search code, and felt like i was just speaking into the wind, mystified that no one was taking the broken search issue seriously.

Then when wordzilla agreed to devote his time to improving it, and went through the considerable effort to creating something others could benefit from, and we posted it, i was sure that wordzilla would see some very happy smf people, thrilled that he shared his improved code.  All the time he was working on it I was telling him how wonderful this would be to all smf forum admins, and cheering him on.  He did a lot of work he didn't have to in order that it could be understood and used by others.

When I finally posted it i was anxiously waiting for the nice replies about wordzilla's mod and the thanks he surely deserved.

Instead, we got one comment, accusing us of violating a rule that makes no sense and threatening to "escalate the issue" if it's not fixed by friday.  If this is how they thank people who are trying to improve their software and help their community, then count me out.

[mouser]

I should note that wordzilla himself was not angry about this, and he isn't the one causing the ruckus -- it's really just me in a very foul mood about what i see as a really bad sign about what's going on at smf.

Maybe my mood will calm down about this, but I'll tell you, if you ever see me reacting like they did to someone offering an improvement to this site or one of my programs, please shoot me immediately.

[cranioscopical]

I understand you are just doing your job but I'd really like the smf coders to take a reconsider if this is the approach they want to take with these things, it's just the wrong attitude in my opinion.  We included a mod, and then the patched version of one file.  It's just silly to object to that.  For people like me who do not use package mods and modify the files themselves, stopping people from including a modified file is insane.

I understand your frustration and often share such feelings when dealing with some of my clients. (In my time I've rendered considerably more pithy responses than these.) Even so, "wrong attitude", "silly", and "insane" are probably not the best-chosen terms to gain co-operation.

Also, instead of pointing out what you want, try headlining the benefits to them. Nobody likes to be told they're wrong, especially not those who are!  It makes little-thinkers intransigent.

[Wordzilla]

As u know mouser has just removed all of his mods on official smf site in protest and I just took my search mod offline in support.

Mouser's been complaining about smf search on SMF forum for over a year, elaborating on detailed cases where smf search fails (and it still does). A few really great guys at SMF posted improved scripts for mouser and our forum -- nevertheless as I read his posts I believe his requests went mostly unheeded.

When mouser asked me to work on this mod back a week ago, he told me in excitement that many SMF forum users will be benefited from my mod if it's a success and we'll be able to contribute "big time" to SMF community.

And as soon as the mod was released, mouser posted on SMF forum to tell ppl there that our improved search mod was available for download.

It's just unacceptable for an SMF official to say, without a "thank you", but:

"If we don't see this fixed by Friday, we will be forced to escalate the issue (and we really don't want to have to do it, it is a waste of time for everyone)."

I'm not saying the SMF person was technically wrong on this, s/he pointed out my license violation and did her/his job -- technically no question about that.

It's still upsetting me that whilst all of our released mod are about open-source, share, community contribution, someone (maybe more ppl) in SMF are thinking the corporation way, the litigation way -- right off their mind!

I don't think mouser and I ever developed mods for profits (not even donations), personal fame, etc. We want our users to enjoy our product as easy as possible, to make life easier for them, since web stuff isn't really simple -- often you f*ck up things without even knowing what went wrong.

it would be a pity to deny the larger community of SMF users the chance to benefit from the good work that's just been completed here.

Exactly. SMF Community is simply great (SO FAR). Some gurus there were really helpful to me sorting out my tricky installer problems and I really appreciate it.

We don't want SMF to lapse into a corporation run by corporate-minded ppl -- from a pro-open-source community where ppl care, develop and share. That's why we protest for a change to their licensing policy.


Anyways, I'll still be improving SMF search (currently just for DC), so don't hide bugs from me!! 

[hamradio]

Maybe we should post about it on digg and let the whole world know how they feel when someone tries to help them for free. Count me out of ever using SMF if I ever have a forum I need to setup I was looking at it for possibly one that I may need to setup, but now its moved to the bottom of the list (even under the pay forums) and even further if they are going to continue to act in this manner. I give them . Hate to act like this but that is my feelings on the issue.  Hope you all understand.

[nudone]

i admit my first reaction was to suggest that SMF was dropped completely in order to use some other open source forum setup. i didn't say that because i realise it's impractical - but if i could wave a magic wand i would make it so.

i hope you find out that 'they' are more understanding and that 'they' don't all share the same hardline approach to the rules they love to enforce.

again, i don't know the back story to the search problem with SMF but so far it sounds like they've been upset by a few things - 1. that their search method is crap, 2. that they are incapable of rectifying their crap search method, 3. that someone had the cheek to make a search method that worked without any of their input - because they are so crap, 4. their personal pride has been destroyed because someone now wants to rub their noses in it by distributing a better search mod, 5. etc.

some people just can't take constructive criticism and get all defensive about their work when you try to help - but threatening legal action makes me wonder if there are other things provoking their reaction. perhaps they've had several occasions already that they felt the need to state their ownership of SMF in a legal manner and so the newly submitted search mod caused a reflex negative response.

[mouser]

I don't think it has to do with anyone being upset at criticism of their search code.. however i do think that it suggests that for every 1 person they have working on improving the code, they have 10 people doing legal and corporate compliance and marketing..  Not really a promising sign.

[mouser]

A reply i posted which addresses some of the "why" issues that cranioscopical suggested i tackle:

Did you ever see the movie "5 easy pieces" -- there is a famous "no substitutions" scene where jack nicholson is in a restaurant, trying to deal with a waitress who is "just following the restaurant rules".. that's how i feel about smf now:
http://www.youtube.com/watch?v=6wtfNE4z6a8

I appreciate you guys taking the time to explain your positions.
I would however direct you to look at this post by Rudolf: http://www.simplemachines.org/community/index.php?topic=118997.msg1156617#msg1156617
which discusses some of the inherent conflict between saying mod writers can't include modified source file but can include a package mod file which is nearly identical.

Much of the argument above seems to deal with it being unwise to copy modified files over your installed files unless you really know what you are doing.  I couldn't agree more.  I never suggest anyone do this.  So you don't have to convince me of the wisdom of that.

The reason i have always included the original+modified versions of the smf files i change in my mods is to make it 1000x easier for people who dont use the package installer to compare files and make the changes manually.

You guys probably already know from many posts on the forum that the package installer, while something to be proud of, can be difficult to get working properly when you have multiple mods that affect the same files.

I do not use the package installer -- i know others don't as well.  We like to modify the files manually so we know what's being changed and where, and we can correct for unanticipated interactions.  In my mods i include the original and modified file so people can use a normal dif/compare tool and see easily the changes that need to be made, and so people can use a proper merge tool to merge in the changes.

The package mod file format is poorly suited to manual merging.  I've done it before and it's a huge pain compared to having a set of modified files and being able to do a visual side by side compare+merge.

I belive it is an unreasonable rule.  No one is talking about allowing forking or redistribution of your entire forum software.

It's easy enough for you to change the rule to say that mod writers can include modified versions of the files. If you want me to rename them so as to avoid the possibility of someone mistakenly overwriting one of their files, fine.

I know this will sound harsh and probably uncalled for, but in my view this is one of those cases where we can see whether a company (smf) is more focused on users+coding, or following corporate rules and unable to adapt these rules when they are harmful.  I know you disagree but for me, this is one of the ways i decide which communities i want to be part of.  It's not like we're getting paid to contribute mods.  I have supported smf financially with donations and i continue to believe it's a great piece of work, but i'm just having real second thoughts about the smf corporate model.

[nudone]

as you say, not a promising sign. perhaps they are flexing their muscles ready for when they feel a change of philosophy is due and move into more profitable areas.

[mouser]

perhaps they are flexing their muscles ready for when they feel a change of philosophy is due and move into more profitable areas.

yes, i have somewhat of a fear that this might be the case and we could see a change of licensing model in the future.

[Wordzilla]

i didn't say that because i realise it's impractical - but if i could wave a magic wand i would make it so.

I didnt study smf licensing terms seriously before, coz mouser's such a fan of SMF forum and I don't run any forum myself, now it seems to me that SMF is actually copyrighted, pretty closed-source forum software.

Read their licensing terms here: http://www.simplemachines.org/about/license.php

Among the terms read:

Any Distribution of this Package, whether as a Modified Package or not, requires express written consent from Simple Machines LLC.

I'm not the mind reader of SMF but the terms mainly impress "copyright + development control" on me, for some reason.

1. SMF controls distribution

2. Therefore controls development (see #1), which makes it hard for independent mod writers to write customizations that affect lots of components because SMF sets the framework. Now they are basically writing addons for existing SMF base.

3. "change the terms of this Agreement at any time" allows the company to do whatever they want - at any time. With proper tech, it could turn into a for-profit entity anytime. (as far as I know PHP encryption is already there, and it's not uncommon for developers to charge users monthly subscription fees)

Note that we don't have copyleft source code, but copyright source code. Just because SMF allows us to use their forum software for free doesn't mean we own the code and do whatever we want.

I haven't discussed this with mouser yet but it is somehow not that assuring to put so much community contribution on copyrighted software (forum software) with somewhat restrictive licensing terms.

Perhaps the more our site functions are integrated with SMF, the less options we'll have. So far it should be not v hard to convert our forum to phpbb (open source) and there are already converters for this.

As we are getting bigger (which of course we are), the merely difficult may evolving into something impractical to do, just as you said nudone.

So I'm saying it pays to study smf alternatives like phpbb3 (I'll try) and design our site components to be less reliant on SMF forum software.

Even thou SMF is v unlikely to make any drastic move in the short term, its (possible) gradual change of policy in the long term may make us suffer. Let's hope we won't get too big and complicated to migrate our system then.