Author Topic: Locate32 Plugin for FARR by Okke (Read 133321 times)

f0dder · « **Reply #75 on:** December 28, 2007, 07:34 PM »

Yes, as PlatformSDK states, WriteConsole will fail when used with redirection... so that's no good. If you redirect file, GetStdHandle() returns a file handle, so things should work fine. I think/hope notepad fails because the file lacks a BOM marker, but I'll do some testing, also of the code from the blog...

At the plugin side, I guess it comes down to whether you want to parse utf-8 or use ucs-16 directly. For locate, it's probably some printf calls that need to be changed to more specific code.

jmj · « **Reply #76 on:** December 29, 2007, 03:17 AM »

Ok, it seems that unicode characters do not work even in console. It worked when I tested before, but probable
used copepage was valid for those charactes (those was cyrillic or chinese alphapets, now I tested random characters).

So what all I should do to get it work like you want? Atm locate.exe uses wprintf

f0dder · « **Reply #77 on:** December 29, 2007, 04:51 AM »

Ok, it seems that unicode characters do not work even in console. It worked when I tested before, but probable
used copepage was valid for those charactes (those was cyrillic or chinese alphapets, now I tested random characters).
-jmj (December 29, 2007, 03:17 AM)

Hm, it did work for my test, but all I used was some russian text. I did have to choose a non-raster font for the console, though. My codepage is 850, I doubt cyrillic characters are valid in that codepage?

So what all I should do to get it work like you want? Atm locate.exe uses wprintf
-jmj (December 29, 2007, 03:17 AM)

I guess we should take a look at the CRT implementation of wprinf (source comes with visual studio) and see how they handle it... but it breaks down to using WriteConsoleW for consoles, and WriteFile (possibly with wide->multibyte translation) for redirected stdout. If VS CRT doesn't do it this way, it means writing your own little printf. But since it's only really necessary for printing out the file paths, you can do without formatting specifiers etc.

Perhaps I ought to access the repository and actually DO something

jmj · « **Reply #78 on:** December 29, 2007, 05:25 AM »

It's not necessary to use wsprintf, until it's something which works.

You can try to fix it if you wish. If you haven't noticed, new address for the repository is http://svn.locate32.net/locate32. The repository can be read by anyone and your old password should work for writing.

Armando · « **Reply #79 on:** December 29, 2007, 12:44 PM »

Armando: I'm using locate 3.0 build 7.12260, from the x64 build.
-f0dder (December 27, 2007, 06:33 PM)

Thanks. so you in the x64 world...

So where does the cure for diacritics lies ?

f0dder · « **Reply #80 on:** December 29, 2007, 12:54 PM »

It's not necessary to use wsprintf, until it's something which works.

You can try to fix it if you wish. If you haven't noticed, new address for the repository is http://svn.locate32.net/locate32. The repository can be read by anyone and your old password should work for writing.
-jmj (December 29, 2007, 05:25 AM)

Yeah, works, just checked out source from repo.

I'm having trouble building it though, but I guess building your foundation-class-thingy library and adding it to %INCLUDE and %LIBS will fix that

jmj · « **Reply #81 on:** December 29, 2007, 01:16 PM »

f0dder: I sent you an email.

Armando · « **Reply #82 on:** December 29, 2007, 01:25 PM »

I mentioned some problems with diacritics and farr in the past (and I think it's been added to the official bug list):

https://www.donation...05.msg82218#msg82218

concerning the locate32 plugin, I'm having other problems, like trying to find filenames when they include characters like _ +

Here are some screenshots (the ! in the farr entry field is just my locate 32 trigger... faster than to write "loc").

I was looking for a gtd diagram. So i typed : ! gtd
And this what I got.

Screenshot - 2007-12-29 , 14_03_07.jpg

But what I was really looking for is

Screenshot - 2007-12-29 , 14_03_35.jpg

I tried different combinations, and it seems like certain characters (_ + and others) just completely messes the search.

Also, if I write __ __ without the locate plugin here's what farr displays

Screenshot - 2007-12-29 , 14_17_27.jpg

And with the locate plugin :

Screenshot - 2007-12-29 , 14_18_45.jpg

note that my locate plug-in is set to display 39 results and not 26, so this is not the problem here. But there might be a syntax problem that I'm not getting... does someone know what's happening ?

Okke ?

losiek · « **Reply #83 on:** December 29, 2007, 04:07 PM »

I'm having trouble building it though, but I guess building your foundation-class-thingy library and adding it to %INCLUDE and %LIBS will fix that
-f0dder (December 29, 2007, 12:54 PM)

If you still have problems with compling this - download source from this page: http://locate32.net/content/view/20/33/. It has necessary 3rd party libs and headers needed for compilation.
I played a little with writing results to a file and I finally forced locate to write in full Unicode:) But I'm rather inexperienced programmer and I used some dirty tricks to achieve that.
Maybe someone in the future will write a plug-in that gets results directly from Locate database;) On the other hand - we *just* have to wait until FARR will have built-in indexing. That will make this plug-in unnecessary... to beautiful to be real.

EDIT:
I'm soo lame :/ (or maybe Windows is?) In order to write wide characters correctly file need to be opened in binary mode. Then fwprintf works perfectly. No dirty hacks are needed.

mouser · « **Reply #84 on:** December 29, 2007, 05:06 PM »

Maybe someone in the future will write a plug-in that gets results directly from Locate database;

i think that this would be a great idea.. since the locate source code is available, it could be done and would bring indexed searching directly into FARR. if the plugin is written in C++ i could even incorporate the code into the main FARR codebase.

f0dder · « **Reply #85 on:** December 29, 2007, 05:46 PM »

losiek: hm, perhaps fwprintf internally detects between console/redirect and chooses the appropriate WriteConsoleW/WriteFile API? Remember, for locate.exe, we ideally want to support unicode both when used from a command prompt and when redirected to a pipe (the FARR plugin).

Dunno if writing a separate plugin directly accessing the locate databases would be that much of an advantage over piping locate.exe, once the unicode issue is fixed...

mouser · « **Reply #86 on:** December 29, 2007, 05:52 PM »

Dunno if writing a separate plugin directly accessing the locate databases would be that much of an advantage over piping locate.exe

sure it would, and it would let you:
1) avoid the startup overhead each time you did a search
2) be able to search on the fly as you type

f0dder · « **Reply #87 on:** December 29, 2007, 05:59 PM »

startup overhead is very very very small for locate.exe. You know how I usually bitch a lot about those matters where other people don't care

, and I don't feel any problems with the current method, even though locate.exe weighs in at a whopping 333kb here.

Search on the fly as you type, that's already being done with the plugin right now, although it could perhaps run slightly more efficient with a "full" plugin.

Armando · « **Reply #88 on:** December 29, 2007, 07:11 PM »

BTW, anybody can help me with the issue mentioned in reply 82 (https://www.donation...01.msg93405#msg93405)

losiek · « **Reply #89 on:** December 29, 2007, 08:15 PM »

f0dder:
OK, so everything I know at the moment:
1. Console in Windows XP can produce full Unicode output but only for displaying. Piping and redirecting is unsupported so it's useless for us.
2. Standard printing functions like wprintf (used in locate.exe) uses built-in WideCharToMultibyte conversion (to DOS codepage in my case), so polish letters have wrong hexcodes in normal windows.
3. I'm able to write raw Unicode to console (swprintf+WriteFile, without widechar->codepage conversion) - when displaying it look like there are spaces between all letters, but after redirecting to a file it works like a charm (even without BOM

).

So everything we need now is widechar implementation of Locate plug-in. I don't have Delphi so I can't play with that

Anybody have seen Okke recently? Maybe you have enough time to make plug-in with integrated Locate

Armando:
I think it's unresolvable at the moment. We have to wait for fixes or full Unicode version of this plug-in.

Armando · « **Reply #90 on:** December 29, 2007, 09:06 PM »

So you think that characters like "_" or "+" affect the search because of incomplete Unicode compatibility ?

jmj · « **Reply #91 on:** December 30, 2007, 01:08 AM »

f0dder:
OK, so everything I know at the moment:
1. Console in Windows XP can produce full Unicode output but only for displaying. Piping and redirecting is unsupported so it's useless for us.
2. Standard printing functions like wprintf (used in locate.exe) uses built-in WideCharToMultibyte conversion (to DOS codepage in my case), so polish letters have wrong hexcodes in normal windows.
3. I'm able to write raw Unicode to console (swprintf+WriteFile, without widechar->codepage conversion) - when displaying it look like there are spaces between all letters, but after redirecting to a file it works like a charm (even without BOM ).
-losiek (December 29, 2007, 08:15 PM)

So locate.exe could be changed to use WriteConsole (if not redirected) and add e.g. '-w' command line argument to use WriteFile and Unicode character set. Then plugin could use that.

But does f0dder want to work with this or do I make required changes?

f0dder · « **Reply #92 on:** December 30, 2007, 06:09 AM »

jmj: I want to play around with it a bit, as soon as I get the thing fully building

. Is the filename output done exclusively from the two callback functions?

losiek: things are a bit different than that

. WriteConsole can write unicode characters because internally there's two versions of it: *A and *W, one for ansi and one for wide/unicode. WriteFile doesn't have ansi/unicode versions, and simply writes to the target device. There's no way for the console device to know what format it receives from WriteFile, so it opts to only support MultiByte...

There shouldn't be any problem using raw WriteFile when you detect output has been redirected. Or well, THE problem is that you're not outputting a BOM marker, which will confuse e.g. notepad, and that some apps might not be prepared to handle unicode text. The best is probably to convert to UTF-8 when output is redirected...

jmj · « **Reply #93 on:** December 30, 2007, 06:50 AM »

jmj: I want to play around with it a bit, as soon as I get the thing fully building . Is the filename output done exclusively from the two callback functions?
-f0dder (December 30, 2007, 06:09 AM)

Yes. The another is for filenames using ANSI, so you don't need to change that (unless you wish it).

f0dder · « **Reply #94 on:** December 30, 2007, 06:59 AM »

ANSI output should probably be left as-is; I dunno if WriteConsole has any advantage over WriteFile except for being able to output unicode properly to the console. Sure, probably less conversion layers to go through than WriteFile, but I doubt it's anything measurable.

Need to do some final pre-newyear shopping now, will look at code when I get home.

losiek · « **Reply #95 on:** December 30, 2007, 02:19 PM »

Armando:
No, I don't think that yours bugs are caused directly by lack of Unicode support, but I think that they're related. It seems that your bug work very similar to my ' (accent) bug. Can you check Location column in "loc +" results? Are there two (or more) paths for one file?

f0dder:
OK, so everybody is waiting now for your Unicode plug-in. Let's hope it will resolve other bugs also

Good luck.

Armando · « **Reply #96 on:** December 30, 2007, 04:46 PM »

Thanks losiek. I'll try that latter.
BTW... Where's Okke ?

f0dder · « **Reply #97 on:** December 30, 2007, 04:54 PM »

Okay, with a bunch of help from jmj, I managed to build locate... took a bit of effort, it's a semi-large and semi-complex project, also regarding library dependencies and such

Hacking up locate.exe to user either WriteConsoleW or WriteFile was easy, and I now have a version that does proper unicode output on console, and UTF-8 output when redirected, seems to work fine. The following screenshot shows what standard locate.exe outputs, and then what my quick hack outputs. Note that you need to set your console to use a non-raster font for this to work!

Locate32 Plugin for FARR by Okke

Note that there's no reason for me to release this new executable before Okke implements UTF-8 support, since you'll get pretty garbled output with the current plugin version

Locate32 Plugin for FARR by Okke

In case anyone is wondering what the text is, it's just snippets from a russian article I found about fSekrit, seemed fitting to use when I was testing unicode support in fSekrit, and it's the best unicode test snippet I have at all, since I know none of those characters are available in my own codepage

Armando · « **Reply #98 on:** December 30, 2007, 06:25 PM »

Nice work f0dder.
Any news from Okke?

f0dder · « **Reply #99 on:** December 30, 2007, 07:28 PM »

I guess I could reverse-engineer Okke's DLL and make it support the utf-8/unicode locate hack, I've already located the "spawn child process and grab output", which is referenced three times, but Delphi code is so naaaasty to deal with

EDIT: downloaded the FARR SDK, and realized the locate32 plugin source code is there, woo-hoo

. The plugin will need some updating (should be simple though), but it seems that FARR itself needs a bunch of updates to handle unicode. Working with mouser on the issue right now.