Yes, as PlatformSDK states, WriteConsole will fail when used with redirection... so that's no good. If you redirect file, GetStdHandle() returns a file handle, so things should work fine. I think/hope notepad fails because the file lacks a BOM marker, but I'll do some testing, also of the code from the blog...

At the plugin side, I guess it comes down to whether you want to parse utf-8 or use ucs-16 directly. For locate, it's probably some printf calls that need to be changed to more specific code.

Ok, it seems that unicode characters do not work even in console. It worked when I tested before, but probable
used copepage was valid for those charactes (those was cyrillic or chinese alphapets, now I tested random characters).

Hm, it did work for my test, but all I used was some russian text. I did have to choose a non-raster font for the console, though. My codepage is 850, I doubt cyrillic characters are valid in that codepage?

So what all I should do to get it work like you want? Atm locate.exe uses wprintf

I guess we should take a look at the CRT implementation of wprinf (source comes with visual studio) and see how they handle it... but it breaks down to using WriteConsoleW for consoles, and WriteFile (possibly with wide->multibyte translation) for redirected stdout. If VS CRT doesn't do it this way, it means writing your own little printf. But since it's only really necessary for printing out the file paths, you can do without formatting specifiers etc.

Perhaps I ought to access the repository and actually DO something

Armando: I'm using locate 3.0 build 7.12260, from the x64 build.

Thanks. so you in the x64 world...

So where does the cure for diacritics lies ? 

f0dder: I sent you an email.

I'm having trouble building it though, but I guess building your foundation-class-thingy library and adding it to %INCLUDE and %LIBS will fix that

If you still have problems with compling this - download source from this page: http://locate32.net/content/view/20/33/. It has necessary 3rd party libs and headers needed for compilation.
I played a little with writing results to a file and I finally forced locate to write in full Unicode:) But I'm rather inexperienced programmer and I used some dirty tricks to achieve that.
Maybe someone in the future will write a plug-in that gets results directly from Locate database;) On the other hand - we *just* have to wait until FARR will have built-in indexing. That will make this plug-in unnecessary... to beautiful to be real.

EDIT:
I'm soo lame :/ (or maybe Windows is?) In order to write wide characters correctly file need to be opened in binary mode. Then fwprintf works perfectly. No dirty hacks are needed.

losiek: hm, perhaps fwprintf internally detects between console/redirect and chooses the appropriate WriteConsoleW/WriteFile API? Remember, for locate.exe, we ideally want to support unicode both when used from a command prompt and when redirected to a pipe (the FARR plugin).

Dunno if writing a separate plugin directly accessing the locate databases would be that much of an advantage over piping locate.exe, once the unicode issue is fixed...

startup overhead is very very very small for locate.exe. You know how I usually bitch a lot about those matters where other people don't care , and I don't feel any problems with the current method, even though locate.exe weighs in at a whopping 333kb here.

Search on the fly as you type, that's already being done with the plugin right now, although it could perhaps run slightly more efficient with a "full" plugin.

f0dder:
OK, so everything I know at the moment:
1. Console in Windows XP can produce full Unicode output but only for displaying. Piping and redirecting is unsupported so it's useless for us.
2. Standard printing functions like wprintf (used in locate.exe) uses built-in WideCharToMultibyte conversion (to DOS codepage in my case), so polish letters have wrong hexcodes in normal windows.
3. I'm able to write raw Unicode to console (swprintf+WriteFile, without widechar->codepage conversion) - when displaying it look like there are spaces between all letters, but after redirecting to a file it works like a charm (even without BOM ).

So everything we need now is widechar implementation of Locate plug-in. I don't have Delphi so I can't play with that Anybody have seen Okke recently? Maybe you have enough time to make plug-in with integrated Locate

Armando:
I think it's unresolvable at the moment. We have to wait for fixes or full Unicode version of this plug-in.

f0dder:
OK, so everything I know at the moment:
1. Console in Windows XP can produce full Unicode output but only for displaying. Piping and redirecting is unsupported so it's useless for us.
2. Standard printing functions like wprintf (used in locate.exe) uses built-in WideCharToMultibyte conversion (to DOS codepage in my case), so polish letters have wrong hexcodes in normal windows.
3. I'm able to write raw Unicode to console (swprintf+WriteFile, without widechar->codepage conversion) - when displaying it look like there are spaces between all letters, but after redirecting to a file it works like a charm (even without BOM ).

So locate.exe could be changed to use WriteConsole (if not redirected) and add e.g. '-w' command line argument to use WriteFile and Unicode character set. Then plugin could use that.

But does f0dder want to work with this or do I make required changes?

jmj: I want to play around with it a bit, as soon as I get the thing fully building . Is the filename output done exclusively from the two callback functions?

Yes. The another is for filenames using ANSI, so you don't need to change that (unless you wish it).

Armando:
No, I don't think that yours bugs are caused directly by lack of Unicode support, but I think that they're related. It seems that your bug work very similar to my ' (accent) bug. Can you check Location column in "loc +" results? Are there two (or more) paths for one file?

f0dder:
OK, so everybody is waiting now for your Unicode plug-in. Let's hope it will resolve other bugs also Good luck.

Okay, with a bunch of help from jmj, I managed to build locate... took a bit of effort, it's a semi-large and semi-complex project, also regarding library dependencies and such

Hacking up locate.exe to user either WriteConsoleW or WriteFile was easy, and I now have a version that does proper unicode output on console, and UTF-8 output when redirected, seems to work fine. The following screenshot shows what standard locate.exe outputs, and then what my quick hack outputs. Note that you need to set your console to use a non-raster font for this to work!



Note that there's no reason for me to release this new executable before Okke implements UTF-8 support, since you'll get pretty garbled output with the current plugin version



In case anyone is wondering what the text is, it's just snippets from a russian article I found about fSekrit, seemed fitting to use when I was testing unicode support in fSekrit, and it's the best unicode test snippet I have at all, since I know none of those characters are available in my own codepage

I guess I could reverse-engineer Okke's DLL and make it support the utf-8/unicode locate hack, I've already located the "spawn child process and grab output", which is referenced three times, but Delphi code is so naaaasty to deal with

EDIT: downloaded the FARR SDK, and realized the locate32 plugin source code is there, woo-hoo . The plugin will need some updating (should be simple though), but it seems that FARR itself needs a bunch of updates to handle unicode. Working with mouser on the issue right now.