MD5Hash 2.9

[MilesAhead]

MD5Hash 1.0 is an x64 only Dialog app to create MD5Sum for files.

I developed it on Vista Home Premium SP1 64 bit.

You may download from this page:
http://www.favessoft.com/downloads.html


Drag & drop one or more files on the dialog.
Or, to process an individual file, type in the
filename or use the Browse Button, then press
OK.

Cancel closes the app, or if it is busy processing, will
prompt to abort the operations.

See the included Readme for particulars.  But, basically,
it's stand-alone. You may just copy to a directory in your
path as it makes no registry settings or .ini files.

It's not the fastest checksum app out there. On my quad core
Core Temp shows about 50% to 60% core usage out of a possible
400%.  But it's faster than my old FileCRC32 without bogging down
your machine.

(thanks to Elmue on Code Project for the md5 c++ algorithm
http://www.codeproje...curity/cryptest.aspx)

[MilesAhead]

MD5Hash 1.1 Now pressing F1 key shows About Box.

[MilesAhead]

MD5Hash 1.2 I removed the About Button since F1 now handles that chore, and added a Clear Button.  If there are any results in the edit box, it will prompt you to erase them. I updated the screen shot in the first post.

edit: I updated without changing the version number just so that if the results edit box is empty, the Clear Button is disabled.  The update is on the page for download.

[MilesAhead]

MD5Hash 1.3 Now folders are filtered from a file drop, instead of quitting with an error. If folders are found in the drop a count is displayed in a message box after processing completes.  Trying to process a single folder using OK Button displays an error dialog.

[f0dder]

Why MD5?
Why x64-only?

As for "not the fastest checksummer", how do you do your file I/O? in case you aren't using either, I'd suggest testing with both memory-mapped files (which are both over- and under-appreciated) as well as overlapped (async) I/O.

[MilesAhead]

Why MD5?
Why x64-only?

As for "not the fastest checksummer", how do you do your file I/O? in case you aren't using either, I'd suggest testing with both memory-mapped files (which are both over- and under-appreciated) as well as overlapped (async) I/O.

-f0dder (September 23, 2010, 01:58 PM)

md5 because that's what I use.
x64 only because I already have FileCRC32 32 bit that does(surprise) crc32 and md5 sum.

It's not the fie i/o.  To make it faster I'd need a thread-safe md5 routine.  Also that complicates updating the UI for huge files and is more prone to crashing.  The program uses about 1/2 a core on my system. Not everyone wants to max out all cores to do download checksums.

btw you code so do you have anything for download?
edit: ok, I see the page now.

btw I already see a bunch of free checksum type programs that run right in the shell extension(or at least seem to.)  Most seem to want to function at max speed.  I see no reason to try to duplicate free programs with a design that doesn't interest me.  I tried 2 or 3 and when multiple files are selected sized a GB or so, at least with the ones I tried, they went away after a few seconds of furious processing. I'm sure there are some that work.  But to me, for my usage, it's a background task.  If the usage was for some other purpose then the design would be different.

I've used the same method in FileCRC32 adapted to this program.  In years of use I don't recall it crapping out because the files were large.  It may be a tortoise, but it gets there with the right results.

[f0dder]

You say it's not the I/O, but that you're only running at about half a core - that indicates that you are I/O bound, or are block-waiting on something else. Async I/O lets you do you disk reads and hash computation in parallel, and I believe MMF should allow the same (you have less control and take a slight (as in should-be-almost-unmeasurable on modern CPUs) CPU hit from it, though). You'll still ultimately be I/O bound, but since you can overlap CPU and I/O you can possibly shave off a bit of execution time.

I don't think MD5 itself can be parallelized across threads, that would kind of defeat the "every input bit should affect every output bit" goal of cryptographic hashes.

[MilesAhead]

I'm familiar with asynchronous i/o. I was not thinking of parallelizing one md5 computation across threads but having an instance of an md5 calculator processing a file for each thread/core.  But, as I say, as it is now, I can use the machine while the program is processing files 8+ GB in size with confidence it will complete correctly.  Why would I want to sit and watch a bunch of md5 sums being calculated? I'd rather, say, look for and download more files while the calculations are carried out.

In any case, for this application, the simpler design is likely better as the user is most likely not going to drop 512 files on the app to process in parallel.  More likely 4 or 5 files will be dropped on the program, then something else, like surfing the net will take place in the foreground.

[f0dder]

Processing multiple files in parallel will only slow down the operation since you'll introduce a fair amount of read/write head movements - unless of course you have the files to sum on different physical drives, but how often does that usecase happen?

The argument for async I/O isn't to try and get full core utilization; it isn't something you can achieve with a nonparallizable algorithm, and I agree it's not something you want for this application anyway. But if you can get a bit closer to full utilization of one one by overlapping reads and computation, the system is still usable but the summing task will finish faster.

I've actually been meaning to do some performance benchmarking for various reading methods with hashing in mind for a short while, but unfortunately haven't found the time to do so yet

[MilesAhead]

I'd like to have a multi-thread safe/instance safe/ md5 class just in case I ever found a use for it.

But on the file i/o I guess I would want to test drive it before making assumptions. If I introduce multi-tasking elements then I have to discriminate between tasks too trivial and those that need it.  So now I have a single-threaded pipe for the trivial stuff and a multi-threaded pipe for the big jobs all of which have to track UI updates so the user knows the 8 GB file isn't hung.  For this case it's way too much work for little gain.

But I'm sure those guys who process video are doing aync i/o as I can see the HD honkin' while they're fixing video bits.

I should have elaborated in first post that people who have used those super fast multi-threaded hash grinders should not expect that approach with this app. I wasn't really lamenting the lack of speed.  Just by implementing a similar hash in 64 bit with the same drag & drop processing scheme I think I got about a 52% increase over the old 32 bit version.  I processed the same file back and forth a couple of times to take file caching out of the result.

edit: just for grins I processed a file a bit less than 8 GB from one external drive, then a file very close to the same size from a faster external.  It made quite an improvement. So that supports your assertion.  I tended to discount it because I tried increasing the file buffer from 1 MB to 4 earlier and didn't detect an improvement.  Eyeballing it can be deceptive.

But this run indicates it is waiting for input.

[MilesAhead]

btw, thanks for the pointer in the right direction. I'm going to cobble together an advanced version.  I know how I'm going to do it now.  A producer/consumer.  For files of size, the producer will load a queue of file chunks and the consumer will pop the chunks, do the md5 calc, and update the progressbar. It will give me an excuse to use parallel_invoke and the supposedly thread-safe concurrent queue thingy.  Should be fun.

[MilesAhead]

Well, it was fun to play around with concurrent_queue, but the application is anchored by the speed the disk can serve up the file. Pulling from my fastest drive not doing anything else, an 8 GB or so file takes just about 1 min 4 sec with the threaded and non-threaded versions.  Played around with various disk buffer sizes etc... but it looks like it does the calculation as fast as the disk can serve it up.

The downsize is, the threaded version spins its wheels a bit waiting for data and spins up my fans with no increase in output. Instead of using about 50 to 60% tops in Core Temp, I'd use about 1.25 cores.  Just noisier.

The multi-thread version would go a bit faster if I let the UI freeze.  Try to drag the window and you get a 2 second delay.  Once I inserted some process message loops to get the responsiveness back, it ran just like the single threaded version.

It looks like the multi-thread version would be best for a command line implementation.  Hit enter and when you get the prompt back, there's the MD5.

[MilesAhead]

MD5Hash 1.4  Rather than using error dialogs most feedback is now returned to the EditBox.  For example, if a "file" turned out to be a folder, IS FOLDER will show where the MD5 sum would be.  If the file was 0 length, ZERO LENGTH or something similar will show.  CAPS are used to make the errors stand out.  This way, one file with zero length in the drag & drop won't halt the job.

edit: I also changed the About Box so that No is the default button.. avoiding trips on the web due to hitting Enter.

[f0dder]

Hm, 60%->125% by threading (without any performance advantage) sounds like there's something being done wrong - like busy-waiting instead of block-waiting. How did you design the threading stuff, what is the "work item"? I assume the language is C++ native?

[MilesAhead]

Hm, 60%->125% by threading (without any performance advantage) sounds like there's something being done wrong - like busy-waiting instead of block-waiting. How did you design the threading stuff, what is the "work item"? I assume the language is C++ native?

-f0dder (September 25, 2010, 04:20 PM)

I tried several things.  But it occurs to me the threading is only going to be advantageous if the app is cpu intensive.  As it is, it can do the calculations in a snap.  It's waiting on the HD no matter the scenario. If I had a ram disk then it might be different.

I did parallel_invoke, then just using _beginthread to feed the file data queue.  Then I did 2 worker threads, the producer, the consumer, and the UI thread did the updating of the gui.  All scenarios with all disk buffer sizes, different schemes for releasing time slice etc.. resulted in 1 min 6 seconds to process a bit under 8 GB.  The 1.25 core was worst case when I just let the consumer thread churn waiting for file data.

I think 8 GB in a minute 6 seconds is pretty close to HD capacity.
The file serving is the determining factor. The app is not waiting
on the calculation.  If you increase the file buffer the calc takes
a bit longer but the disk pull takes a lot longer.  If you go with smaller file
chunks, you do more calc calls on smaller chunks of data.

HD is cows ear I think is the moral of the story.
If it was a lengthy calculation then loading the data in the
background may help.  But in this case I think the md5 calc
part is negligible.

On the plus side, I learned to to do parallel_invoke with lamdas.

[MilesAhead]

This is interesting. I compiled a 32 bit version and it's just as fast.  The md5 routine has asm code for 32 bit.  But both are much faster than my old FileCRC32.  So I'll include the 32 bit in the download.

edit: the available download now includes both 32 and 64 bit versions.  At least on my quad the 32 bit version is just as fast, but the 64 bit is a tad easier on the CPU.  Both are a lot faster than FileCRC32. To be expected since that was implemented in Delphi 5.

Even with both versions the download is only about 78 KB.  



edit: concurrent_queue looks like a good means of emptying  the drop loop quickly though.

[MilesAhead]

MD5Hash 1.5 Both 64 and 32 bit versions have been updated to 1.5.

Without radically changing the design I reimplemented the drag & drop processing.  Now you can drag & drop one or more files onto the program even if it is in the middle of processing the previous drop.

However, currently, if you select a single file to process by typing in the file name/using the Browse Button, and pressing OK Button, it will block until completion. In either case you may press the Cancel Button or the Close 'x' in the corner of the window to bring up an abort dialog.

[MilesAhead]

MD5Hash 1.6 Both versions have been updated.  The Browse Button now allows multiple file selection.  Multiple files are added to the job queue. If only a single file is selected, it's inserted into the edit box. Pressing OK Button will process the lone file in blocking mode.  After a multiple file selection or drag & drop, you may still add files to the queue via drag & drop.

Minor fixes such as F1 not displaying the About Box after a job has been run.  Seems dialog boxes are quirky about processing WM_HELP unless you force the mouse click someplace receptive.  So you may notice after jobs have completed the cursor is back in the top edit box.  A kludge but it works.

edit: the zip file online has been updated. Programs are the same. The Readme.txt files were updated.

[MilesAhead]

MD5Hash 1.7 Both versions updated to 1.7.

Some minor bug fixes.

The input has been reworked.

When individual files are processed they are added to the job queue instead of blocking. Now you can type in the filename of a large file or select file(s) with the Browse Button and still add files to the queue using drag & drop.

Fixed a bug where the OK button lost default pushbutton status after running a job. (Typing in a filename and hitting Enter only worked if it was the first job after program load.)

edit: I didn't up the version number, but the Clear Button prompt now offers to save the cut to the clipboard.  Yes cuts all text to clipboard, No just empties the EditBox, Cancel takes no action. The latest build is available for download at the link in the first post.

[MilesAhead]

MD5Hash 1.8 Both exes updated to 1.8.

Added logic so that OK Button is only enabled when something is in the edit line.

The new version has been uploaded.

[f0dder]

The 1.25 core was worst case when I just let the consumer thread churn waiting for file data.

Sounds like you did a spinning instead of blocking wait for file data - this is bad

The file serving is the determining factor. The app is not waiting
on the calculation.

Yes, as long as you're not CPU bound. Overlapping reads and computation will give you the biggest advantage when processing also takes some time. Problem with today's CPUs: we programmers get lazy :p

But both are much faster than my old FileCRC32.

I guess the algorithm used there wasn't table-based?

edit: concurrent_queue looks like a good means of emptying  the drop loop quickly though.

As a building block - you definitely don't want to be doing "try_pop" in a loop with nothing else, since you'll burn a lot of CPU doing that. Needs to be combined with an event you can block-wait on in case the queue is empty.

[MilesAhead]

As a building block - you definitely don't want to be doing "try_pop" in a loop with nothing else, since you'll burn a lot of CPU doing that. Needs to be combined with an event you can block-wait on in case the queue is empty.

-f0dder (September 28, 2010, 04:07 PM)

I don't wait at all. I reimplemented it so that the user input side just adds the file names to the queue and quits.  If the input routines detect the consumer queue handler isn't running, they post a message to start it at the end of the routine(non-blocking.)  Once the consumer has emptied the queue, he quits.

The file drop is used sort of like an interrupt routine in the old Dos.  Using the concurrent_queue and a couple of globals with Interlocked calls I can tell if the consumer is running when adding to the queue etc..

If you really wanted to maximize the throughput to the nth degree then smaller disk buffers with a rotating buffer scheme, so that somehow the disk can be read more in the background, might be it.  If a solid state drive was hooked up it might benefit.  But the only thing that seems to speed it up is a faster drive.  Looks like there's only so much blood in the stone.

But I like this design much better.  You can be processing and still add to the queue with drag & drop. I suppose I could work it so that the file browser and other stuff could be active but it just seems to me like the type of application where you download 5 files and check the MD5 to make sure you got 'em ok. It's not designed as a folder contents monitor.

Afa the old Delphi app, the CRC and MD5 routines were table-based with inline asm statements.  But Delphi was made to compile fast more than to run fast. At least I know 5 was. That was the last version I bought.  Hit the button, one pass compile, run, find bug.. repeat.  Plus the compiler technology is over 10 years old.

edit: vc++ 2010 is definitely much more fun than vc++ 6 though. When I first started with C# 1.0 Standard I hated that real-time lint thing in the editor. I like it now though.  Points up the typos as they happen.  Plus the new language features are cool.  I'm surprised how easy 64 bit is to do out of the IDE in Express with the 7.1 SDK added on.  Only thing is sometimes if you change the active project in the editor it doesn't really change in the properties.  You have to go in and check it.  But that's a quibble.  I don't think I really want to do forms based apps in it though.  Everyone online says do the UI in C# and the engine in vc++.

[MilesAhead]

MD5Hash 2.0 The main enhancement in v. 2.0 is Folder handling and optional entry in the Explorer Context Menu.

A single filename or folder name may be specified on the command line.  Folders contained in the specified folder are ignored.  Only normal files are added to the job queue.  Folders may also be included in the drag & drop.

The download includes AutoIt3 scripts to add and remove MD5Hash from the Context Menu as a file and folder handler. If multiple items are selected Explorer will launch an instance for each item. This is convenient for processing several small files.  But for large files, such as multi-GB videos, I suggest using drag & drop and a single instance of MD5Hash to avoid hard drive thrashing.

There's a readme file included with directions on script usage.
Enjoy.

edit: a bug in the context menu removal script was fixed. Please refresh with the latest download.

[MilesAhead]

MD5Hash 2.1 Subfolders are now processed recursively.  The max size of the job queue is set to 4096 files.  Also the results edit box was replaced with a Rich Edit to remove the line restrictions on results.

According to the MS chart for RichEdit, if you have XP SP1 or newer the dll for RichEdit 4.1 should already be on the system.

[MilesAhead]

MD5Hash 2.2 Bug fix release.  Progress bar no longer used on small files. Fixed caption bar update.