Main Area and Open Discussion > General Software Discussion

MD5Hash 2.9

<< < (5/9) > >>

f0dder:
The 1.25 core was worst case when I just let the consumer thread churn waiting for file data.
--- End quote ---
Sounds like you did a spinning instead of blocking wait for file data - this is bad :)

The file serving is the determining factor. The app is not waiting
on the calculation.
--- End quote ---
Yes, as long as you're not CPU bound. Overlapping reads and computation will give you the biggest advantage when processing also takes some time. Problem with today's CPUs: we programmers get lazy :p

But both are much faster than my old FileCRC32.
--- End quote ---
I guess the algorithm used there wasn't table-based?

edit: concurrent_queue looks like a good means of emptying  the drop loop quickly though.
--- End quote ---
As a building block - you definitely don't want to be doing "try_pop" in a loop with nothing else, since you'll burn a lot of CPU doing that. Needs to be combined with an event you can block-wait on in case the queue is empty.

MilesAhead:
As a building block - you definitely don't want to be doing "try_pop" in a loop with nothing else, since you'll burn a lot of CPU doing that. Needs to be combined with an event you can block-wait on in case the queue is empty.
-f0dder (September 28, 2010, 04:07 PM)
--- End quote ---

I don't wait at all. I reimplemented it so that the user input side just adds the file names to the queue and quits.  If the input routines detect the consumer queue handler isn't running, they post a message to start it at the end of the routine(non-blocking.)  Once the consumer has emptied the queue, he quits.

The file drop is used sort of like an interrupt routine in the old Dos.  Using the concurrent_queue and a couple of globals with Interlocked calls I can tell if the consumer is running when adding to the queue etc..

If you really wanted to maximize the throughput to the nth degree then smaller disk buffers with a rotating buffer scheme, so that somehow the disk can be read more in the background, might be it.  If a solid state drive was hooked up it might benefit.  But the only thing that seems to speed it up is a faster drive.  Looks like there's only so much blood in the stone.

But I like this design much better.  You can be processing and still add to the queue with drag & drop. I suppose I could work it so that the file browser and other stuff could be active but it just seems to me like the type of application where you download 5 files and check the MD5 to make sure you got 'em ok. It's not designed as a folder contents monitor.

Afa the old Delphi app, the CRC and MD5 routines were table-based with inline asm statements.  But Delphi was made to compile fast more than to run fast. At least I know 5 was. That was the last version I bought.  Hit the button, one pass compile, run, find bug.. repeat.  Plus the compiler technology is over 10 years old.

edit: vc++ 2010 is definitely much more fun than vc++ 6 though. When I first started with C# 1.0 Standard I hated that real-time lint thing in the editor. I like it now though.  Points up the typos as they happen.  Plus the new language features are cool.  I'm surprised how easy 64 bit is to do out of the IDE in Express with the 7.1 SDK added on.  Only thing is sometimes if you change the active project in the editor it doesn't really change in the properties.  You have to go in and check it.  But that's a quibble.  I don't think I really want to do forms based apps in it though.  Everyone online says do the UI in C# and the engine in vc++. :)

MilesAhead:
MD5Hash 2.0 The main enhancement in v. 2.0 is Folder handling and optional entry in the Explorer Context Menu.

A single filename or folder name may be specified on the command line.  Folders contained in the specified folder are ignored.  Only normal files are added to the job queue.  Folders may also be included in the drag & drop.

The download includes AutoIt3 scripts to add and remove MD5Hash from the Context Menu as a file and folder handler. If multiple items are selected Explorer will launch an instance for each item. This is convenient for processing several small files.  But for large files, such as multi-GB videos, I suggest using drag & drop and a single instance of MD5Hash to avoid hard drive thrashing. :)

There's a readme file included with directions on script usage.
Enjoy.

edit: a bug in the context menu removal script was fixed. Please refresh with the latest download.

MilesAhead:
MD5Hash 2.1 Subfolders are now processed recursively.  The max size of the job queue is set to 4096 files.  Also the results edit box was replaced with a Rich Edit to remove the line restrictions on results.

According to the MS chart for RichEdit, if you have XP SP1 or newer the dll for RichEdit 4.1 should already be on the system.

MilesAhead:
MD5Hash 2.2 Bug fix release.  Progress bar no longer used on small files. Fixed caption bar update.

Navigation

[0] Message Index

[#] Next page

[*] Previous page