Main Area and Open Discussion > General Software Discussion

MD5Hash 2.9

<< < (3/9) > >>

MilesAhead:
btw, thanks for the pointer in the right direction. I'm going to cobble together an advanced version.  I know how I'm going to do it now.  A producer/consumer.  For files of size, the producer will load a queue of file chunks and the consumer will pop the chunks, do the md5 calc, and update the progressbar. It will give me an excuse to use parallel_invoke and the supposedly thread-safe concurrent queue thingy.  Should be fun. :)

MilesAhead:
Well, it was fun to play around with concurrent_queue, but the application is anchored by the speed the disk can serve up the file. Pulling from my fastest drive not doing anything else, an 8 GB or so file takes just about 1 min 4 sec with the threaded and non-threaded versions.  Played around with various disk buffer sizes etc... but it looks like it does the calculation as fast as the disk can serve it up.

The downsize is, the threaded version spins its wheels a bit waiting for data and spins up my fans with no increase in output. Instead of using about 50 to 60% tops in Core Temp, I'd use about 1.25 cores.  Just noisier.

The multi-thread version would go a bit faster if I let the UI freeze.  Try to drag the window and you get a 2 second delay.  Once I inserted some process message loops to get the responsiveness back, it ran just like the single threaded version.

It looks like the multi-thread version would be best for a command line implementation.  Hit enter and when you get the prompt back, there's the MD5.

MilesAhead:
MD5Hash 1.4  Rather than using error dialogs most feedback is now returned to the EditBox.  For example, if a "file" turned out to be a folder, IS FOLDER will show where the MD5 sum would be.  If the file was 0 length, ZERO LENGTH or something similar will show.  CAPS are used to make the errors stand out.  This way, one file with zero length in the drag & drop won't halt the job.

edit: I also changed the About Box so that No is the default button.. avoiding trips on the web due to hitting Enter. :)

f0dder:
Hm, 60%->125% by threading (without any performance advantage) sounds like there's something being done wrong - like busy-waiting instead of block-waiting. How did you design the threading stuff, what is the "work item"? I assume the language is C++ native?

MilesAhead:
Hm, 60%->125% by threading (without any performance advantage) sounds like there's something being done wrong - like busy-waiting instead of block-waiting. How did you design the threading stuff, what is the "work item"? I assume the language is C++ native?
-f0dder (September 25, 2010, 04:20 PM)
--- End quote ---

I tried several things.  But it occurs to me the threading is only going to be advantageous if the app is cpu intensive.  As it is, it can do the calculations in a snap.  It's waiting on the HD no matter the scenario. If I had a ram disk then it might be different.

I did parallel_invoke, then just using _beginthread to feed the file data queue.  Then I did 2 worker threads, the producer, the consumer, and the UI thread did the updating of the gui.  All scenarios with all disk buffer sizes, different schemes for releasing time slice etc.. resulted in 1 min 6 seconds to process a bit under 8 GB.  The 1.25 core was worst case when I just let the consumer thread churn waiting for file data.

I think 8 GB in a minute 6 seconds is pretty close to HD capacity.
The file serving is the determining factor. The app is not waiting
on the calculation.  If you increase the file buffer the calc takes
a bit longer but the disk pull takes a lot longer.  If you go with smaller file
chunks, you do more calc calls on smaller chunks of data.

HD is cows ear I think is the moral of the story. :)
If it was a lengthy calculation then loading the data in the
background may help.  But in this case I think the md5 calc
part is negligible.

On the plus side, I learned to to do parallel_invoke with lamdas. :)

Navigation

[0] Message Index

[#] Next page

[*] Previous page