Hm, 60%->125% by threading (without any performance advantage) sounds like there's something being done wrong - like busy-waiting instead of block-waiting. How did you design the threading stuff, what is the "work item"? I assume the language is C++ native?
I tried several things. But it occurs to me the threading is only going to be advantageous if the app is cpu intensive. As it is, it can do the calculations in a snap. It's waiting on the HD no matter the scenario. If I had a ram disk then it might be different.
I did parallel_invoke, then just using _beginthread to feed the file data queue. Then I did 2 worker threads, the producer, the consumer, and the UI thread did the updating of the gui. All scenarios with all disk buffer sizes, different schemes for releasing time slice etc.. resulted in 1 min 6 seconds to process a bit under 8 GB. The 1.25 core was worst case when I just let the consumer thread churn waiting for file data.
I think 8 GB in a minute 6 seconds is pretty close to HD capacity.
The file serving is the determining factor. The app is not waiting
on the calculation. If you increase the file buffer the calc takes
a bit longer but the disk pull takes a lot longer. If you go with smaller file
chunks, you do more calc calls on smaller chunks of data.
HD is cows ear I think is the moral of the story.
If it was a lengthy calculation then loading the data in the
background may help. But in this case I think the md5 calc
part is negligible.
On the plus side, I learned to to do parallel_invoke with lamdas.