Welcome Guest.   Make a donation to an author on the site October 25, 2014, 07:22:42 PM  *

Please login or register.
Or did you miss your validation email?


Login with username and password (forgot your password?)
Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.


You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.
 
Learn about the DonationCoder.com microdonation system (DonationCredits).
   
   Forum Home   Thread Marks Chat! Downloads Search Login Register  
Pages: [1]   Go Down
  Reply  |  New Topic  |  Print  
Author Topic: "Do copy acceleration utilities actually lower file transfer speeds?"  (Read 5580 times)
paulobrabo
Supporting Member
**
Posts: 87


The Brazilian Bomber

View Profile WWW Give some DonationCredits to this forum member
« on: August 10, 2012, 04:12:12 AM »

Hey guys, long time no talk, but lurking daily.

I'd love to hear your take on this: Samer from freewaregenius concludes that copy acceleration utilities on Windows make things actually worse:

http://www.freewaregenius...speeds-our-tests-say-yes/



What now? Do copy acceleration software works? From a technical standpoint, can it work?
Logged

English will never be my first language, it doesn't meter how hard I try.
mouser
First Author
Administrator
*****
Posts: 33,597



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #1 on: August 10, 2012, 06:26:34 AM »

Excellent report by Samer.

However, I will echo what some of the commenters on the post have said, which is that when I have used suched utilities such as SuperCopier and Terracopy, it's not because I wanted to speed up the absolute transfer time for copying a large number of files, it's because they are much more robust and flexible -- and fail much more gracefully -- wheras default windows copy will completely abort and die midway if a single file copy out of a thousand fails.
Logged
Renegade
Charter Member
***
Posts: 11,639



Tell me something you don't know...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #2 on: August 10, 2012, 06:33:49 AM »

Interesting... I would like to know HOW those utilities are doing it. Are they using a Windows API? Or have they written their own drivers?
Logged

Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #3 on: August 10, 2012, 06:36:01 AM »

Interesting... I would like to know HOW those utilities are doing it. Are they using a Windows API? Or have they written their own drivers?
Obviously using APIs, you'd have to be insane involving drivers in this.

But even at the API level, there's a ton of ways to go about it...

memory mapped files.
normal I/O, buffered vs. unbuffered, and chunksize.
async I/O, possibly using I/O Completion Ports (IOCP would be insane overkill for filecopy, but hey).

Also, a lot of stuff happened from XP to Vista. Pre-Vista, the maximum transfer size you'd get (can't remember if this was all the way down at DMA level, or "just" some kernel<>usermode thing) was 64kb. For Vista/Win2k8 this was increased to (iirc) 4MB for desktop versions and 64MB for server versions.
Logged

- carpe noctem
40hz
Supporting Member
**
Posts: 10,736



see users location on a map View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #4 on: August 10, 2012, 06:42:06 AM »

+1 with Mouser and some of the comments on the original article. I was only in the habit of using TeraCopy for it's error reporting and the ability to continue after it encountered a bad file. However, TeraCopy did seem to speed up massive transfers to and from networked drives. But I never did any serious testing on that so it may have only seemed faster to me.
« Last Edit: August 10, 2012, 06:49:56 AM by 40hz » Logged

Don't you see? It's turtles all the way down!
tomos
Charter Member
***
Posts: 8,619



see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #5 on: August 10, 2012, 07:14:42 AM »

[copy acceleration utilities] did seem to speed up massive transfers to and from networked drives.

there's at least one comment to that effect in the comments there.
Logged

Tom
40hz
Supporting Member
**
Posts: 10,736



see users location on a map View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #6 on: August 10, 2012, 07:39:38 AM »

^IIRC TeraCopy also touts network file transfer speed-ups as a feature, which is probably why I started using it in the first place. Grin
Logged

Don't you see? It's turtles all the way down!
Shades
Member
**
Posts: 1,673


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #7 on: August 10, 2012, 07:55:47 AM »

Teracopy shaves easily about an hour to two hours off when copying Oracle dump files to and from network drives. And yes, quite regularly I need to transfer 350GByte of dump files.

"Funny" thing is that extracting to the network drive directly is slower than extracting dump files locally and then transfer these with TeraCopy.

Also, I concur with the others about TeraCopy being a reliable way of copying data without supervision. Something that is nigh impossible with the Windows Explorer or any file-manager that uses the default explorer facilities.
Logged
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #8 on: August 10, 2012, 12:46:15 PM »

"Funny" thing is that extracting to the network drive directly is slower than extracting dump files locally and then transfer these with TeraCopy.
Have you tried mounting via NFS instead of SMB/CIFS? The SMB protocol, especially before the vista/win2k8 updated version, is notoriously slow - haven't played with NFS myself, but it might be worth a try?
Logged

- carpe noctem
paulobrabo
Supporting Member
**
Posts: 87


The Brazilian Bomber

View Profile WWW Give some DonationCredits to this forum member
« Reply #9 on: August 10, 2012, 12:59:18 PM »

I use Teracopy myself (on XP, Vista and 7), and for the reasons given.

What I find interesting, though, is that we all tend to treat what are in theory copy acceleration utilities as real-life copy management utilities. And if Samer is right (and I believe he probably is) copy management has a speed cost in Windows.

My original question remains: is a *real* copy acceleration utility for Windows theoretically possible? How come none of the available utilities seem to achieve the acceleration it promises?
Logged

English will never be my first language, it doesn't meter how hard I try.
40hz
Supporting Member
**
Posts: 10,736



see users location on a map View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #10 on: August 10, 2012, 05:15:27 PM »

My original question remains: is a *real* copy acceleration utility for Windows theoretically possible? How come none of the available utilities seem to achieve the acceleration it promises?

I think within a given PC, file copying speeds would be more dependent on hardware (and possibly drivers) than anything else. Windows is a mature operating system. So I'd suspect Microsoft has by now identified and taken care of most of the safely fixable boottlenecks. Which would account for why copy accelerators had a more pronounced effect under XP - and may well be getting in the way with newer versions of Windows.

Just guessing though. smiley

Network file copying is a different matter however, and there are definitely areas for improvement there. And also lots of ways to optimize and tweak network performance. However, hardware can once again play a major role since a faster network infrastructure yields faster transfers even if all other factors remain the same. So sometimes it's just more practical to put in faster NICs and data switches to get 'wire' transfer speed increases rather than bother with too much protocol or OS tinkering.


Logged

Don't you see? It's turtles all the way down!
Nod5
Supporting Member
**
Posts: 738



View Profile Give some DonationCredits to this forum member
« Reply #11 on: August 12, 2012, 04:44:17 AM »

file copying in Explorer in Windows 7 is sometimes slowed down by having to deal with dialogs like this

It is sometimes useful but I dislike that it pops up even for files that are exact copies. A smarter file manager would do a file hash comparison and skip copying that file if the hashes match, all in the background.
Logged
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #12 on: August 12, 2012, 11:24:43 AM »

A smarter file manager would do a file hash comparison and skip copying that file if the hashes match, all in the background.
That would be pretty disastrous speed-wise - definitely not something you want for a general file-copying routine smiley
Logged

- carpe noctem
x16wda
Supporting Member
**
Posts: 477


what am I doing in this handbasket?

see users location on a map View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #13 on: August 12, 2012, 05:03:53 PM »

That's very strange... I did a comparison a couple years ago between Windows and several of the other accelerators (Fastcopy, Supercopier 2, Copy Handler, FF Copy, Ultracopier and a couple more, all that would be no cost for commercial use, which ruled out Teracopy) to see what they did locally and over a lan.  It wasn't a double blind, duplicate hardware kind of test but it was the same sets of files and targets.  In those tests Fastcopy was the clear winner speed-wise (although the interface is lousy, which is not an issue if you're scripting something).  A few others were clumped behind that, generally not too far behind.  Windows was never the fastest option, although some of the utilities were slower in some of the tests.

I don't have the details any more, but that said, the real end result is that I have Supercopier installed on my personal PCs.  It was almost as fast as Fastcopy and does a good job with the management part that has been mentioned, and it has always been reliable.  That was the surprising part to me -- with half of the other utilities I tested, I had issues with installation, lockups or copy failures.  To my mind that immediately ruled them out.

That was back when I could do stuff like that for my previous employer.  I can't recall if was running XP or 7 at the time, but would have been 32 bit in eiother case.  Would probably be worth revisiting.
Logged

vi vi vi - editor of the beast
Nod5
Supporting Member
**
Posts: 738



View Profile Give some DonationCredits to this forum member
« Reply #14 on: August 13, 2012, 03:50:05 PM »

A smarter file manager would do a file hash comparison and skip copying that file if the hashes match, all in the background.
That would be pretty disastrous speed-wise - definitely not something you want for a general file-copying routine smiley
I think it could be designed to avoid speed-problems in many use cases. First, the hashing would only be done for identical file names in source and target folder. Second, the user could configure how large files to do the automatic hash check for, depending on system speed and user preference. For example if the operation only ran for same named files under 100 Megabytes in size then would there really be any problematic slowdown on computer with a newish CPU? If you are copying thousands of files with name conflicts then yes, delays will add up. But a smart file manager could also calculate the total number of such conflicts prior to operation and, if the number reaches some upper limit, skip auto hashing and display the regular interaction popups (including the checkbox labelled something like "do this action for all similar files"). So smartly designed it could avoid the possible slow-down cases and still save the user time and attention in all other cases.
Logged
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #15 on: August 14, 2012, 12:24:22 PM »

For example if the operation only ran for same named files under 100 Megabytes in size then would there really be any problematic slowdown on computer with a newish CPU?
CPU speed isn't really your main concern - something like Adler32 is pretty damn fast and "probably good enough". MD5 is also pretty fast, and since you're just comparing local files and not trying to be cryptographically safe, it will probably be sufficient.

The problem is that you're doing disk I/O. Instead of "1 read, 1 write", you'll be doing "2xRead + CPU, THEN perhaps 1 read, 1 write". OK, since you propose to use hashing, at least the 2xRead will run at full disk speed, whereas compare-the-bytes would be slower (seeking back and forth on a mechanical drive kills performance). But there's still a lot of overhead in this!
Logged

- carpe noctem
Nod5
Supporting Member
**
Posts: 738



View Profile Give some DonationCredits to this forum member
« Reply #16 on: August 18, 2012, 05:04:53 AM »

Ok, but doing a SHA1 check on a 150 MB file takes about 0,3 seconds on my system. So a bit over 0,6 seconds for a hash comparison. That is not so bad, especially compared to how Explorer works now, requestion user attention and delaying the file transfer with a popup window with a lot of text.
Logged
def
Participant
*
Posts: 19

View Profile Give some DonationCredits to this forum member
« Reply #17 on: August 18, 2012, 11:43:03 AM »

How can one write an article about copy accelaration utilities and then forget to mention the exact Windows version used?

"Windows version: Windows Home 64 bit."

Very meaningful indeed...
Logged
Pages: [1]   Go Up
  Reply  |  New Topic  |  Print  
 
Jump to:  
   Forum Home   Thread Marks Chat! Downloads Search Login Register  

DonationCoder.com | About Us
DonationCoder.com Forum | Powered by SMF
[ Page time: 0.043s | Server load: 0.03 ]