topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Monday December 2, 2024, 11:20 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: teracopy: copy your files faster  (Read 126419 times)

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 11,190
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #100 on: February 23, 2008, 08:28 AM »
Open Source isn't the holy grail to me.  It's useful for some things, and not so much for others.  Teracopy already has a lot going for it and it's free... so I'm not going to start with something that doesn't do what I need, just because it's OS.  And teracopy itself is actually only a little bigger... so I can allow it a few bytes since after you added all the additional features to PC, it would probably be just as big.  ;)

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #101 on: February 23, 2008, 12:21 PM »
ut... it doesn't do the primary thing teracopy does... copy things faster!
Benchmarked, or throwing a tongue-in-cheek statement? :)

Does the teracopy folks explain how they achieve better speed, or is it all black box magic? There's a lot of parameters to tweak in a file copy routine... like, whether you scan all source files for size so you can check if there's enough room on destination, whether to use buffered or unbuffered file I/O, how large blocks you operate on, etc.
- carpe noctem

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 11,190
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #102 on: February 23, 2008, 09:29 PM »
I've compared it to plain vanilla copying, and it is indeed faster. :P

As far as the how, this is what it says on the site: TeraCopy uses dynamically adjusted buffers to reduce seek times. Asynchronous copy speeds up file transfer between two physical hard drives.

I understand the words, but it's still black box to me. :)

Other features:
# Pause and resume file transfers. Pause copy process at any time to free up system resources and continue with a single click.
# Error recovery. In case of copy error, TeraCopy will try several times and in the worse case just skips the file, not terminating the entire transfer.
# Interactive file list. TeraCopy shows failed file transfers and lets you fix the problem and recopy only problem files.
# Shell integration. TeraCopy can completely replace Explorer copy and move functions, allowing you work with files as usual.
# Full Unicode support.

The pause and resume, error recovery, and interactive file list are very useful too...

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #103 on: February 24, 2008, 07:07 AM »
wraith808: I thought you were on about PerigreeCopy vs. TeraCopy, not vanilla explorer copy vs.  TeraCopy :)

"dynamically adjusted buffers to reduce seek times" sounds a bit bull, because you'd have to take file fragmentation into consideration as well, to reduce seek times... async (aka overlapped) file I/O is good though, instead of issuing a read, waiting for that, issuing a write, wait for that (etc.) you overlap the operations - which basically means you can have read #2 taking place at the same time as write #1.
- carpe noctem

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 11,190
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #104 on: February 24, 2008, 08:43 AM »
Well, the reason I didn't compare PC to TC is because on PC's page it makes no claim to making the copy faster- it just adds some niceties to the copying process like not failing out if one file fails, etc. ;)

As far as the second part- it makes more sense when you explain it than when they do. :D

KD7LRJ

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 6
    • View Profile
    • Donate to Member
New multi-threaded copy program
« Reply #105 on: January 07, 2009, 05:15 PM »
Anyone want to try out yet another copy program? It is multi-threaded and was designed to improve copy speeds on high-end network storage devices. It's mostly for use with lots (millions!) of small files, but works in other situations as well. Not a big deal - just thought I'd share if anyone can use such a thing...

McTool - web page

  • Multi-threaded
  • Drag & drop
  • Directory synchronization
  • E-mail report when finished
  • Save parameter files
  • Run parameter files from command line
  • Log actions to file
  • Retry on error
  • Wildcard (Regular Expression) file matching
  • Pause/resume

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #106 on: January 07, 2009, 10:21 PM »
Hmm, what's the point of doing it multi-threaded? Isn't this just going to thrash the disk drives seek requests? I guess the filesystem cache might rescue you though, since you mention it's made mainly for small files. I wouldn't have thought multiple threads worked that well wrt. network copying, but perhaps it triggers multiple SMB sessions - in which case it is useful (I haven't made SMBv1 sessions go much faster than ~30MB/s, and SMBv1 is all you're going to get unless both machines run Vista or 2008-Server).

Personally I'd experiment with fewer threads but include async I/O (possibly using I/O Completion Ports).
- carpe noctem

KD7LRJ

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 6
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #107 on: January 08, 2009, 12:04 AM »
Hmm, what's the point of doing it multi-threaded? Isn't this just going to thrash the disk drives seek requests?
On drives with a single spindle, you're right to suspect minimal or even negative improvement, yet most of the time with relatively small files (perhaps even as large as, say, files in an MP3 collection), you can still gain a performance advantage by using a small number of threads (2-5 has worked well for me in the past) on single drives.

Some things that affect whether multi-threading will improve copy speeds:

  • The sizes of the files.
  • The availability of NCQ on the drives (both source and destination).
  • The sizes of the internal memory caches on the drives.
  • The number of spindles that make up the volumes (e.g. a RAID array)
  • Other hardware caching of data on the drives (NAS controllers, etc.)

On high-end systems (like these) I often see 10 times the throughput using multiple threads (whether using McTool or multiple instances of RoboCopy) when copying smallish files from one system to another. I have seen throughput speeds of over 90 MB/s in a single instance of the tool.

I don't know enough about SMB, CIFS, or Completion Ports to talk intelligently about them, but in developing this application, I have run into enough information about them to think it would be worth learning more...

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #108 on: January 08, 2009, 12:30 AM »
Dunno if number of platters/spindles has much to say, as you don't really have control over which files go where :)

NCQ helps mitigate seek thrashing, but does not remove it - if you can avoid excessive seeking, you should.

I don't know how much drive cache memory matters to be honest, the benchmarks I've seen have been unable to show a difference between 16MB and 32MB of disk cache. I'm sure that some amount is crucial, but when do we stop seeing gains? Certainly the benefit would be a lot larger is your Operation System didn't do read/write caching. But since it does, what are the (quantifiable!) situations where the on-disk cache matters? I've been pondering this for a couple of weeks now :)

If you have a system that does "insane caching" of writes (the kind only battery-backed devices usually dare), then multiple writing threads very likely isn't a problem, and the hardware can take it's time to re-order the writes and do as much sequential bursting as possible, avoiding seek thrashing, and giving good speeds.

As for SMB/CIFS (two names, same thing), SMBv1 (the protocol used up to and including XP) shows it's age and limitations. Iirc the problem is a mix of request-packet size as well as ACKs for each "packet", whereas SMBv2 (again, iirc - I should read up on this!) increases the "packet" size substantially, as well as allows for pipelining. The net effect is that SMBv2 should be able to utilize gigabit (and faster) connections a lot better, whereas SMBv1 (even on copying a single huge file) will usually cap out at around 30MB/s or so, even with various network tweaks (I get ~32MB/s on my fileserver via SMB, vs. 55+MB/s via FTP, depending on source disk speed - and if I didn't have everything AES-256 encrypted, it might even go faster ;)).

As for I/O Completion ports, the main rationale behind them is to combine with Async disk I/O. The principle is that you can issue  a load of async I/O requests (which the OS can hopefully "be smart" about, at least that's the theory!), and instead of manually creating a bunch of threads, the completion routines get scheduled to a thread pool. It's mostly useful for high-performance many-connections internet servers (where you can avoid resource starvation from threads, and avoid some context switch overhead as well), but I think it could be interesting for your file copying scenario as well, if the OS is smart enough. At any rate, at least it would save you from the (relative :)) wastefulness of creating a lot of threads - most of them will be spending a lot of time blocking on I/O, so a thread pool should serve just fine.

However, that is all theory, and it might not hold up in practice - I've been meaning to toy with IOCP myself for a long time, but haven't gotten around to it... and while the theory behind it is indeed very nice, I dunno if it'll work in practice for the file copying example. And I don't have super-high-end gear to test it on, even if I wrote the code :)
- carpe noctem

wreckedcarzz

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 1,626
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #109 on: January 08, 2009, 12:51 AM »
Just wanted to say that I read this topic a while back, and for the last year or so I've been using TeraCopy almost exclusively (except in situations where launching an external file copy app would take up more time than would be gained by it, and that means... Explorer). I finally got the portable version for my new 16GB flash drive (couldn't fit it on my tiny 1GB), and am using it everywhere. Gotta love it. :)

KenR

  • Super
  • Blogger
  • Joined in 2006
  • ***
  • Posts: 826
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #110 on: January 08, 2009, 01:27 PM »
Ok, so let's say I was copying from my laptop's internal drive from one partition to another or from a drive to an external drive connected via USB. At last virus scan, the file count was close to 2M. Most of these are very small files. Would either terracopy or McTool copy them faster than Windows Vista 64, directory opus, or xyplorer?

Thanks, Ken
Kenneth P. Reeder, Ph.D.
Clinical Psychologist
Jacksonville, North Carolina  28546

J-Mac

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 2,918
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #111 on: January 08, 2009, 01:42 PM »
Ok, so let's say I was copying from my laptop's internal drive from one partition to another or from a drive to an external drive connected via USB. At last virus scan, the file count was close to 2M. Most of these are very small files. Would either terracopy or McTool copy them faster than Windows Vista 64, directory opus, or xyplorer?

Thanks, Ken

Ken, I haven't tried any copy jobs quite that large!  But I have done some rather large jobs and Teracopy does do it faster than Directory Opus. I haven't clocked it but I remember this was the reason I tried the free version of Teracopy in the first pace. DOpus was copying well but not very fast, so I thought I would try Teracopy - It definitely was faster but I haven't actually timed it.

I have not copied anything using bare Explorer in a while - I am using DOpus for about a year and Total Commander for several years before that.

Jim

Paul Keith

  • Member
  • Joined in 2008
  • **
  • Posts: 1,989
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #112 on: January 08, 2009, 08:43 PM »
Haven't read the entire topic but I did a search and this app wasn't mentioned here:

http://www.ranvik.net/totalcopy/

I can't remember why I switched from teracopy to this app but I think it was just the comfort in also having the regular copy paste operations in case something went wrong.

J-Mac

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 2,918
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #113 on: January 08, 2009, 10:37 PM »
Haven't read the entire topic but I did a search and this app wasn't mentioned here:

http://www.ranvik.net/totalcopy/

I can't remember why I switched from teracopy to this app but I think it was just the comfort in also having the regular copy paste operations in case something went wrong.

I use Teracopy and I still have my copy and paste functions. Why do you say that?

Jim

Paul Keith

  • Member
  • Joined in 2008
  • **
  • Posts: 1,989
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #114 on: January 08, 2009, 11:54 PM »
Oh sorry. I'm just recalling from memories.

I think I felt that way because switching from the normal copy-paste functions to teracopy's copy-paste function felt more unnatural than total copy's instant separation of both functions through only changing the right click copy options.

J-Mac

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 2,918
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #115 on: January 09, 2009, 01:13 AM »
Oh sorry. I'm just recalling from memories.

I think I felt that way because switching from the normal copy-paste functions to teracopy's copy-paste function felt more unnatural than total copy's instant separation of both functions through only changing the right click copy options.
Maybe I'm not seeing that because I installed the Copy and Move icons into the Directory Opus and Total Commander interfaces and use them to sue Teracopy. If I just drag and drop the native copy is used.

Jim

Paul Keith

  • Member
  • Joined in 2008
  • **
  • Posts: 1,989
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #116 on: January 09, 2009, 01:41 AM »
It could be. I think that might not have been it. I think I could have been used to using the ctrl+c, v hotkeys so I wasn't comfortable having it be using Teracopy's method.

I think I wanted it the reverse so Total Copy suited me more.

Curt

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 7,566
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #117 on: January 09, 2009, 04:00 AM »
Shell integration. TeraCopy can completely replace Explorer copy and move functions, allowing you work with files as usual.

J-Mac

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 2,918
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #118 on: January 09, 2009, 12:00 PM »
Shell integration. TeraCopy can completely replace Explorer copy and move functions, allowing you work with files as usual.

Can but doesn’t have to. It is configurable. Make it what you want.

Jim

Curt

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 7,566
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #119 on: January 09, 2009, 01:24 PM »
- yes, that is usually what the word "can" implies. "Why do you say that?"

wreckedcarzz

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 1,626
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #120 on: January 12, 2009, 09:06 PM »
"Why do you say that?"

Because we can. :D

Gergith

  • Participant
  • Joined in 2009
  • *
  • Posts: 5
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #121 on: January 16, 2009, 02:03 PM »
Hey, I registered just to post in this thread. I read it beggining to end.

I have been using TeraCopy for at least a year for the following reasons:

I really like seeing the queue.
auto CRC checks


I have 5Terabytes of storage spanning 10-12 hard drives (2 every size generation :p 200, 250, 300, 500, 750, 1tb :D give or take)
and at least 4 Tb of content.

Everytime I buy a new drive, I juggle all my files. So thats typically 30-500 gig transfers at a time between hard drives. Its VERY VERY re-assuring to have a Hash check.

I download alot of full series of cartoons and documentary shows.
So it is very amazing to be able to copy files, and to also see that a CRC check was done on the source and destination.

Thats my reasoning for picking it. I haven't found better yet.


Of the ones mentioned so far, can you guys tell me which ones for certain can do a CRC, or any sort of Hash check on source and destination files?

Thanks again

And thats for all sharing your opinions, this thread really is good and informative (between the typical fun banter)

Gergith

KD7LRJ

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 6
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #122 on: January 16, 2009, 07:27 PM »
Of the ones mentioned so far, can you guys tell me which ones for certain can do a CRC, or any sort of Hash check on source and destination files?

I added this ability to McTool today. If it fails to compare, it will retry (up to the specified number of times) until it gets it right.

With verbose logging turned on you can see the SHA1 hash in the log for each file, otherwise, it just reports when a compare fails.

Gergith

  • Participant
  • Joined in 2009
  • *
  • Posts: 5
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #123 on: January 16, 2009, 07:41 PM »
Cool thanks! KD7LRJ! your fast response earned you a try ;-)

SHA1 is the only way it hash checks right? (thats not critisism, thats a question :p)

Why did you pick that one? (im asking from lack of knowledge, its not a trap :p)

Gergith

  • Participant
  • Joined in 2009
  • *
  • Posts: 5
    • View Profile
    • Donate to Member
Re: teracopy: copy your files faster
« Reply #124 on: January 16, 2009, 07:43 PM »
one other thing that i really like from Tera copy. is if the file exists at the destination, instead of just skipping/error'ing , you could Hash check the already existing file (file size first for speed :p)