blanking file data while keeping folder-file structure

[Steven Avery]

My Eudora has a couple of hundred files (.mbx and .toc) I would like to clear out the data and keep the structure.

The reason now is funny. When I copied the 50+ gigabytes of data from puter to USB with XYPlorer it glitched out on the leading spaces.  (It sometimes left less spaces in the target usb.) This will mess up the filters.

Now I am 800 miles away and I want to fix the file names on the USB restored files.  If I could clear out the data on the remote source files with a simple script or utiity, I could copy over the structure with Teamviewer.  Then fix the structure by hand.  (Granted I could probably do this remotely, but in a more cumbersome manner.)

Also I could see this having possible additional future uses. e.g. Making a yearly archive of all the email.

Good method to use?

Thanks!

[Shades]

Some general advice:
While spaces are allowed in filenames by any of the filesystems that common operating systems employ, you should resist the urge to use them. Not all file system are as forgiving with the use of spaces and from your description I understand that you use leading spaces in filenames.

Granted, it is visually not as "attractive", but replacing spaces with '.' or '_' is a far better practice. Furthermore, there is the thing that file explorers aren't created equally. And I suspect that the amount of leading spaces will trip up almost all of them. If you think that file managers all use the exact same way to access available file systems, you are dearly mistaken. Even when they do use standardized dll to make those API calls, it isn't always certain that the latest & greatest versions of these are used. Or maybe you are using a (barely) older version of a file manager that didn't have the latest fixes for accessing the file system on your USB device.

If you are dead set on using leading spaces, I would only transfer these using only the bog standard Windows Explorer and nothing else.

Using a different file name structure could help you more than a script. For example: start the name of whatever .mbx file you receive personal mail into with '0.<insert here the amount of spaces you want><name of file>.mbx'. Business related .mbx file names could start with '1.' etc.

By doing something as simple as that, you will make transferring files with whatever file manager much more reliable and prevent the need for scripts and/or manual file name reconstruction in the future.

Adjusting your filters won't be too problematic that way either. I would even go as far as recommending 'Mail store home', which backs up your mail by storing these in a database. Not only allows you to search much more efficiently through huge piles of mail, it also allows you to transfer between file formats and different mail clients very easily.

The freeware version will likely be all you need and you an get it as a portable app too. With that in mind you only need to transfer the Mail store application to you USB device to have super-deluxe access to all of the mail you have put into it. I personally use it in this way with 18GByte of mail.

[Steven Avery]

MailStore is a good program, with which I have tinkered a bit. I'm not sure that its search facility is as friendly and strong as Eudora, with the current folder and sub-folder status.

As far as I can tell, Total Commander, my main file manager, works properly with the leading spaces. I just brought XyPlorer into play because of a time crunch.  It messing up the leading spaces was a bit of a surprise.  The puter is Windows 7.

Your advice on avoiding spaces (and especially leading spaces) is good, and noted.  However, it is a longer term solution.  I am trying to have a good one-time fix with what I have now.  That will be most easily handled by blanking out the data bringing over the file structure and making some hand adjustments to the one with data (most of the files do have leading blanks.)

Steven

[Steven Avery]

The data where the file structure matches the filters is 800 miles away.  If I change the names there (the filters will automatically match and change) in a slightly cumbersome manner with Teamviewer, I can not get the 50GB worth of data over here in any convenient way for a month or so. 

If I bring the structure over to my current computer, I can do the fix fairly simply.

An alternative: If someone knows a way to transport 70GB (more accurate) over the net in a reasonable time affordably, I could use that solution as well.

===============

However, looking for a way to make a copy of a file structure, blanking out the data, seems like a fairly simple routine, in one method it would be 3-step.

Place the name of the file in a variable.
Delete the original
Build a new empty file with the variable name.

So theoretically it could be done with something as easy as a batch file. Or, more elegantly, AutoHotkey. However, I always like a designed tool and avoid PC batch stuff, having been spoiled by the OCL on the System 36.  Should I just try AutoHotKey, or the similar one?

Steven

[DonL]

XYplorer dev here: I checked that in XYplorer and indeed when using the standard copy (using the shell copy API) any leading spaces are automatically removed from the copied filenames.

I recommend to enable XYplorer's own "Custom Copy" which does not have this sort of undesired smartness. Or use XYplorer's Backup function which also does not have this problem.

OTOH, I would generally not recommend leading spaces in filenames. Sooner or later they will get you in trouble.

Don

[4wd]

However, looking for a way to make a copy of a file structure, blanking out the data, seems like a fairly simple routine, in one method it would be 3-step.

If you just want a copy of the file structure so you can compare and rename your local files to match, wouldn't skwire's Zero Zipper do the job?

Or RoboCopy, (should be able to handle the spaces fine since it's using the normal filesystem), and zip the resulting cloned structure.

You'd end up with a file/folder structure clone of minimal size to transfer, then you could use a dual pane filemanager, (or something like Beyond Compare, set it to only compare filenames and display only discrepencies), to simultaneously browse the clone structure and data copy, fixing names as necessary.

Not as automated as you'd like but it is reasonably simple and doesn't involve any tedious program development.

[antekgla]

I see something siimilar in this post.

As 4wd pointed I think what Robocopy is the answer.

Robocopy is a command-line tool included with Windows 7 and later that easily does this job:

You could use Robocopy to create the same folder structure and zero-lenght files in another location. This command not erase your older files rather create the same structure (with zero-lenght files) in another place.

Code

ROBOCOPY sourcefolder destinationfolder /CREATE /S

/CREATE is the key here as it is what creates the zero-length files.

Robocopy Help.

[Steven Avery]

Thanks for the info.

Zero Zipper looks excellent. I ran it and it was fun. Maybe using Robocopy as well.

One extra feature would be if they gave you an inclusion capability on file types. e.g. Only .mbx and .toc (e.g. some folders have thousands of .jpgs form attachments that do not need their zero byte new equivalent.)  However, the 0 bytes files are easy enough to delete en masse so that can be seen as a frill.

It turns out my save was incomplete so I am going to do email remotely till I am back in Dutchess County.  And then do a new full save.

Thanks for the excellent feed back from DonL of XYplorer.

============

If you need the fastest type copy, what is recommended these days?  Which tool? What mode? 
Maybe Syncovery?  Maybe a special fast mode in Total Commander or XYplorer?  The regular modes I used on both really was too slow for my 70 Gigabytes. Many hours.

Another variable is how good they are on dealing with oddball things like ultra-long names.  Each tool varies, and you like to be uninterrupted (e.g. ask the questions at the end, and always have a "Skip All" answer.)

Some of this actually repeated from earlier, 2009.

Recreate files but without content to target drive
http://www.donationcoder.com/forum/index.php?topic=18682.5

Steven

[4wd]

One extra feature would be if they gave you an inclusion capability on file types. e.g. Only .mbx and .toc (e.g. some folders have thousands of .jpgs form attachments that do not need their zero byte new equivalent.)  However, the 0 bytes files are easy enough to delete en masse so that can be seen as a frill.

RoboCopy has exclude filtering capability along with wildcards for input, so you could run two RoboCopy commands, one including *.mbx and the other *.toc.

As long as you don't set the mirror or purge switches (/MIR or /PURGE) then it shouldn't delete the files created from the first run.

If you need the fastest type copy, what is recommended these days?  Which tool? What mode? 
Maybe Syncovery?  Maybe a special fast mode in Total Commander or XYplorer?  The regular modes I used on both really was too slow for my 70 Gigabytes. Many hours.

You could use RoboCopy for the copy, (RoboCopy = Robust Copy), since it's using the normal Windows API you're almost guaranteed to get a 1:1 copy of the data and it multi-threads the copy by default unless you tell it otherwise.

There's also RichCopy if you want a GUI version of RoboCopy.

[Steven Avery]

Thanks, great info.  Now downloaded.

RichCopy – Microsoft’s Multi-Threaded File Copy Utility - 2013 - Paul G. Tarver
http://www.tpcqpc.com/richcopy-microsofts-multi-threaded-file-copy-utility/

Great text and comments. They also reference.

15 Free File Copy Tools Tested for the Fastest Transfer Speeds
https://www.raymond.cc/blog/12-file-copy-software-tested-for-fastest-transfer-speed/

And a reminder to turn off anti-virus scanning if you want speed. 
And obviously, making sure that if it is a USB port, it should be USB 3 rather than 2.

There is a fair amount of discussion about Network copying.  Is that including Internet TCP-IP?  I may have to read more carefully.
Can you get 50 Gigabytes in hours rather than days with any of these tools like RichCopy/Robocopy?

The other RoboCopy offshoots?  RichCopy updated in 2011.

RoboMirror - updated 2014
https://sourceforge.net/projects/robomirror/?source=navbar

WinRoboCopy - updated 2016
http://www.upway2late.com/#/downloads

I don't think enough consideration is given to the question of 100% accuracy.  Although in my case, not needed.
Here are a few more of the interesting discussions.

==========

RichCopy alternative file copy program to RoboCopy (RichCopy more granular than WinRoboCopy)
https://frustrateditengineer.wordpress.com/2013/02/05/richcopy-alternative-file-copy-program-to-robocopy/

Add a Graphical User Interface (GUI) to the Microsoft Robocopy Command Line Tool - 2012 - RoboMirror and Richcopy
http://www.howtogeek.com/130729/add-a-graphical-user-interface-gui-to-the-microsoft-robocopy-command-line-tool/

How do I ... use RichCopy for high-powered file copy and transfers? (helpful on choices)
http://www.techrepublic.com/blog/windows-and-office/how-do-i-use-richcopy-for-high-powered-file-copy-and-transfers/


Any recommendations for a seamless 'Robust File Copy' addition to Windows? [closed]
http://superuser.com/questions/200782/any-recommendations-for-a-seamless-robust-file-copy-addition-to-windows

If You Like RoboCopy, Consider RichCopy Too - 2012
http://thenubbyadmin.com/2012/01/13/if-you-like-robocopy-consider-richcopy-too/

Need fast advice on a Windows file server migration - 400+ GB - 2013
https://community.spiceworks.com/topic/323883-need-fast-advice-on-a-windows-file-server-migration-400-gb

GS RichCopy 360 Multithreading File Copying Technology
https://www.youtube.com/watch?v=g8PxSuOVTP0

Steven

[Shades]

Regarding LAN copy:
On my gigabit LAN Directory Opus reports speeds of 70 to 80MByte/sec it can sustain when copying sets of database dump files (each 4GByte in size). Because of this I don't even use any software to help and/or speed up copying anymore.  But I distinctly remember when I still was using a 100Mbit LAN, that the use of Teracopy improved LAN copying speeds significantly. However, this was years ago.

Regarding Internet copy:
I use robocopy (command-line) to transfer files from one continent to another. Actually I set up my own WebDAV server and use a script to start the transfer of files to my WebDAV server. Can't complain about the speeds. As a test I used Google drive and I was able to manually transfer a set of files in 10 minutes. Using my own WebDAV server I can transfer the same set of files fully automated in 12 minutes at night or around 15 minutes during the day.

If you use Google Drive, Dropbox, OneDrive etc. you have been using WebDAV. It allows you to transfer files over the HTTP protocol (port 80). Setting up a WebDAV server isn't too difficult in Linux (Apache2 webserver is what I use, but Ngix and Lighttpd web servers are also fully supported) and only slightly more complicated on IIS (Internet Information Server) in Windows Server.

If you are on a relatively simple 20Mbit/sec or higher connection (not bound by (too) much routing and/or firewall rules) between your computer at home and your current location you would be amazed how much data you can push over during one night of sleep. The higher the upload speed is of your connection at home, the better (if you transfer from home to your current location). If you can find out what the ip number of your current location is, you could even take a look at a (portable) piece of freeware called: CarotDAV. It allows you to set up a minimalistic WebDAV server on your current location and with the IP number of your current location you could try to initiate a transfer session on your home computer (TeamViewer or similar software) to your ad-hoc WebDAV server at your current location. Perhaps there is someone at your home location to start the transfer on your behalf, in case you don't have teamviewer.

For transferring stuff you don't want to park even for a moment on the common commercial cloud solutions, you can do way worse than using your own WebDAV server. WinSCP can also be used in combination with WebDAV. Also freeware, also portable and comes in both GUI-based and command-line application and is with regards to file transfers very powerful and reliable too. In power, complexity and reliability WinSCP and Robocopy are matched.

[4wd]

The other RoboCopy offshoots?

RoboCopy derivatives that I mentioned here:

BTW, if you're going to use Robocopy where there's any chance that the files will already be opened then it will fail on those files, (it won't access open files).  You can get around that by either using:
1) ShadowSpawn (which is a replacement for Hobocopy below) which will create a VSS copy before running Robocopy, or;
2) Hobocopy which was a less featured version of Robocopy that could use the VSS, (written by the author of ShadowSpawn).

Although, ShadowSpawn is really just a pre-command to RoboCopy, (or any other command), that creates a VSS copy so that open files can be accessed.

Wikipedia also mentions a couple of other GUIs for RoboCopy^w, like Easy RoboCopy which has been updated to cope with new args in the Win10 version.

Can you get 50 Gigabytes in hours rather than days with any of these tools like RichCopy/Robocopy?

File Transfer Time Calculator

eg. 50GB @ ~20Mb/s = ~6.3 hours

It all depends on whether you can saturate your connection and the fact that a lot of small file transfers will slow down the transfer due to increased overhead, (more negotiation). In which case it's probably better to Zip/7z the folder structure beforehand just using the store option instead of compression so you only have one big file to transfer at maximum throughput.

+1 for CarotDAV, although I use it as a single interface for accessing various cloud storage rather than as a WebDAV server.

[IainB]

I'm not sure whether it can cope with leading spaces in file/folder names (I've never come across that before), but, using xplorer², the user can:

1. Copy directory structures, without the data.
2. List as a "Flat-file" display all the files in a directory structure, enabling the user to see, filter, sort and manipulate/edit/delete all files listed.
3. Individually rename or mass rename files listed in a display, and the mass rename features are rather smart.

That might all be useful in your case.

[tomos]

For straight sync, Bvckup was getting very good press here -- fast and accurate were the reports:
http://www.donationcoder.com/forum/index.php?topic=36388.0

haven't used it myself so dont know how it deals with any potential problem files.

Edit// I use Syncovery myself: subjectively very fast; again very accurate; partial file updating; can find moved files and move at destination rather than delete/recopy. It can ignore conflicts (there are a few options, let me look there), and send you an email if there's a problem (or if there's no problem as well).
But I havent really used it in this type of scenario so cant say a whole lot more about it.

[tomos]

I use Syncovery myself: subjectively very fast; again very accurate; partial file updating; can find moved files and move at destination rather than delete/recopy. It can ignore conflicts (there are a few options, let me look there), and send you an email if there's a problem (or if there's no problem as well).
But I havent really used it in this type of scenario so cant say a whole lot more about it.

I actually cant find the email notification setting in Syncovery, I know it's there, I have it enabled for at least one profile :-/

Some relevant options:





see mouse arrow for how to access this dialogue (below):

[Steven Avery]

For now I did the zip thing with 7-Zip .. got it under 15 gigabytes. The count of files looked like it did not take a 100 or so from 20,000, but I did not really check that.

Even though Teamviewer has all sorts of concerns about speed and bandwidth in the free account, so far it has done almost 3 gb, at about a gigabtye an hour. (Just taking the simplest method since Teamviewer was active.)

Steven

[Steven Avery]

It worked perfectly for most of the day.  Then the power was shut off, in order to get the TV up (unrelated issue).  Oops. Aborted. Teamviewer's resume is not working in #11.

So may do it again.

Here is my question.  If I am looking to use Syncovery, or Robocopy or something.  With one 15 Gigabyte file.

Which software is friendliest working with two PCs (Teamviewer connected) to make the connection. Do you do a WebDav thing.  Which one is intuitive?

Steven

[Steven Avery]

Let me try to understand some basics. 
We will discuss Syncovery which is helpful with its Wizard Mode and Advanced Mode settings for creating backup Profiles. 

(Would it be simpler just to set up a free FTP server on one puter and client on the other? Not sure.)

In the Wizard Mode, Syncovery seems to indicate that you have a few methods that could work when dealing with your own remote computers.

Network Path - example given : \\mylaptop\C\Users\pureb\Documents
Browse - File Path (here I presume you have made a WebDav connection for this to apply, if you are working with a remote puter)
Globe --> Internet Protocol Settings --> FTP, SSH, WebDav, AmazonS3, HTTP, also many servers. Note that Amazon S3 and somes others are modest cost for my 15 Gb File if I want to put them in the middle, and if either they take such a large file, or I go into splitting.

(Maybe this WebDav only needs the external IP number and the User-Password? Hmm.. that is pretty loosey-goosey.)

There is also a "Device" that brings up "MTP Device Settings" which seems to be local devices, maybe relates to some media backup.
Note: I remember that Syncovery was pretty smart when I made a WebDav connection to DriveHQ and suggested making their own connection.

So, what specifically do you do? Can Syncovery or CarotDev or something walk me through having the WebDav path connection?  Clearly, that has a big advantage in that your Total Commander or other File Manager is then in play?

If you want to do the FTP with Syncovery, does that mean that you set up a free server and then access it with Syncovery?

Is there a VPN method that creates a Network Path that would be usable by Syncovery?

Maybe a little discussion, or finding a good youtube, can hash out how this is really done without too much complexity.  e.g. When I opened CarotDev, I saw all sorts of blank entry fields, presuming more familiarity than I have.

I realize that I am asking a lot here.  It just seems that there is a gap between the presumed solutions and the place where the not-real-PC-tech people begin.

FYI - I will also mention offhand that Binfer looks comfortable for inexpensive transfer of large files, with a funny email interface and modest cost. I did a couple of small xfers with it, but for clear communication I ended up opening two free accounts, with file limits. It is unclear then if both would need to be paid accounts. Also the software seems to be somewhat dormant the last five years. However, it might do a 15GB direct file transfer quickly for $5 or $10 as part of a 40Gb package. 

A whole nother possibility involves simply using a server I have, as of today, available for WordPress and such. Presumably that would be an easy FTP in triangle mode. Why do I have a server for that? Well, a biz e-commerce operation had three, with lots of redundancy, and very inexpensive. So we cleared two out and I inherited one .

Steven

[Steven Avery]

Note: the above stuff might also be good for the Sync forum

Syncovery
https://www.backupcreations.com/forum/windows-support-no-bug-reports-f8.html

And yes, I might be missing some obvious explanations and documentations.

[Shades]

Speed of FTP transfers degrades quickly when transferring small sized files. In that case you better archive everything in one big lump of data and start the transfer to your e-biz server. Are you sure this server has enough storage space for this? You might get into trouble if this is a "shared" server setup in your e-biz solution.

If you have no immediate commercial plans for this e-biz thing, it might be possible to create your own personal cloud solution on it, such as OwnCloud. That way you won't get charged for (more) money and could be an interesting learning endeavor for yourself. However, if you have no time and/or inclination to get into such things, there are more ways to "skin this cat" (a link to an overview of free tools available to transfer files through your own cloud).

[Steven Avery]

Great suggestion.  I do have the time and the need to learn and the space. The e-biz is running on another server and there is little point in reducing the number since we have a super-duper deal.  Also I have a web programmer personal friend, Carolina Custom Designs, who can help me with modest cost since he is actively working the e-biz (bullion company, WordPress and WooCommerce) end as well.

Sure, later I may move things around a bit, but this is a server I can use for my WordPress and VBulletin and other stuff, often on Bible issues. Right now spread over a couple of other hosts.  And get my skill level up.

OwnCloud, (or Cozy or Seafile, et al) here we come.  (First, I have to get the FTP credentials ).

Steven