Dumb question but maybe somebody knows how. Probably a form of Sync.

[questorfla]

Best Program or method to:
Copy all files from a location on drive D: to a location on drive E: with the following conditions:
If the file already exists on E:  AND  is newer  OR      larger  than the one on D:    Then the copy on E: is left in place left in place and copy on D: is deleted.
Otherwise the copy on D: is Moved to E: and overwrites without asking.

Sounds simple but I cannot find the right combination of switches to do the either or part
This should end up merging the two locations onto the E: drive location keeping the copies that are either newest or largest regardless of where they are.
The reason for this is that there are some newer copies of the same files that are from zero to 5 bytes or so in size created in error.  Need to keep the larger copy in that case even if it is older

[c.gingerich]

I have used Clickteam SynchronX and it has worked well for me - http://www.brothersoft.com/synchronx-4732.html

[tomos]

Dopus doesnt do this.

Snycovery looks like it can do this - but for some reason it's greyed out here:

[attachimg=#]

One problem with the larger file replacing smaller is if a more recent file is slightly smaller, it will get replaced by older version - but this may not be relevant to your situation.
Syncovery gets around that by only doing it for files that have the same datestamp.

[questorfla]

Thanks to you both for such quick answers.  Tomos, yours as displayed would probably work and i working with drive copies for the first run.  If it works, i have maybe 30 backup drives to merge into a single drive with the most relevant data so i will know for sure after the first run and can't hurt anyway.
I have never used either program so will try both.   With hard-drives cheaper than backup tapes used to be this is just how they have been doing it to provide failsafe redundancy but after 5 years i need to get this all onto a single new master copy.  I don't trust 5 year old hard-drives all that much.
almost forgot, do either "clean up" the original?  The reason for the cleanup is so there is no doubt that something was done with everything.
I cn';t see any instance (other than an immovable file) where it could be left over on the drive D:.  If it does as it szhould D: shold be empty.  I can do a lowlevel format and check for any glitches.  If it is one of the newer drives I will keep it in rotation.  Those 3 years or older though get "shredded" (hard to believe you have to shred a harddrivethese days    . 

But such is the world of the "future".  I can't wait for the 3d-printable hard-drives to comes out  .  Then maybe I can just burn them after use

[tomos]

Syncovery will move the data - so source drive should be empty at the end if your settings are correct -
re the greyed out options - the author is extremely helpful so should be able to clarify what's going on there. There is a forum I think too.
It's not the cheapest, but it is (1) able to do almost everything** (2) very dependable, and, (3) the support is first class.

** takes a bit of time to setup - requires going through all the settings for the job

[IainB]

I was interested in this discussion and went to check FreeFileSync and the FreeFileSync - automated backup - Mini-Review, to see if FFS could meet the requirements as described in this discussion thread.

Conclusion: It doesn't look like it can.
Reason:

[attach]

[x16wda]

I think most of the copy/sync utilities I have seen need one consistent criterion for filtering selections. That said, it would be fairly straightforward to write in a scripting language [fill in your favorite here, my choice would be Rexx]. If I can scare up some time over the next week or two I'll have a go at it (although someone may beat me to it).

[questorfla]

Yeah.   Probably not an off the shelf issues for sure.  I already posted the first road bump.  Encryption.
Some of these files are copied direct from old server drives.  The old server 2008 R2 (and maybe even server 2003) had cases where the files ended up with EFS encryption but when the drives were pulled no one bothered to save the keys.
Because of the fact that I have all these drives from many years to work with I found that eventually I can find the same file somewhere with no EFS.
But:  In any case where an encrypted file (even if it is one with a zero file length) syncs to the same location of files that are NOT zero length, the EFS files always get kept instead regardless of date or size.  They cannot be overwritten either so I cannot go back.  EFS wins all the time.

If you take a look at my other post about this you can see a pretty nifty script I came up with to look for and make a list of all zero length files on the entire hard-drive.  And surprise !  it works    and just one line too.  (or two lines in batch)

My reasoning was that most of these EFS files had no data because of their lack of keys when copied.  They names are there but no "size".  The script did a great job and even located a ton of trash I can remove but.. It did not find all the EFS files or even close.  Many of these files turn out to be inside whole EFS folders.  So .. I next needed a way to search for Only EFS files regardless of size and that turned out to be a little harder.
Apparently, the fact they are EFS makes them somewhat invisible to many search utilities.  That encryption flag is apparently not so easy to locate or make notes of the path to the file.  

On several sites many SYS admins point out that this is a critical "first thing to do" when working with old data servers becasue of this.  Early EFS setups can easily be transferred forward onto new drives without their keys and apparently all it takes is to put a regular file into an EFS folder and the flag gets set for the file.  So the users unwittingly makes their own files unreadable.  This seems to be an inherited permission though I have not tried so hard to find out, I just want to get this over with.

If someone knows the correct "tag" to use instead of file length, i could just plug it into other script I have and get the same list of EFS files instead of Zero Length files.  End of problem.

[tomos]

If you take a look at my other post about this you can see a pretty nifty script I came up with to look for and make a list of all zero length files on the entire hard-drive.  And surprise !  it works    and just one line too.  (or two lines in batch)

where's that (dont see it above)

[4wd]

If someone knows the correct "tag" to use instead of file length, i could just plug it into other script I have and get the same list of EFS files instead of Zero Length files.  End of problem.

Or just use the cipher command: cipher /U /N >encryptedfiles.txt

If you don't have encryption enabled it'll just bomb out:


Otherwise (I don't have any encrypted files but you get the idea):


I wouldn't be surprised if it required all the encryption keys for every user on the system to be able to detect encrypted files belonging to each user.

You could also probably use Powershell to recurse a tree looking at the extended file attributes, (which is where the encryption attribute hides - the only way to see it from DOS is the fsutil usn readdata command).

Addendum: There's some methods shown on the following page: A Fast Way to Find EFS Folders and Files

Including a VB Script and the EFSInfo command off the Server 2003 CD.

Addendum2: And a Powershell script from TechNet:

Code: (text)

get-childitem c:\ -Recurse -Force -ErrorAction SilentlyContinue | Where-Object {$_.Attributes -ge "Encrypted"} | format-list fullname, attributes

[questorfla]

Tomos.  I guess it is too small to even call a script .  More like a complicated command line but it still did the job and surprised the heck out of me when it ran
I also just figured out WHY all the problems on these backups.  Some nit-wit has been putting each sequential run inside the folder of the previous run instead of Overwriting the older files, it made whole new copies of them in another sub folder.
I also know who it was    but that is another story.  They will find ourt soon enough.
This is why things were getting exponentially out of hand.  If I had not found this I wonder how deep they could have gone before something crashed?  Or maybe windows would just error out on too deep to copy?  All these drives are full of multiple duplicates . well sort of dupliates.  ONE of them wil be the newest of every file and it should be in be deepest sub-folder.
Maybe i will just leave them,  Extra backups.

[Shades]

Do not consider a file that is stored in 2 different locations on the same hard disk as 'backed up'. That attitude will bite you at your most inconvenient moment(s).

Different disk - mweh
Different computer - getting there
Different house/company - better
Different country - even better
Different continent - great
Different planet - awesome!

[questorfla]

Sorry Tomos:  Thought i had put it with this question.  It has been a LOOONG week.  I had put it on another thread.
But repeated here just in case.  I gwet so much help from everyone maybe this is will be of some use to some other poor sucker like me who hired on to work with (FOR!!) a bunch of techno-nit-wits.
Still barely qualifies as script but darn sure does the job for FINDing them.  Getting Rid of them .. well I had issues there as it finds and stores complete paths including spaces in the file path.  When I tried other variations using the same logic to DELETE them i can see what why it stopped by looking at what it found.  a whole list of name as this:
      "C:\etc\etc\and you see\spaces in the way\zerolengthfile.txt."  And many more like it. 
(probably due to the same nit who multileveled her backups)  I got a lot of invalid path etc.  Once i found them and had the list of where they were most were in the same places anyway. 
Not too hard to remove but.. if this could have done the delete too...
i could have created a list of the filenames that were getting deleted AND delete them in one swoop.  Then add in 4WD's cipher and...all the Old encrypts would be gone as well
  As long as i don't end up wiping out the entire windows folder i should be OK   Maybe?

Code: (generic)

   @echo off
   for /r %%F in (*) do if %%~zF==0 echo “%%F” >> zerobytefiles.txt

ps:  change the double % to single % if running from prompt.  Also the >> I am chicken and run from batch.
So I have some small chance of stopping if major bad things happen.

[Shades]

@echo off
echo @ECHO OFF>> zerobytefilesremover.bat
for /r %%F in (*) do if %%~zF==0 echo DEL /F /Q “%%F”>> zerobytefilesremover.bat

This creates a batch file that should delete all the 0-byte files when executed.

[4wd]

Then add in 4WD's cipher and...all the Old encrypts would be gone as well
  As long as i don't end up wiping out the entire windows folder i should be OK   Maybe?

Use PowerShell:

Code: ( powershell)

get-childitem r:\ -Recurse -Force -ErrorAction SilentlyContinue | Where-Object {$_.Attributes -ge "Encrypted"} | Remove-Item -whatIf

Change the r:\ to whatever path you require that isn't the Windows drive.

The -whatIf parameter on the end tells you what will happen if the command was to execute (Remove-Item) but it doesn't actually execute it.

Remove the -whatIf if you're happy about what will happen.

Sample output:

Code: (text)

PS C:\> get-childitem r:\ -Recurse -Force -ErrorAction SilentlyContinue | Where-Object {$_.Attributes -ge "Encrypted"} | Remove-Item -whatIf

Confirm
The item at Microsoft.PowerShell.Core\FileSystem::R:\test\encrypt has children and the Recurse parameter was not
specified. If you continue, all children will be removed with the item. Are you sure you want to continue?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): A
What if: Performing the operation "Remove Directory" on target "R:\test\encrypt".
What if: Performing the operation "Remove File" on target "R:\test\encrypt\20150505-121915.gpx".
PS C:\>

[questorfla]

Darn Shades:  That is "slick:.  
It created the bat perfect but it still won't delete the files.  Even if I run the bat as admin.   I have no idea why it wont work but I still get that message like I was getting on mine about the "structure or path" not being right.  I thought it might be the quotes but yours has the quotes
"The filename, directory name or volume label syntax is incorrect"  Looks fine to me.
I ran a test of it on my desktop just to grab a  zero length files I knew was there.
It made the bat with this statement:
DEL /F /Q “C:\Users\ASA4\Desktop\DEC 16th 23 sync users somebody not there yet.txt”  
This is a file I can see and it is zero length.  But I still get the error and it is stil there.  
I can right click and choose delete and POOF
I LIKE the "WAY" you did it though.  Each statement is a standalone command.

Your "deletezerobvte.bat" creator worked a charm and I have never thought of doing it like that before.
I am going to save that logic for future use!  I could rewrite to make the del command into any command needed to run on multiple files found by using a similar logic tree.

I Wish I knew why it wont delete the files but even if I copy the del command statement verbatim and paste to a command prompt,  I get the same error.  Got to be something minor.
And minor it is but...
for some reason their is an odd character created in from of the C:\  so the actual delete line in the batch file says "DEL /F /Q    (here there appears a character that looks like a small letter o with the ^ above it)    then the normal c:\archives\etc etc.  the rest of the normal path.  The thing is the quote marks are gone.  I can only see this if I remove the "@echo off" so I can watch what is happening.  Is a quote mark one of those characters you have to escape from twice in a batch file like the >> and double %?

[4wd]

DEL /F /Q “C:\Users\ASA4\Desktop\DEC 16th 23 sync users somebody not there yet.txt”

Look at the quote marks.

It'll look the same as normal quote marks at the DOS prompt but it won't work, the following will:

DEL /F /Q "C:\Users\ASA4\Desktop\DEC 16th 23 sync users somebody not there yet.txt"

To illustrate:

Command file - top line is taken from posts above, third line replaced the quote marks with normal ones off the keyboard using Notepad2 (which makes the difference more obvious):


Result:


The angled quote marks are usually caused by a word processor, with a basic text editor they always look like a pair of vertical parallel lines.


Fixed command file without the funny quote marks:

Code: (text)

@echo off
echo @ECHO OFF>> zerobytefilesremover.bat
for /r %%F in (*) do if %%~zF==0 echo DEL /F /Q "%%F">> zerobytefilesremover.bat

[questorfla]

WHY OH WHY!!!

  WHY didn't I see that
I have run into thAT at least once before and recently i have noticed that on some of my office programs the quotes don't tilt the right way either

also did you know spell-check refuses to offer "don't" and an option to a misspelled DON'T no matter how wrong you do it that is never EVER offered as an option.
Word and Outlook have a "think" about "you and "Your" and other variations like they want to get rid of the word altogether

it's gonna be a big World O' WE!  Microsoft's Big Huggy World.

If ONLY they could do something about the blasted quote marks

[questorfla]

OMG!  I just opened the last box of drives anbd they are ALL IDE.  I am not sure i even havd a bvoard with an IDE plug.  Nothing but fun around here.  Other day one guy comes up with an old Okidata (OLD-Old) printer and asks me why his USB can't work on it.  This was NOT the model that had both. Wanted an adapter for a USB to ..SERIAL..  Where in the heck he found the Printer itself I have no idea  I have not seen a serial printer since..  I think i was watching the Fonz on Happy Days?


I adapted it right to the trash bin.  Adapter wold cost at least 10 bucks IF one exists which is more than the printer would be worth IF it worked and it probably didn't.  This stuff is so old the salvation army wont take it.

That is why they want me to get these files off before there IS no more IDE even.

[Shades]

OMG!  I just opened the last box of drives anbd they are ALL IDE.  I am not sure i even havd a bvoard with an IDE plug.  Nothing but fun around here.  Other day one guy comes up with an old Okidata (OLD-Old) printer and asks me why his USB can't work on it.  This was NOT the model that had both. Wanted an adapter for a USB to ..SERIAL..  Where in the heck he found the Printer itself I have no idea  I have not seen a serial printer since..  I think i was watching the Fonz on Happy Days?


I adapted it right to the trash bin.  Adapter wold cost at least 10 bucks IF one exists which is more than the printer would be worth IF it worked and it probably didn't.  This stuff is so old the salvation army wont take it.

That is why they want me to get these files off before there IS no more IDE even.

Serial printers are common in POS setups. Industrial label printers (2000 USD and up) also work with the serial port. Thermal printers  often make use of the serial port too.

You might have to hunt a bit, but there are 3.5 inch portable hard disk enclosures that work with IDE drives.

[Stoic Joker]

One of my techs has one of these: C2G 30504 33in USB 2.0 to IDE/Serial ATA Drive Adapter Cable

It has proven to be about twice as handy as it looks...and it looks pretty handy to start with..

[questorfla]

What I DO have is a 2.5" IDE enclosure that I might be able to modify for a 3.5" drive.  
On my side a tool that I NEVER believed I would ever use is my WORMHOLE switch from by j5create. It sat sealed up for probably 2 months before I ever even tried it.

Claimed to have 5 GBPS data transfer rate and don't have to install anything other than the drivers that are built into the ends od the cable  The first time I used it I swear I thought it was some kind of trick.  Fastest file transfer I have ever seen: Even drive to drive transfer!    Won't be without one again!

REQUESTS from all of you who have been so helpful on this:

I have been looking for the fix to the issue that 4WD caught with the "quote marks" and I am sure I have seen one somewhere.  it is a patch or setting or something that makes ALL Quotes be the RIGHT quotes for someone who works on scripts not for people typing letters.  I am not sure but it may be a windows thing where you tell it not to use "smart-quotes" anywhere.  I never want to have to deal with the DUMB QUOTES again.  That was too much.

I also need ways to modify the specs on this nifty "junk cleaner" script.  What is the right way to tell it to look for all files ending in (for example)  .tmp"???" 

For this type of Archive I would love to search for/review/(and probably delete) all the ".exe" or ".com" files.  I have found during my transfers that there are quite a few and some are being scanned and removed by MalwareBytes or AVAST anyway.  OLD viruses and Malware that were, at the time, not in the database but are now.

And there are other extensions that if  could find them ALL just like I am finding the zero length files and have the option of removing them ALL  Or I could change the first echo command to "xcopy" so I could copy them to elsewhere before I ran the DEL version to get rid of them.

If I had been doing these from the start I hate to think how many junk files I now have in my nice clean 5TB Complete Documents Archive.

I need to come up with  list of "JUNK" like that and create a statement to cover each.  Another is those files created by WORD with the " ~ " as the first character to be a "ghost" copy of the file you are working on. 

Honestly, I have seen some where it looks like the people copied everything on their hard drive  including the entire Windows folder plus ProgramfilesX86 etc.  (Not many Thank God). 

The purpose of this whole effort is to create a massive database of every DOCUMENT type doesn't matter who wrote it.  I don't want to get too specific but almost anything that isn't a document is not even supposed to be here.  I have been making good use of a program called "Everything"  from the people at "Voidtools" as it is the absolute fastest "search and find" utility I have ever seen.  Finding all the ".exe" and removing them.

Still, I like the current scripted setup more because it allows me to have a list of all the files I moved or got rid of and what I did with (  to  )  them.

Thanks for all the input this has been an extremely helpful thread.

[4wd]

I would have thought that the type of files you want to keep would be a much shorter list than the type of files you want to remove.

But if you want to check on file extension just use that for the comparison instead of file size, eg.

"%%~xF"==".tmp"

IIRC, the ~ Word files all have an extension of .tmp so will be picked up using extension comparison providing you've also searched for hidden files.

[Shades]

Code: (text)

   @echo off
    echo @ECHO OFF>> zerobytefilesremover.bat
    for /r %%F in (*) do (if %%~zF==0 echo DEL /F /Q "%%F">> zerobytefilesremover.bat
                         ) else (
                            if %%~zF==".tmp" echo DEL /F /Q "%%F">> tempfilesremover.bat
                            ) else (
                                <fill in other arguments/conditions if you like> 
                             )

IF...ELSE... structure is allowed in batch script. But it makes the execution of it (a lot) slower. Just make sure that when you use the () characters in a script, you terminate each condition properly with a ). Decent text editors light these characters up when a set is terminated properly...so if you select one such character and it doesn't light up, you have to investigate your code. Although this lighting up is a great help, it won't save you from all possible error scenarios with the use of () characters.

But that is something you learn best the hard way...
Code indentation is your friend. It makes working with IF...ELSE... structures easier.

Another word to the wise, don't start the holy war between the use of spaces or tabs for indentation back up... 

[4wd]

if %%~zF==".tmp" echo DEL /F /Q "%%F">> tempfilesremover.bat

Should be %%~xF  


ScaryCmdFile.cmd:

Code: (text)

@echo off
rem ScaryCmdFile.cmd <path>
rem
rem eg. ScaryCmdFile.cmd U:\silly\files

rem Extensions to eliminate besides 0 byte files, just add them to the line below separated by a
rem semi-colon
rem LINE HAS TO END WITH SEMI-COLON
set ext=.tmp;.exe;.com;.bak;

for /r "%~1" %%F in (*) do (
  if %%~zF==0 echo DEL /F /Q "%%F">> zerobytefilesremover.bat
  echo.%ext% | findstr /C:"%%~xF;">nul && echo del /f /q "%%F">> naughtyextensionfileremover.bat
  )

Doesn't do the hidden temporary Word files naturally but that should just be a matter of dir /b /s /a+h "~*.*" > TempWordFiles.txt to get a list.

You could also turn ScaryCmdFile.cmd into SaveOurFiles.cmd by just specifying what extensions you want to keep and changing the last del /f /q to xcopy/robocopy/etc.

eg.

SaveOurFiles.cmd:

Code: (text)

@echo off
rem SaveOurFiles.cmd <path>
rem
rem eg. SaveOurFiles.cmd U:\silly\files

rem Extensions to keep, just add them to the line below separated by a semi-colon
rem Still removes 0 byte files
rem LINE HAS TO END WITH SEMI-COLON
set ext=.doc;.jpg;.xls;

for /r "%~1" %%F in (*) do (
  if %%~zF==0 echo DEL /F /Q "%%F">> zerobytefilesremover.bat
  echo.%ext% | findstr /C:"%%~xF;">nul && echo xcopy "%%F" c:\savedfiles\ >> filessavedfromextinction.bat
  )

NOTE: I've only tested the ScaryCmdFile.cmd.

Another word to the wise, don't start the holy war between the use of spaces or tabs for indentation back up...  

TABs?!  We don't need no steenkin' TABs!!