Author Topic: Recreate files but without content to target drive (Read 60703 times)

vixay · « **on:** June 09, 2009, 07:14 AM »

Is there a way to recreate file/folder structure from source to target but without any content?
better yet, is it possible to zip it up?

I think this will be better understood with an example.

Say if i have a source drive D: with the following

D:\test.txt (size 100 MB)
D:\a\b.ini (size 1 MB)
D:\z\y\x\c.pdf (size 1 GB)

After the zip/copy operation i want this on the target E: drive (can be another computer)

E:\test.txt (size 0 bytes)
E:\a\b.ini (size 0 bytes)
E:\z\y\x\c.pdf (size 0 bytes)

...

I was thinking that xxcopy or robocopy or some other copy utility or zip utility might have this feature but for the life of me i can't find it.

This is useful in recreating directory and file structures on which you can run various tests for finding files ...etc. and it would be easy to transfer over the internet without the size implications.
I have to do this for about 50000+ files
I've used Cathy, Locate, Everything, and i know those things are wicked fast, and they have the information i need, but then how to create a routine/plugin to create those files?

I know there is a manual way to do it, something like dump all filenames to a text file, zip it up, and then extract it, and send it through a program that will create a file for each line. But then this requires the program to exist on one end, whereas if it were a zip it would work wherever.

Sigh, i'm over thinking this. But any ideas will be appreciated.

MilesAhead · « **Reply #1 on:** June 09, 2009, 11:02 AM »

I haven't used it myself, but this looks like what you mean:
http://www.rjlsoftwa...eecopy/default.shtml

Edvard · « **Reply #2 on:** June 09, 2009, 01:03 PM »

xPlorer² will do that...

Select the folder you want to copy, then right-click -> copy.

Navigate to where you want to paste it and right-click -> Paste Special -> Folder Structure.

Done.

akx · « **Reply #3 on:** June 09, 2009, 02:15 PM »

I wrote a 45-line Python script to create a contentless ZIP file that duplicates a directory structure for you

.

It's a command line script. Invoke it using "python maketree.py target-zip-file.zip mydir1 mydir2 ..." You can combine multiple trees into one zip.

Source code here (public domain). If you need an .exe version, let me know.

skwire · « **Reply #4 on:** June 09, 2009, 06:15 PM »

I think everybody is missing the key element here...it's not only the folder structure the OP wants, it's all the files, too (as zero-byte files).

Steven Avery · « **Reply #5 on:** June 09, 2009, 07:04 PM »

Hi Folks,

Good question. A while back I was also thinking about this with an email program like Eudora when you want to "start fresh" with new data. You might need all the .mbx files (or whatever extension) and optionally the index or table of contents files (those would build themselves) but you want them empty to start, like they are before they receive any mail.

Then all your mail up to that point could be saved or archived, and you can start receiving mail using the exact same filter structure. (The key point, without the empty files the filters will give errors, they do not build on the fly, although maybe in Bat-land you would have that option.) In a rudimentary form it would be drive-to-drive (eg. c:/myfiles to d:/myfiles ) .. in a slightly more sophisticated form it would be any point-to-point. (e.g c:/myfiles to c:/newfiles/mail ).

Although it is a far less sophisticated need than the one you are mentioning, a tool would be nice, even a two step tool (tree-copy followed by "copy file to empty").

Shalom,
Steven Avery

skwire · « **Reply #6 on:** June 10, 2009, 01:03 AM »

Is there a way to recreate file/folder structure from source to target but without any content?
better yet, is it possible to zip it up?
-vixay (June 09, 2009, 07:14 AM)

I have this mostly done but have some questions.

1) Do you have need of the recreated zero-byte structure? Or, do you just care about the resulting zip file? The reason I ask is that if you don't, I'll just build the new structure in the user's temp folder and delete it when I'm done. Make sense?

2) I currently have the zip being generated with relative paths set. What this means is that you can unzip the resulting zip file to any folder and the structure will be built there. Is that okay?

skwire · « **Reply #7 on:** June 10, 2009, 06:21 AM »

Download Zero Zipper and see if this is what you had in mind.

Download: http://skwire.dcmemb...ipper/ZeroZipper.zip

vixay · « **Reply #8 on:** June 10, 2009, 07:10 AM »

SkWire that's perfect!

And exactly what i had in mind! Pure genius! I tried it out and works. And you guys did this in less than 24 hours! Sending some credits your way.

And you are right, I needed the 0 byte files not just the folders.

Some further thoughts. The test zip file i created has 18,375 files and weighs in at 3.27 MB. That must mean that the filenames storage takes up that much space. And that zips don't optimize that part? or something else?

Scanning for files was slower than everything/locate, i don't know how they do it, but they can scan filenames super fast. just a thought, to see if it could be sped up. Since you don't actually need to touch the file, and just need the information from the MFT.

The date/time is not preserved for filenames. This was not an original requirement but when i think about it, it's better to have that information preserved (not required though). If it will make the program slower then it should not be there (or maybe as an optional thing).

Can you share your ideas on how you did it? You used IZarc to compress, did you just use flags for the command line , or dump the filename list and then create the zip?

Fabulous work!

f0dder · « **Reply #9 on:** June 10, 2009, 07:20 AM »

Some further thoughts. The test zip file i created has 18,375 files and weighs in at 3.27 MB. That must mean that the filenames storage takes up that much space. And that zips don't optimize that part? or something else?
-vixay (June 10, 2009, 07:10 AM)

Indeed they don't - it would make reading the zip folder structure much slower, and to achieve any kind of reasonable compression you'd need to compress just the filenames... which would mean quite a restructure of the file format, and require either unpacking all the filename information at once, or some "somewhat interesting" code.

Scanning for files was slower than everything/locate, i don't know how they do it, but they can scan filenames super fast. just a thought, to see if it could be sped up. Since you don't actually need to touch the file, and just need the information from the MFT.
-vixay (June 10, 2009, 07:10 AM)

Locate builds an efficient index file, which can be read faster than the MFT (and doesn't have security-check overhead either). Everything, as far as I can tell, scans the MFT directly (which is why it requires administrator privileges).

The date/time is not preserved for filenames. This was not an original requirement but when i think about it, it's better to have that information preserved (not required though). If it will make the program slower then it should not be there (or maybe as an optional thing).
-vixay (June 10, 2009, 07:10 AM)

It shouldn't make the zip-creation slower, since you also get file dates when scanning for files.

PS: you could try zipping the generated zipfile, might be able to shave off a little of the filenames.

skwire · « **Reply #10 on:** June 10, 2009, 08:01 AM »

Some further thoughts. The test zip file i created has 18,375 files and weighs in at 3.27 MB. That must mean that the filenames storage takes up that much space. And that zips don't optimize that part? or something else?
-vixay (June 10, 2009, 07:10 AM)

Yes, the filenames take up that much space. I tried it with compression and it didn't make a difference in size.

Scanning for files was slower than everything/locate, i don't know how they do it, but they can scan filenames super fast. just a thought, to see if it could be sped up.
-vixay (June 10, 2009, 07:10 AM)

It's not the scanning that's slow, it's the creation of all the zero-byte files that slows things down.

The date/time is not preserved for filenames. This was not an original requirement but when i think about it, it's better to have that information preserved (not required though). If it will make the program slower then it should not be there (or maybe as an optional thing).
-vixay (June 10, 2009, 07:10 AM)

It shouldn't make it too much slower. I'll add it in.

Can you share your ideas on how you did it? You used IZarc to compress, did you just use flags for the command line , or dump the filename list and then create the zip?
-vixay (June 10, 2009, 07:10 AM)

Just command-line flags (izarcc.exe -a -r -p -c0). So, the flow goes like this:

1) Recursive scan through source folder and re-create it using zero-byte files in user's temp folder.
2) Zip up temp folder with relative paths set.
3) Delete temp folder.

Does that help clear things up?

mouser · « **Reply #11 on:** June 10, 2009, 08:05 AM »

make sure you exclude the temp folder from the zipping operation if the user ever tries to add all of C:

skwire · « **Reply #12 on:** June 10, 2009, 08:07 AM »

Bahaha...the recursive spiral of doom.

skwire · « **Reply #13 on:** June 10, 2009, 08:21 AM »

vixay, try the latest...it should keep the timestamps properly.

Download: http://skwire.dcmemb...ipper/ZeroZipper.zip

f0dder · « **Reply #14 on:** June 10, 2009, 08:32 AM »

Ooooh, you use an external tool to create the zip instead of constructing it directly? No wonder it's slow then

(Since you're only dealing with 0-byte files, creating the zip manually from code shouldn't be that big a deal, since you only need to deal with the logical zip structure and not compression of data).

skwire · « **Reply #15 on:** June 10, 2009, 10:36 AM »

It's not that slow.

I gave it a test on a folder with 5200 files (184 folders) in it and it was done in around eight seconds.

akx · « **Reply #16 on:** June 10, 2009, 03:10 PM »

I guess I was unclear in my Python script post, but it also adds the zero-byte files with correct modification dates >__>

skwire · « **Reply #17 on:** June 10, 2009, 04:29 PM »

I guess I was unclear in my Python script post, but it also adds the zero-byte files with correct modification dates >__>
-akx (June 10, 2009, 03:10 PM)

I just tried your script...very nice work. =]

vixay · « **Reply #18 on:** June 10, 2009, 11:06 PM »

Akx, you are right. I didn't realize it! Thanks for the script! I didn't have python installed, (doing it now), so i didn't have a quick way of trying it. I guess that means an exe would be welcome as well

.

skwire, you should upload your webpage so that we can quickly see your apps. and regarding the speed, it is not slow, it's just that some of those nifty utilities have spoiled us with their speed, what with their scans of terabytes of data in seconds.

I did a test, 56 seconds for 18,736 files

same test for the script. it failed, doesn't support unicode.

J:\Share\Programs\@Network\eToolz\Language\English.lng
Traceback (most recent call last):
File "makeTreeZip.py", line 43, in <module>
mtime=time.localtime(os.stat(f).st_mtime)
WindowsError: [Error 123] The filename, directory name, or volume label syntax i
s incorrect: 'J:\\Share\\Programs\\@Network\\eToolz\\Language\\Fran?ais.bmp'

mouser · « **Reply #19 on:** June 10, 2009, 11:18 PM »

Nice work both of you..

akx: very very cool stuff

akx · « **Reply #20 on:** June 11, 2009, 11:33 AM »

same test for the script. it failed, doesn't support unicode.
J:\Share\Programs\@Network\eToolz\Language\English.lng
Traceback (most recent call last):
File "makeTreeZip.py", line 43, in <module>
mtime=time.localtime(os.stat(f).st_mtime)
WindowsError: [Error 123] The filename, directory name, or volume label syntax i
s incorrect: 'J:\\Share\\Programs\\@Network\\eToolz\\Language\\Fran?ais.bmp'
-vixay (June 10, 2009, 11:06 PM)

Oopsy. Let me fix that!
.... There! http://codepad.org/ftWwtTGy

That version should handle
(a) zipping empty directories
(b) unicode filenames (as long as it can decode the command line...)
Picture proof here!

vixay · « **Reply #21 on:** July 20, 2009, 10:49 PM »

i had another thought that might work.
Grab list of file & folder names, and put it in text file with created/modified dates.
zip the text file. output file

on destination take output file, extract text file, create files according to text file.

This would result in a smaller file ultimately and be quicker as there are fewer disk I/O.

Actually this could be done with one utility that dumps directory structure to file, and can recreate directory structure from file. the zipping part could be manual, this way we could easily use text/excel programs to generate a desired file structure as well. I'm sure unix has a tool to do this, where you can redirect a text file to the mkdir command or something.
The problem i see with this approach is how to handle created/modified times. I guess you could output a tab delimited file.

I guess this is a tangent and just ideas. The current implementation will have a wider compatibility since it uses a standard zip file.

skwire · « **Reply #22 on:** July 20, 2009, 10:59 PM »

I can easily modify Zero Zipper to do this. In fact, it's a great idea.

Brycestro · « **Reply #23 on:** November 15, 2009, 08:43 AM »

same test for the script. it failed, doesn't support unicode.
J:\Share\Programs\@Network\eToolz\Language\English.lng
Traceback (most recent call last):
File "makeTreeZip.py", line 43, in <module>
mtime=time.localtime(os.stat(f).st_mtime)
WindowsError: [Error 123] The filename, directory name, or volume label syntax i
s incorrect: 'J:\\Share\\Programs\\@Network\\eToolz\\Language\\Fran?ais.bmp'
-vixay (June 10, 2009, 11:06 PM)
Oopsy. Let me fix that!
.... There! http://codepad.org/ftWwtTGy

That version should handle
(a) zipping empty directories
(b) unicode filenames (as long as it can decode the command line...)
Picture proof here!
-akx (June 11, 2009, 11:33 AM)

Hi akx,

Great script - I'm using it to replicate the file and folder structure on my hard-disk on a schedule so i can take the result and email it to myself with a little extra python scripting. I already clone my hard-disk's contents once every week or so, but I can use this script to replicate the structure on a more regular basis (say once every 12 hours) and so when a drive does fail I can do a directory compare between the replicated structure on email and my last full clone to find out exactly what files I've completely lost, accurate to within 12 hours. Of course this doesn't help telling me what new changes I've lost to files that did make it into my last disk clone, but it's still helpful.

One small thing I noticed about your script is that it doesn't close off the zipfile when it's done replicating the folder/file structure. I added the line zf.close() to mine, before adding my own logic to email it to myself. I found myself running into problems with the email part if I didn't do this first. I imagine that once your script is fully done it must have been closing off the zipfile implicitly and so it's not really a problem for people not adding anything to it, but thought it could be worth mentioning for anyone else who may be looking to tweak your script.

vixay · « **Reply #24 on:** March 25, 2010, 07:23 AM »

Ok I needed to use the utility again and decided to do the speed comparison.

I ran it on A folder with

[Select]

67.8 GB (72,906,108,928 bytes)
54,824 Files, 5,364 Folders

Here's ZeroZipper

Start - 06:29:44 PM
End - 06:36:41 PM

It took 7 minutes!! Out of which 5 minutes was spent just recreating the 0-byte structure! 1 minute to delete files and <1 to create the zip.
So this method is definitely not good for large folders.
And I think I found a bug. For the zip I didn't put a path just a name "zz.zip", the zip was created in the temp correctly but then it didn't move it to the same location as the zerozipper... it was strange... and the zip was then deleted! so I didn't get any output after waiting for so long!

I then had high hopes for the python version, as it would avoid the slow Disk I/O. So
i finally used the py2exe and converted the code to an exe and tried that (rather than the script just to make a fair comparison).
It died again on unicode filenames

. I did use the new version too this time!

Traceback (most recent call last):
File "makeTreeZip.py", line 61, in <module>
mtime=time.localtime(os.stat(f).st_mtime)
WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: 'J:\\Share\\Categorized Appz\\Compression Utilities\\Just Extractor\\??-readme.txt'

Can someone help with this problem? It seems like the filenames which contain unrecognizable characters are converted to ?? and then we we use them again it crashes... how do we retain the filenames in the original format? regardless of whether it contains Legible/illegible unicode?

After this i tried running the script directly and it seemed to work but then
I encountered another mysterious error

Traceback (most recent call last):
File "makeTreeZip.py", line 63, in <module>
mtime=time.localtime(os.stat(f).st_mtime)
ValueError: (22, 'Invalid argument')

I tried debugging this the value for st_mtime was 1189361328.0 i converted that using time.localtime() and it worked, so i can't figure out exactly what is causing this error.

Actually the file in question have a modified date of

Tuesday, September 25, 3427, 10:40:30 PM

so i can see how python would choke on that. how can i resolve this issue?