Welcome Guest.   Make a donation to an author on the site November 28, 2014, 07:30:56 AM  *

Please login or register.
Or did you miss your validation email?


Login with username and password (forgot your password?)
Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.


You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.
 
The N.A.N.Y. Challenge 2012! Download dozens of custom programs!
   
   Forum Home   Thread Marks Chat! Downloads Search Login Register  
Pages: [1] 2 Next   Go Down
  Reply  |  New Topic  |  Print  
Author Topic: Recreate files but without content to target drive  (Read 20380 times)
vixay
Supporting Member
**
Posts: 137



ViXaY

see users location on a map View Profile Give some DonationCredits to this forum member
« on: June 09, 2009, 07:14:25 AM »

Is there a way to recreate file/folder structure from source to target but without any content?
better yet, is it possible to zip it up?

I think this will be better understood with an example.

Say if i have a source drive D: with the following

Quote
D:\test.txt (size 100 MB)
D:\a\b.ini (size 1 MB)
D:\z\y\x\c.pdf (size 1 GB)

After the zip/copy operation i want this on the target E: drive (can be another computer)

Quote
E:\test.txt (size 0 bytes)
E:\a\b.ini (size 0 bytes)
E:\z\y\x\c.pdf (size 0 bytes)

...

I was thinking that xxcopy or robocopy or some other copy utility or zip utility might have this feature but for the life of me i can't find it.

This is useful in recreating directory and file structures on which you can run various tests for finding files ...etc. and it would be easy to transfer over the internet without the size implications.
I have to do this for about 50000+ files
I've used Cathy, Locate, Everything, and i know those things are wicked fast, and they have the information i need, but then how to create a routine/plugin to create those files?

I know there is a manual way to do it, something like dump all filenames to a text file, zip it up, and then extract it, and send it through a program that will create a file for each line. But then this requires the program to exist on one end, whereas if it were a zip it would work wherever.

Sigh, i'm over thinking this. But any ideas will be appreciated.

« Last Edit: June 09, 2009, 07:19:18 AM by vixay » Logged

"Drunk on the Nectar of Life!" -me
MilesAhead
Member
**
Posts: 5,055



View Profile WWW Give some DonationCredits to this forum member
« Reply #1 on: June 09, 2009, 11:02:07 AM »

I haven't used it myself, but this looks like what you mean:
http://www.rjlsoftware.co...ty/treecopy/default.shtml
Logged

"It's only called research if you didn't find it the first time."
-- MilesAhead
Edvard
Coding Snacks Author
Charter Honorary Member
***
Posts: 2,625



View Profile Give some DonationCredits to this forum member
« Reply #2 on: June 09, 2009, 01:03:03 PM »

xPlorer2 will do that...

Select the folder you want to copy, then right-click -> copy.

Navigate to where you want to paste it and right-click -> Paste Special -> Folder Structure.

Done.
Logged

All children left unattended will be given a mocha and a puppy.
akx
Supporting Member
**
Posts: 20

View Profile Give some DonationCredits to this forum member
« Reply #3 on: June 09, 2009, 02:15:09 PM »

I wrote a 45-line Python script to create a contentless ZIP file that duplicates a directory structure for you smiley.

It's a command line script. Invoke it using "python maketree.py target-zip-file.zip mydir1 mydir2 ..." You can combine multiple trees into one zip.

Source code here (public domain). If you need an .exe version, let me know.
Logged
skwire
Global Moderator
*****
Posts: 4,158



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #4 on: June 09, 2009, 06:15:23 PM »

I think everybody is missing the key element here...it's not only the folder structure the OP wants, it's all the files, too (as zero-byte files).
Logged

Steven Avery
Supporting Member
**
Posts: 671

View Profile Give some DonationCredits to this forum member
« Reply #5 on: June 09, 2009, 07:04:32 PM »

Hi Folks,

Good question.  A while back I was also thinking about this with an email program like Eudora when you want to "start fresh" with new data. You might need all the .mbx files (or whatever extension) and optionally the index or table of contents files (those would build themselves) but you want them empty to start, like they are before they receive any mail.  

Then all your mail up to that point could be saved or archived, and you can start receiving mail using the exact same filter structure. (The key point, without the empty files the filters will give errors, they do not build on the fly, although maybe in Bat-land you would have that option.)  In a rudimentary form it would be drive-to-drive (eg. c:/myfiles to d:/myfiles ) .. in a slightly more sophisticated form it would be any point-to-point. (e.g c:/myfiles to c:/newfiles/mail ).

Although it is a far less sophisticated need than the one you are mentioning, a tool would be nice, even a two step tool (tree-copy followed by "copy file to empty").

Shalom,
Steven Avery
« Last Edit: June 09, 2009, 07:15:14 PM by Steven Avery » Logged
skwire
Global Moderator
*****
Posts: 4,158



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #6 on: June 10, 2009, 01:03:38 AM »

Is there a way to recreate file/folder structure from source to target but without any content?
better yet, is it possible to zip it up?

I have this mostly done but have some questions.

1) Do you have need of the recreated zero-byte structure?  Or, do you just care about the resulting zip file?  The reason I ask is that if you don't, I'll just build the new structure in the user's temp folder and delete it when I'm done.  Make sense?

2) I currently have the zip being generated with relative paths set.  What this means is that you can unzip the resulting zip file to any folder and the structure will be built there.  Is that okay?
Logged

skwire
Global Moderator
*****
Posts: 4,158



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #7 on: June 10, 2009, 06:21:43 AM »

Download Zero Zipper and see if this is what you had in mind.



Download: http://skwire.dcmembers.c...ero_zipper/ZeroZipper.zip
« Last Edit: July 04, 2009, 10:51:50 AM by skwire » Logged

vixay
Supporting Member
**
Posts: 137



ViXaY

see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #8 on: June 10, 2009, 07:10:35 AM »

SkWire that's perfect!  thumbs up thumbs up And exactly what i had in mind! Pure genius! I tried it out and works. And you guys did this in less than 24 hours! Sending some credits your way.

And you are right, I needed the 0 byte files not just the folders.

Some further thoughts. The test zip file i created has 18,375 files and weighs in at 3.27 MB. That must mean that the filenames storage takes up that much space. And that zips don't optimize that part? or something else?

Scanning for files was slower than everything/locate, i don't know how they do it, but they can scan filenames super fast. just a thought, to see if it could be sped up. Since you don't actually need to touch the file, and just need the information from the MFT.

The date/time is not preserved for filenames. This was not an original requirement but when i think about it, it's better to have that information preserved  (not required though). If it will make the program slower then it should not be there (or maybe as an optional thing).

Can you share your ideas on how you did it? You used IZarc to compress, did you just use flags for the command line , or dump the filename list and then create the zip?

Fabulous work!
Logged

"Drunk on the Nectar of Life!" -me
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #9 on: June 10, 2009, 07:20:57 AM »

Some further thoughts. The test zip file i created has 18,375 files and weighs in at 3.27 MB. That must mean that the filenames storage takes up that much space. And that zips don't optimize that part? or something else?
Indeed they don't - it would make reading the zip folder structure much slower, and to achieve any kind of reasonable compression you'd need to compress just the filenames... which would mean quite a restructure of the file format, and require either unpacking all the filename information at once, or some "somewhat interesting" code.

Scanning for files was slower than everything/locate, i don't know how they do it, but they can scan filenames super fast. just a thought, to see if it could be sped up. Since you don't actually need to touch the file, and just need the information from the MFT.
Locate builds an efficient index file, which can be read faster than the MFT (and doesn't have security-check overhead either). Everything, as far as I can tell, scans the MFT directly (which is why it requires administrator privileges).

The date/time is not preserved for filenames. This was not an original requirement but when i think about it, it's better to have that information preserved  (not required though). If it will make the program slower then it should not be there (or maybe as an optional thing).
It shouldn't make the zip-creation slower, since you also get file dates when scanning for files.

PS: you could try zipping the generated zipfile, might be able to shave off a little of the filenames.
Logged

- carpe noctem
skwire
Global Moderator
*****
Posts: 4,158



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #10 on: June 10, 2009, 08:01:18 AM »

Some further thoughts. The test zip file i created has 18,375 files and weighs in at 3.27 MB. That must mean that the filenames storage takes up that much space. And that zips don't optimize that part? or something else?
Yes, the filenames take up that much space.  I tried it with compression and it didn't make a difference in size.

Scanning for files was slower than everything/locate, i don't know how they do it, but they can scan filenames super fast. just a thought, to see if it could be sped up.
It's not the scanning that's slow, it's the creation of all the zero-byte files that slows things down.

The date/time is not preserved for filenames. This was not an original requirement but when i think about it, it's better to have that information preserved  (not required though). If it will make the program slower then it should not be there (or maybe as an optional thing).
It shouldn't make it too much slower.  I'll add it in.

Can you share your ideas on how you did it? You used IZarc to compress, did you just use flags for the command line , or dump the filename list and then create the zip?
Just command-line flags (izarcc.exe -a -r -p -c0).  So, the flow goes like this:

1) Recursive scan through source folder and re-create it using zero-byte files in user's temp folder.
2) Zip up temp folder with relative paths set.
3) Delete temp folder.

Does that help clear things up?
Logged

mouser
First Author
Administrator
*****
Posts: 33,703



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #11 on: June 10, 2009, 08:05:14 AM »

make sure you exclude the temp folder from the zipping operation if the user ever tries to add all of C:
Logged
skwire
Global Moderator
*****
Posts: 4,158



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #12 on: June 10, 2009, 08:07:50 AM »

Bahaha...the recursive spiral of doom.
Logged

skwire
Global Moderator
*****
Posts: 4,158



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #13 on: June 10, 2009, 08:21:19 AM »

vixay, try the latest...it should keep the timestamps properly.

Download: http://skwire.dcmembers.c...ero_zipper/ZeroZipper.zip
Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #14 on: June 10, 2009, 08:32:06 AM »

Ooooh, you use an external tool to create the zip instead of constructing it directly? No wonder it's slow then smiley

(Since you're only dealing with 0-byte files, creating the zip manually from code shouldn't be that big a deal, since you only need to deal with the logical zip structure and not compression of data).
Logged

- carpe noctem
skwire
Global Moderator
*****
Posts: 4,158



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #15 on: June 10, 2009, 10:36:07 AM »

It's not that slow.  Sad  I gave it a test on a folder with 5200 files (184 folders) in it and it was done in around eight seconds.
Logged

akx
Supporting Member
**
Posts: 20

View Profile Give some DonationCredits to this forum member
« Reply #16 on: June 10, 2009, 03:10:35 PM »

I guess I was unclear in my Python script post, but it also adds the zero-byte files with correct modification dates >__>
Logged
skwire
Global Moderator
*****
Posts: 4,158



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #17 on: June 10, 2009, 04:29:35 PM »

I guess I was unclear in my Python script post, but it also adds the zero-byte files with correct modification dates >__>
I just tried your script...very nice work.  =]
Logged

vixay
Supporting Member
**
Posts: 137



ViXaY

see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #18 on: June 10, 2009, 11:06:17 PM »

Akx, you are right. I didn't realize it! Thanks for the script! I didn't have python installed, (doing it now), so i didn't have a quick way of trying it. I guess that means an exe would be welcome as well tongue.

skwire, you should upload your webpage so that we can quickly see your apps. and regarding the speed, it is not slow, it's just that some of those nifty utilities have spoiled us with their speed, what with their scans of terabytes of data in seconds. tongue

I did a test, 56 seconds for 18,736 files

same test for the script. it failed, doesn't support unicode. Sad
Quote
J:\Share\Programs\@Network\eToolz\Language\English.lng
Traceback (most recent call last):
  File "makeTreeZip.py", line 43, in <module>
    mtime=time.localtime(os.stat(f).st_mtime)
WindowsError: [Error 123] The filename, directory name, or volume label syntax i
s incorrect: 'J:\\Share\\Programs\\@Network\\eToolz\\Language\\Fran?ais.bmp'
« Last Edit: June 10, 2009, 11:24:17 PM by vixay » Logged

"Drunk on the Nectar of Life!" -me
mouser
First Author
Administrator
*****
Posts: 33,703



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #19 on: June 10, 2009, 11:18:57 PM »

Nice work both of you..

akx: very very cool stuff  thumbs up
Logged
akx
Supporting Member
**
Posts: 20

View Profile Give some DonationCredits to this forum member
« Reply #20 on: June 11, 2009, 11:33:40 AM »

same test for the script. it failed, doesn't support unicode. Sad
Quote
J:\Share\Programs\@Network\eToolz\Language\English.lng
Traceback (most recent call last):
  File "makeTreeZip.py", line 43, in <module>
    mtime=time.localtime(os.stat(f).st_mtime)
WindowsError: [Error 123] The filename, directory name, or volume label syntax i
s incorrect: 'J:\\Share\\Programs\\@Network\\eToolz\\Language\\Fran?ais.bmp'
Oopsy. Let me fix that!
.... There! http://codepad.org/ftWwtTGy

That version should handle
(a) zipping empty directories
(b) unicode filenames (as long as it can decode the command line...)
Picture proof here!
Logged
vixay
Supporting Member
**
Posts: 137



ViXaY

see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #21 on: July 20, 2009, 10:49:23 PM »

i had another thought that might work.
Grab list of file & folder names, and put it in text file with created/modified dates.
zip the text file. output file

on destination take output file, extract text file, create files according to text file.

This would result in a smaller file ultimately and be quicker as there are fewer disk I/O.

Actually this could be done with one utility that dumps directory structure to file, and can recreate directory structure from file. the zipping part could be manual, this way we could easily use text/excel programs to generate a desired file structure as well. I'm sure unix has a tool to do this, where you can redirect a text file to the mkdir command or something.
The problem i see with this approach is how to handle created/modified times. I guess you could output a tab delimited file.

I guess this is a tangent and just ideas. The current implementation will have a wider compatibility since it uses a standard zip file.

« Last Edit: July 20, 2009, 10:52:14 PM by vixay » Logged

"Drunk on the Nectar of Life!" -me
skwire
Global Moderator
*****
Posts: 4,158



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #22 on: July 20, 2009, 10:59:28 PM »

I can easily modify Zero Zipper to do this.  In fact, it's a great idea.
Logged

Brycestro
Participant
*
Posts: 1

View Profile Give some DonationCredits to this forum member
« Reply #23 on: November 15, 2009, 08:43:12 AM »

same test for the script. it failed, doesn't support unicode. Sad
Quote
J:\Share\Programs\@Network\eToolz\Language\English.lng
Traceback (most recent call last):
  File "makeTreeZip.py", line 43, in <module>
    mtime=time.localtime(os.stat(f).st_mtime)
WindowsError: [Error 123] The filename, directory name, or volume label syntax i
s incorrect: 'J:\\Share\\Programs\\@Network\\eToolz\\Language\\Fran?ais.bmp'
Oopsy. Let me fix that!
.... There! http://codepad.org/ftWwtTGy

That version should handle
(a) zipping empty directories
(b) unicode filenames (as long as it can decode the command line...)
Picture proof here!

Hi akx,

Great script - I'm using it to replicate the file and folder structure on my hard-disk on a schedule so i can take the result and email it to myself with a little extra python scripting.  I already clone my hard-disk's contents once every week or so, but I can use this script to replicate the structure on a more regular basis (say once every 12 hours) and so when a drive does fail I can do a directory compare between the replicated structure on email and my last full clone to find out exactly what files I've completely lost, accurate to within 12 hours.  Of course this doesn't help telling me what new changes I've lost to files that did make it into my last disk clone, but it's still helpful. 

One small thing I noticed about your script is that it doesn't close off the zipfile when it's done replicating the folder/file structure.  I added the line zf.close() to mine, before adding my own logic to email it to myself.  I found myself running into problems with the email part if I didn't do this first.  I imagine that once your script is fully done it must have been closing off the zipfile implicitly and so it's not really a problem for people not adding anything to it, but thought it could be worth mentioning for anyone else who may be looking to tweak your script.
Logged
vixay
Supporting Member
**
Posts: 137



ViXaY

see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #24 on: March 25, 2010, 07:23:08 AM »

Ok I needed to use the utility again and decided to do the speed comparison.

I ran it on A folder with
[copy or print]
67.8 GB (72,906,108,928 bytes)
54,824 Files, 5,364 Folders

Here's ZeroZipper
Quote
Start - 06:29:44 PM
End - 06:36:41 PM
It took 7 minutes!! Out of which 5 minutes was spent just recreating the 0-byte structure! 1 minute to delete files and <1 to create the zip.
So this method is definitely not good for large folders.
And I think I found a bug. For the zip I didn't put a path just a name "zz.zip", the zip was created in the temp correctly but then it didn't move it to the same location as the zerozipper... it was strange... and the zip was then deleted! so I didn't get any output after waiting for so long!

I then had high hopes for the python version, as it would avoid the slow Disk I/O. So
 i finally used the py2exe and converted the code to an exe and tried that (rather than the script just to make a fair comparison).
It died again on unicode filenames Sad. I did use the new version too this time!
Quote
Traceback (most recent call last):
  File "makeTreeZip.py", line 61, in <module>
    mtime=time.localtime(os.stat(f).st_mtime)
WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: 'J:\\Share\\Categorized Appz\\Compression Utilities\\Just Extractor\\??-readme.txt'

Can someone help with this problem? It seems like the filenames which contain unrecognizable characters are converted to ?? and then we we use them again it crashes... how do we retain the filenames in the original format? regardless of whether it contains Legible/illegible unicode?

After this i tried running the script directly and it seemed to work but then
I encountered another mysterious error
Quote
Traceback (most recent call last):
  File "makeTreeZip.py", line 63, in <module>
    mtime=time.localtime(os.stat(f).st_mtime)
ValueError: (22, 'Invalid argument')

I tried debugging this the value for st_mtime was 1189361328.0 i converted that using time.localtime() and it worked, so i can't figure out exactly what is causing this error.

Actually the file in question have a modified date of
Quote
Tuesday, September 25, 3427, 10:40:30 PM
so i can see how python would choke on that. how can i resolve this issue?
« Last Edit: March 25, 2010, 07:26:43 AM by vixay » Logged

"Drunk on the Nectar of Life!" -me
Pages: [1] 2 Next   Go Up
  Reply  |  New Topic  |  Print  
 
Jump to:  
   Forum Home   Thread Marks Chat! Downloads Search Login Register  

DonationCoder.com | About Us
DonationCoder.com Forum | Powered by SMF
[ Page time: 0.055s | Server load: 0.07 ]