Author Topic: Compressing files into multiple .zip archives with each .zip being independent (Read 202,041 times)

testgames · « **Reply #50 on:** December 10, 2007, 10:56 AM »

Hi,

1. we are talking about around a few hundreds of sub-directories, which contain many small .dat files.
2. There are about 11 levels. every level contains around 4-5 levels so the hole tree is about 400,000 different directories (an encoded game). The hole tree is about 7 GB.
3. I don't really understand what do you mean by output mode, but I guess standard.

skwire · « **Reply #51 on:** December 10, 2007, 11:34 AM »

Quote from: testgames on December 10, 2007, 10:56 AM

1. we are talking about around a few hundreds of sub-directories, which contain many small .dat files.
2. There are about 11 levels. every level contains around 4-5 levels so the hole tree is about 400,000 different directories (an encoded game). The hole tree is about 7 GB.

Since it's such a huge directory that you're working on, is there any particular reason you want to use SpInZip over any of the much more complete and capable compression apps out there (honest question...I'm not trying to be a smartass)?

Quote from: testgames on December 10, 2007, 10:56 AM

3. I don't really understand what do you mean by output mode, but I guess standard.

Sorry, by output mode, I mean which of the three output modes that SpInZip offers were you attempting to use (and the settings you were attempting to use with that output mode)? If the aforementioned 2GB zip limit is a problem, I can easily add support for .7z files. However, you will need 7-zip (free @ http://www.7-zip.org) to unpack the resultant files.

testgames · « **Reply #52 on:** December 12, 2007, 03:54 PM »

here is the story:
1. My customer has an old version of Winzip. He can not unzip files more than 2 GB size.
2. I am sending to this customers many games (hugh sized) which are more that 2 GB each.
3. what we did until now is to manually close zip files up to 2 GB and send them seperatly (so he can unzip them one by one). for example: a 7 GB game = 4 separate zip files.
4. He does not have tar or rar or any other unzip mechanizm nor he will not upgrade his winzip version to support more than 2 GB files.
5. I need the zip files to be stand alone - so I can open them by themsleves and not X.zip.001, X.zip.002 which comines together are a part of a bigger zip.
got it ?

testgames · « **Reply #53 on:** December 13, 2007, 12:52 PM »

1. I used the 1st option for Output mode: "By Approzimate Zip File Size" = 2048 MB
2. I don't want to use the 7-zip to unpack the file (the customer will not install it at his network) so I need it to be open only by unzip.

skwire · « **Reply #54 on:** December 17, 2007, 01:53 PM »

Quote from: testgames on December 12, 2007, 03:54 PM

4. He does not have tar or rar or any other unzip mechanizm nor he will not upgrade his winzip version to support more than 2 GB files.
5. I need the zip files to be stand alone - so I can open them by themsleves and not X.zip.001, X.zip.002 which comines together are a part of a bigger zip.

Since he won't upgrade, have you tried making a self-extracting spanned zip file that you could send him?

Quote from: testgames on December 13, 2007, 12:52 PM

1. I used the 1st option for Output mode: "By Approzimate Zip File Size" = 2048 MB
2. I don't want to use the 7-zip to unpack the file (the customer will not install it at his network) so I need it to be open only by unzip.

The closest directory I have to yours would be my mail server folder. It currently has 581,395 files in it comprising 12.1GB of space. I'll try testing against it and see what happens.

testgames · « **Reply #55 on:** December 18, 2007, 12:02 PM »

I can make self extracting spanned zip files (at 2 GB size) but the problem is I have around 400 of these to make and it will take me too much time, therefore I've been looking for a tool to make it.

Testing it on the mail server can be a great test - thanks a lot, looking forward for the results.

skwire · « **Reply #56 on:** December 18, 2007, 02:06 PM »

Quote from: testgames on December 18, 2007, 12:02 PM

I can make self extracting spanned zip files (at 2 GB size) but the problem is I have around 400 of these to make and it will take me too much time, therefore I've been looking for a tool to make it.

Gotcha.

Quote from: testgames on December 18, 2007, 12:02 PM

Testing it on the mail server can be a great test - thanks a lot, looking forward for the results.

I've made some changes and they look promising. I'm currently running a test on my mail server directory using similar settings to what you were trying (Option 1, 2000MB). It seems to be working fine. The reason I'm not using 2048MB is that, as I mentioned in a previous post, I am not doing any sort of magical algorithm that attempts to make perfects-sized zip files. I simply recurse through the input folder and keep a running total of the file sizes. As soon as that total goes OVER your specified size, I zip those files. Since you're attempting to push the upper limit of a normal zip file, I'd suggest you use something lower than 2048MB since you run the risk of exceeding that amount. That why I put "approximate" on the app's interface. Does that make sense?

In addition, I should be able to support zips greater than 2GB as well. I'll test that next.

skwire · « **Reply #57 on:** December 18, 2007, 04:46 PM »

Test #1:

Code

skwire@hankoku /cygdrive/c/4
$ ls -al e*
-rwx------+ 1 skwire None 2037532760 Dec 18 07:51 email-1.zip
-rwx------+ 1 skwire None 2019501177 Dec 18 08:01 email-2.zip
-rwx------+ 1 skwire None 2010861617 Dec 18 08:07 email-3.zip
-rwx------+ 1 skwire None 2007484481 Dec 18 08:13 email-4.zip
-rwx------+ 1 skwire None 2003196991 Dec 18 08:19 email-5.zip
-rwx------+ 1 skwire None 2049079438 Dec 18 08:40 email-6.zip
-rwx------+ 1 skwire None 1058473649 Dec 18 08:48 email-7.zip

Notes:

1) 12GB+ directory with over half a million files in it.
2) Output mode #1, 2000 as the size setting.
3) It took about an hour for all seven zips to be created. This was using zero (0) for the compression setting (quickest mode).

Test #2:

Code

skwire@hankoku /cygdrive/c/4
$ ls -al i*
-rwx------+ 1 skwire None 2956007035 Dec 18 09:12 images-1.zip

Notes:

1) Source folder has 3593 files in it comprising 2.75 GB of space.
2) Output mode #1, 3000 as the size setting.
3) Generated file is a 2.75GB zip.

skwire · « **Reply #58 on:** December 18, 2007, 10:26 PM »

Here's the latest: http://www.autohotkey.net/~skwire/apps/spinzip/spinzip-setup-v0.0.6.0.exe

Give it a try and let me know if it works any better this time around. Also, would anybody be interested in encryption or password protection of the generated zip files?

testgames · « **Reply #59 on:** December 19, 2007, 09:27 AM »

wow !!! it works !!!

I've tried it and I got 3 zip files from a 5 GB directory tree.
Thanks a lot, for the quick responses, the clear answers and the willingness to help.
Cheers !

testgames · « **Reply #60 on:** December 19, 2007, 09:31 AM »

one last question:
can I use a command line (DOS) to do the same ?

if so, what should be the formant ?

backtomind · « **Reply #61 on:** December 20, 2007, 12:49 PM »

Hi,
it is possible and if so what is the way to get the source code of the SpinZip ?

sm00ter · « **Reply #62 on:** February 26, 2008, 02:42 PM »

skwire,

This app is awesome! I do think the password/encryption option would be nice!

Hey, while we are adding awesome features, how about cleaning up the original file/s once they are zipped?

For example: If I request each demo.log file to be zipped to demo.zip, once it is zipped - delete the original demo.log file!

Woohoo!

Thanks,

sm00ter

f0dder · « **Reply #63 on:** February 26, 2008, 03:55 PM »

Quote from: testgames on December 12, 2007, 03:54 PM

2. I am sending to this customers many games (hugh sized) which are more that 2 GB each.

Is it just me, or does this sound just a slight bit fishy?

J-Mac · « **Reply #64 on:** February 27, 2008, 06:45 AM »

Quote from: f0dder on February 26, 2008, 03:55 PM

Quote from: testgames on December 12, 2007, 03:54 PM
2. I am sending to this customers many games (hugh sized) which are more that 2 GB each.
Is it just me, or does this sound just a slight bit fishy?

Or signing up just to ask for source code?

skwire · « **Reply #65 on:** March 11, 2008, 04:28 AM »

Quote from: backtomind on December 20, 2007, 12:49 PM

Hi,
it is possible and if so what is the way to get the source code of the SpinZip ?

I'm not interested in making the source available at this time.

Quote from: sm00ter on February 26, 2008, 02:42 PM

skwire,

This app is awesome! I do think the password/encryption option would be nice!

Hey, while we are adding awesome features, how about cleaning up the original file/s once they are zipped?

For example: If I request each demo.log file to be zipped to demo.zip, once it is zipped - delete the original demo.log file!

Thanks for your kind words and apologies for the late reply. Are you still interested in these additional features?

skwire · « **Reply #66 on:** March 11, 2008, 04:29 AM »

Quote from: J-Mac on February 27, 2008, 06:45 AM

Quote from: f0dder on February 26, 2008, 03:55 PM
Quote from: testgames on December 12, 2007, 03:54 PM
2. I am sending to this customers many games (hugh sized) which are more that 2 GB each.
Is it just me, or does this sound just a slight bit fishy?

Or signing up just to ask for source code?

Agreed. I'm all for FOSS but it seems that some folks feel entitled to it these days.

ampedm · « **Reply #67 on:** March 19, 2008, 02:15 AM »

Quote from: skwire on March 11, 2008, 04:28 AM

Quote from: backtomind on December 20, 2007, 12:49 PM
Hi,
it is possible and if so what is the way to get the source code of the SpinZip ?

I'm not interested in making the source available at this time.

Quote from: sm00ter on February 26, 2008, 02:42 PM
skwire,

This app is awesome! I do think the password/encryption option would be nice!

Hey, while we are adding awesome features, how about cleaning up the original file/s once they are zipped?

For example: If I request each demo.log file to be zipped to demo.zip, once it is zipped - delete the original demo.log file!

Thanks for your kind words and apologies for the late reply. Are you still interested in these additional features?

I found your program while looking for ones on google. I think the password feature is the only feature (besides comments... but thats IZARCCs fault

) that this program needs to be more unique than winrar.

I have tried the following to put files in separate archives:

Winrar - need to pay,
PeaZip - uses more than one instance - a dirty trick
TugZip - No other feature beside simple archiving.

So i am sure everyone including me would love to see the next release have password support & if you got time, maybe a file browser to pick and choose files (CTRL + click thing) + other easy to implement features with the IZARCC tool...

- Eagerly waiting!!

skwire · « **Reply #68 on:** March 19, 2008, 03:57 AM »

Quote from: ampedm on March 19, 2008, 02:15 AM

I found your program while looking for ones on google. I think the password feature is the only feature (besides comments... but thats IZARCCs fault ) that this program needs to be more unique than winrar.

I have tried the following to put files in separate archives:

Winrar - need to pay,
PeaZip - uses more than one instance - a dirty trick
TugZip - No other feature beside simple archiving.

So i am sure everyone including me would love to see the next release have password support & if you got time, maybe a file browser to pick and choose files (CTRL + click thing) + other easy to implement features with the IZARCC tool...

- Eagerly waiting!!

I've already coded in password support but I'm hesitant to add file deletion support. I'd really hate to screw that up, you know? Though, on the other hand, if I delete to the recycle bin maybe it wouldn't be such a risk. What are your thoughts on this? The encryption idea will not be implemented since the IZarc command-line binary and the IZarc GUI version are the only things capable of opening those encrypted zips (I use the IZarc command-line binary as the backend for SpInZip).

As for a file browser, I think that goes beyond the scope of this tool. I've never intended for SpInZip to be anything close to a complete compression suite/app. Like I mentioned before, SpInZip is nothing more than a fancy front-end for the IZarc command-line binary. That being said, I do realise that Coding Snacks sometimes do take on a life of their own, hence, all the additional output options I've added to SpInZip. I trust you can understand that. =]

At any rate, here's v0.0.7.0 with the password support. Please let me know how it works out for you. Thanks.

http://www.autohotkey.net/~skwire/apps/spinzip/spinzip-setup-v0.0.7.0.exe

ampedm · « **Reply #69 on:** March 19, 2008, 04:15 AM »

Quote from: skwire on March 19, 2008, 03:57 AM

Quote from: ampedm on March 19, 2008, 02:15 AM
I found your program while looking for ones on google. I think the password feature is the only feature (besides comments... but thats IZARCCs fault ) that this program needs to be more unique than winrar.

I have tried the following to put files in separate archives:

Winrar - need to pay,
PeaZip - uses more than one instance - a dirty trick
TugZip - No other feature beside simple archiving.

So i am sure everyone including me would love to see the next release have password support & if you got time, maybe a file browser to pick and choose files (CTRL + click thing) + other easy to implement features with the IZARCC tool...

- Eagerly waiting!!

I've already coded in password support but I'm hesitant to add file deletion support. I'd really hate to screw that up, you know? Though, on the other hand, if I delete to the recycle bin maybe it wouldn't be such a risk. What are your thoughts on this? The encryption idea will not be implemented since the IZarc command-line binary and the IZarc GUI version are the only things capable of opening those encrypted zips (I use the IZarc command-line binary as the backend for SpInZip).

As for a file browser, I think that goes beyond the scope of this tool. I've never intended for SpInZip to be anything close to a complete compression suite/app. Like I mentioned before, SpInZip is nothing more than a fancy front-end for the IZarc command-line binary. That being said, I do realise that Coding Snacks sometimes do take on a life of their own, hence, all the additional output options I've added to SpInZip. I trust you can understand that. =]

At any rate, here's v0.0.7.0 with the password support. Please let me know how it works out for you. Thanks.

http://www.autohotkey.net/~skwire/apps/spinzip/spinzip-setup-v0.0.7.0.exe

i found an error:

when i try to zip @ level 9 filter *.chm a file named "textbook_of_international_health__2nd_ed_0195132041.chm"

or

"Electrodiagnosis in Diseases of Nerve and Muscle - Principles and Practice.chm"

extra files mysteriously appear in the archive:

skwire · « **Reply #70 on:** March 19, 2008, 04:33 AM »

Quote from: ampedm on March 19, 2008, 04:15 AM

i found an error:

when i try to zip @ level 9 filter *.chm a file named "textbook_of_international_health__2nd_ed_0195132041.chm"

or

"Electrodiagnosis in Diseases of Nerve and Muscle - Principles and Practice.chm"

extra files mysteriously appear in the archive:

Hmmm... Please provide the following information:

1) What input & output paths are you using?
2) Is the "Scan subdirectories too?" checkbox checked?
3) Which output mode are you using along with its settings.

Thanks.

ampedm · « **Reply #71 on:** March 19, 2008, 04:36 AM »

Quote from: skwire on March 19, 2008, 04:33 AM

Quote from: ampedm on March 19, 2008, 04:15 AM
i found an error:

when i try to zip @ level 9 filter *.chm a file named "textbook_of_international_health__2nd_ed_0195132041.chm"

or

"Electrodiagnosis in Diseases of Nerve and Muscle - Principles and Practice.chm"

extra files mysteriously appear in the archive:

Hmmm... Please provide the following information:

1) What input & output paths are you using?
2) Is the "Scan subdirectories too?" checkbox checked?
3) Which output mode are you using along with its settings.

Thanks.

E:\TODO-test, (copy 1)
No
Last output mode with extension

with and without password

skwire · « **Reply #72 on:** March 19, 2008, 05:15 AM »

Gah...silly bug on my part. Thanks for catching it.

Changelog:

Code

18 Mar 2008 - v0.0.8.0

    ! Fixed a pathing bug when using the "One file per zip" output mode.
      Thanks, ampedm.

12 Mar 2008 - v0.0.7.0

    + Added the ability to configure a password for generated zips.

New link here: http://www.autohotkey.net/~skwire/apps/spinzip/spinzip-setup-v0.0.8.0.exe

Apologies for the trouble and thanks!

mouser · « **Reply #73 on:** March 19, 2008, 06:20 AM »

i love seeing this teamwork

ampedm · « **Reply #74 on:** March 19, 2008, 09:43 AM »

Works amazing might i suggest new features now

- Making the priority of spinzip higher than izarcc and izarcc's lower so the spinzip window doesn't freeze.
- Cancel button of process.
- That's all i got...