topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Friday March 29, 2024, 5:57 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: Compressing files into multiple .zip archives with each .zip being independent  (Read 140929 times)

testgames

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 10
    • View Profile
    • Donate to Member
Hi,

1. we are talking about around a few hundreds of sub-directories, which contain many small .dat files.
2. There are about 11 levels. every level contains around 4-5 levels so the hole tree is about 400,000 different directories (an encoded game). The hole tree is about 7 GB.
3. I don't really understand what do you mean by output mode, but I guess standard.


skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
1. we are talking about around a few hundreds of sub-directories, which contain many small .dat files.
2. There are about 11 levels. every level contains around 4-5 levels so the hole tree is about 400,000 different directories (an encoded game). The hole tree is about 7 GB.

Since it's such a huge directory that you're working on, is there any particular reason you want to use SpInZip over any of the much more complete and capable compression apps out there (honest question...I'm not trying to be a smartass)?

3. I don't really understand what do you mean by output mode, but I guess standard.

Sorry, by output mode, I mean which of the three output modes that SpInZip offers were you attempting to use (and the settings you were attempting to use with that output mode)?  If the aforementioned 2GB zip limit is a problem, I can easily add support for .7z files.  However, you will need 7-zip (free @ http://www.7-zip.org) to unpack the resultant files.

testgames

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 10
    • View Profile
    • Donate to Member
here is the story:
1. My customer has an old version of Winzip. He can not unzip files more than 2 GB size.
2. I am sending to this customers many games (hugh sized) which are more that 2 GB each.
3. what we did until now is to manually close zip files up to 2 GB and send them seperatly (so he can unzip them one by one). for example: a 7 GB game = 4 separate zip files.
4. He does not have tar or rar or any other unzip mechanizm nor he will not upgrade his winzip version to support more than 2 GB files.
5. I need the zip files to be stand alone - so I can open them by themsleves and not X.zip.001, X.zip.002 which comines together are a part of a bigger zip.
got it ?

testgames

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 10
    • View Profile
    • Donate to Member
1. I used the 1st option for Output mode: "By Approzimate Zip File Size" = 2048 MB
2. I don't want to use the 7-zip to unpack the file (the customer will not install it at his network) so I need it to be open only by unzip.


skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
4. He does not have tar or rar or any other unzip mechanizm nor he will not upgrade his winzip version to support more than 2 GB files.
5. I need the zip files to be stand alone - so I can open them by themsleves and not X.zip.001, X.zip.002 which comines together are a part of a bigger zip.

Since he won't upgrade, have you tried making a self-extracting spanned zip file that you could send him?

1. I used the 1st option for Output mode: "By Approzimate Zip File Size" = 2048 MB
2. I don't want to use the 7-zip to unpack the file (the customer will not install it at his network) so I need it to be open only by unzip.

The closest directory I have to yours would be my mail server folder.  It currently has 581,395 files in it comprising 12.1GB of space.  I'll try testing against it and see what happens.

testgames

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 10
    • View Profile
    • Donate to Member
I can make self extracting spanned zip files (at 2 GB size) but the problem is I have around 400 of these to make and it will take me too much time, therefore I've been looking for a tool to make it.

Testing it on the mail server can be a great test - thanks a lot, looking forward for the results.


skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
I can make self extracting spanned zip files (at 2 GB size) but the problem is I have around 400 of these to make and it will take me too much time, therefore I've been looking for a tool to make it.

Gotcha.

Testing it on the mail server can be a great test - thanks a lot, looking forward for the results.

I've made some changes and they look promising.  I'm currently running a test on my mail server directory using similar settings to what you were trying (Option 1, 2000MB).  It seems to be working fine.  The reason I'm not using 2048MB is that, as I mentioned in a previous post, I am not doing any sort of magical algorithm that attempts to make perfects-sized zip files.   I simply recurse through the input folder and keep a running total of the file sizes.  As soon as that total goes OVER your specified size, I zip those files.  Since you're attempting to push the upper limit of a normal zip file, I'd suggest you use something lower than 2048MB since you run the risk of exceeding that amount.  That why I put "approximate" on the app's interface.  Does that make sense?

In addition, I should be able to support zips greater than 2GB as well.  I'll test that next.

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
Test #1:

skwire@hankoku /cygdrive/c/4
$ ls -al e*
-rwx------+ 1 skwire None 2037532760 Dec 18 07:51 email-1.zip
-rwx------+ 1 skwire None 2019501177 Dec 18 08:01 email-2.zip
-rwx------+ 1 skwire None 2010861617 Dec 18 08:07 email-3.zip
-rwx------+ 1 skwire None 2007484481 Dec 18 08:13 email-4.zip
-rwx------+ 1 skwire None 2003196991 Dec 18 08:19 email-5.zip
-rwx------+ 1 skwire None 2049079438 Dec 18 08:40 email-6.zip
-rwx------+ 1 skwire None 1058473649 Dec 18 08:48 email-7.zip

Notes:

1) 12GB+ directory with over half a million files in it.
2) Output mode #1, 2000 as the size setting.
3) It took about an hour for all seven zips to be created.  This was using zero (0) for the compression setting (quickest mode).


Test #2:

skwire@hankoku /cygdrive/c/4
$ ls -al i*
-rwx------+ 1 skwire None 2956007035 Dec 18 09:12 images-1.zip

Notes:

1) Source folder has 3593 files in it comprising 2.75 GB of space.
2) Output mode #1, 3000 as the size setting.
3) Generated file is a 2.75GB zip.
 

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
Here's the latest: http://www.autohotke...p-setup-v0.0.6.0.exe

Give it a try and let me know if it works any better this time around.  Also, would anybody be interested in encryption or password protection of the generated zip files?

testgames

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 10
    • View Profile
    • Donate to Member
wow !!! it works !!!

I've tried it and I got 3 zip files from a 5 GB directory tree.
Thanks a lot, for the quick responses, the clear answers and the willingness to help.
Cheers !

testgames

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 10
    • View Profile
    • Donate to Member
one last question:
can I use a command line (DOS) to do the same ?

if so, what should be the formant ?

backtomind

  • Participant
  • Joined in 2007
  • *
  • default avatar
  • Posts: 1
    • View Profile
    • Donate to Member
Hi,
it is possible and if so what is the way to get the source code of the SpinZip ?

sm00ter

  • Participant
  • Joined in 2008
  • *
  • default avatar
  • Posts: 1
    • View Profile
    • Donate to Member
skwire,

This app is awesome!  I do think the password/encryption option would be nice!

Hey, while we are adding awesome features, how about cleaning up the original file/s once they are zipped?

For example:  If I request each demo.log file to be zipped to demo.zip, once it is zipped - delete the original demo.log file!

Woohoo!

Thanks,

sm00ter

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
2. I am sending to this customers many games (hugh sized) which are more that 2 GB each.
Is it just me, or does this sound just a slight bit fishy?
- carpe noctem

J-Mac

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 2,918
    • View Profile
    • Donate to Member
2. I am sending to this customers many games (hugh sized) which are more that 2 GB each.
Is it just me, or does this sound just a slight bit fishy?

Or signing up just to ask for source code?

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
Hi,
it is possible and if so what is the way to get the source code of the SpinZip ?

I'm not interested in making the source available at this time.

skwire,

This app is awesome!  I do think the password/encryption option would be nice!

Hey, while we are adding awesome features, how about cleaning up the original file/s once they are zipped?

For example:  If I request each demo.log file to be zipped to demo.zip, once it is zipped - delete the original demo.log file!

Thanks for your kind words and apologies for the late reply.  Are you still interested in these additional features?

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
2. I am sending to this customers many games (hugh sized) which are more that 2 GB each.
Is it just me, or does this sound just a slight bit fishy?

Or signing up just to ask for source code?

Agreed.  I'm all for FOSS but it seems that some folks feel entitled to it these days.

ampedm

  • Participant
  • Joined in 2008
  • *
  • default avatar
  • Posts: 5
    • View Profile
    • Donate to Member
Hi,
it is possible and if so what is the way to get the source code of the SpinZip ?

I'm not interested in making the source available at this time.

skwire,

This app is awesome!  I do think the password/encryption option would be nice!

Hey, while we are adding awesome features, how about cleaning up the original file/s once they are zipped?

For example:  If I request each demo.log file to be zipped to demo.zip, once it is zipped - delete the original demo.log file!

Thanks for your kind words and apologies for the late reply.  Are you still interested in these additional features?

I found your program while looking for ones on google. I think the password feature is the only feature (besides comments... but thats IZARCCs fault :) ) that this program needs to be more unique than winrar.

I have tried the following to put files in separate archives:

Winrar - need to pay,
PeaZip - uses more than one instance - a dirty trick
TugZip - No other feature beside simple archiving.

So i am sure everyone including me would love to see the next release have password support & if you got time, maybe a file browser to pick and choose files (CTRL + click thing)  + other easy to implement features with the IZARCC tool...

- Eagerly waiting!!
« Last Edit: March 18, 2008, 09:17 PM by ampedm »

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
I found your program while looking for ones on google. I think the password feature is the only feature (besides comments... but thats IZARCCs fault :) ) that this program needs to be more unique than winrar.

I have tried the following to put files in separate archives:

Winrar - need to pay,
PeaZip - uses more than one instance - a dirty trick
TugZip - No other feature beside simple archiving.

So i am sure everyone including me would love to see the next release have password support & if you got time, maybe a file browser to pick and choose files (CTRL + click thing)  + other easy to implement features with the IZARCC tool...

- Eagerly waiting!!

I've already coded in password support but I'm hesitant to add file deletion support.  I'd really hate to screw that up, you know?  Though, on the other hand, if I delete to the recycle bin maybe it wouldn't be such a risk.  What are your thoughts on this?  The encryption idea will not be implemented since the IZarc command-line binary and the IZarc GUI version are the only things capable of opening those encrypted zips (I use the IZarc command-line binary as the backend for SpInZip). 

As for a file browser, I think that goes beyond the scope of this tool.  I've never intended for SpInZip to be anything close to a complete compression suite/app.  Like I mentioned before, SpInZip is nothing more than a fancy front-end for the IZarc command-line binary.  That being said, I do realise that Coding Snacks sometimes do take on a life of their own, hence, all the additional output options I've added to SpInZip.  I trust you can understand that.  =]

At any rate, here's v0.0.7.0 with the password support.  Please let me know how it works out for you.  Thanks.

http://www.autohotke...p-setup-v0.0.7.0.exe

ampedm

  • Participant
  • Joined in 2008
  • *
  • default avatar
  • Posts: 5
    • View Profile
    • Donate to Member
I found your program while looking for ones on google. I think the password feature is the only feature (besides comments... but thats IZARCCs fault :) ) that this program needs to be more unique than winrar.

I have tried the following to put files in separate archives:

Winrar - need to pay,
PeaZip - uses more than one instance - a dirty trick
TugZip - No other feature beside simple archiving.

So i am sure everyone including me would love to see the next release have password support & if you got time, maybe a file browser to pick and choose files (CTRL + click thing)  + other easy to implement features with the IZARCC tool...

- Eagerly waiting!!

I've already coded in password support but I'm hesitant to add file deletion support.  I'd really hate to screw that up, you know?  Though, on the other hand, if I delete to the recycle bin maybe it wouldn't be such a risk.  What are your thoughts on this?  The encryption idea will not be implemented since the IZarc command-line binary and the IZarc GUI version are the only things capable of opening those encrypted zips (I use the IZarc command-line binary as the backend for SpInZip). 

As for a file browser, I think that goes beyond the scope of this tool.  I've never intended for SpInZip to be anything close to a complete compression suite/app.  Like I mentioned before, SpInZip is nothing more than a fancy front-end for the IZarc command-line binary.  That being said, I do realise that Coding Snacks sometimes do take on a life of their own, hence, all the additional output options I've added to SpInZip.  I trust you can understand that.  =]

At any rate, here's v0.0.7.0 with the password support.  Please let me know how it works out for you.  Thanks.

http://www.autohotke...p-setup-v0.0.7.0.exe

i found an error:

when i try to zip @ level 9 filter *.chm a file named "textbook_of_international_health__2nd_ed_0195132041.chm"

or

"Electrodiagnosis in Diseases of Nerve and Muscle - Principles and Practice.chm"

extra files mysteriously appear in the archive:


skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
i found an error:

when i try to zip @ level 9 filter *.chm a file named "textbook_of_international_health__2nd_ed_0195132041.chm"

or

"Electrodiagnosis in Diseases of Nerve and Muscle - Principles and Practice.chm"

extra files mysteriously appear in the archive:

Hmmm...  Please provide the following information:

1) What input & output paths are you using?
2) Is the "Scan subdirectories too?" checkbox checked?
3) Which output mode are you using along with its settings.

Thanks.

ampedm

  • Participant
  • Joined in 2008
  • *
  • default avatar
  • Posts: 5
    • View Profile
    • Donate to Member
i found an error:

when i try to zip @ level 9 filter *.chm a file named "textbook_of_international_health__2nd_ed_0195132041.chm"

or

"Electrodiagnosis in Diseases of Nerve and Muscle - Principles and Practice.chm"

extra files mysteriously appear in the archive:

Hmmm...  Please provide the following information:

1) What input & output paths are you using?
2) Is the "Scan subdirectories too?" checkbox checked?
3) Which output mode are you using along with its settings.

Thanks.

E:\TODO-test, (copy 1)
No
Last output mode with extension

with and without password

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
Gah...silly bug on my part.  Thanks for catching it.

Changelog:

18 Mar 2008 - v0.0.8.0

    ! Fixed a pathing bug when using the "One file per zip" output mode.
      Thanks, ampedm.

12 Mar 2008 - v0.0.7.0

    + Added the ability to configure a password for generated zips.

New link here:  http://www.autohotke...p-setup-v0.0.8.0.exe

Apologies for the trouble and thanks!
« Last Edit: March 19, 2008, 12:19 AM by skwire »

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
i love seeing this teamwork  :up: :up: :up:

ampedm

  • Participant
  • Joined in 2008
  • *
  • default avatar
  • Posts: 5
    • View Profile
    • Donate to Member
Works amazing might i suggest new features now :)

- Making the priority of spinzip higher than izarcc and izarcc's lower so the spinzip window doesn't freeze.
- Cancel button of process.
- That's all i got...