Author Topic: Separate out and move files to subfolders (Read 29011 times)

nkormanik · « **on:** November 05, 2013, 08:23 PM »

Suppose we have a large collection of images (.jpg) in one single folder -- 120,000 of 'em.

We would like to move these image files to separate subfolders, 1000 per subfolder.

I presently have a way of doing this, but it's fairly tedious.

Wondering if someone might like to write a program to to it automatically.

Here's how I'm imagining it might work:

Fire up program. Several questions are asked:

-- Where are the files you want to separate out?

(possible answer: c:\images)

-- Where do you want to move these files to?

(possible answer: c:\separated)

-- How many files per subfolder?

(possible answer: 1000)

-- In what order are the files to be moved?

(possible answer: smallest to largest)

-- Press proceed when ready....

After program runs, there will be, say, 120 new folders in c:\separated:

c:\separated\001
c:\separated\002
c:\separated\003
.
.
.
c:\separated\120

Each of these subfolders will contain 1000 files. From smallest (in subfolder 001) to largest (subfolder 120).

What do you think? If there exists a way already out there, please let me know. I'd greatly appreciate your help.

Thanks,
Nicholas Kormanik

c.gingerich · « **Reply #1 on:** November 06, 2013, 01:14 PM »

I do not know of any other application but seems easy enough to do. I'll see about getting an app together. :-)

TaoPhoenix · « **Reply #2 on:** November 06, 2013, 03:09 PM »

That looks pretty neat for an idea!

skwire · « **Reply #3 on:** November 06, 2013, 03:36 PM »

@Chris: If you don't have time for this one, I can do it as well.

c.gingerich · « **Reply #4 on:** November 06, 2013, 04:02 PM »

@skwire: Ok I did start working on it. If I cannot finish I'll let you know.

c.gingerich · « **Reply #5 on:** November 07, 2013, 11:01 AM »

It is working well. I just need to tweak a few things and I should have it ready for download either later today or tomorrow morning. :-)

Screen Shot

http://i.imm.io/1k4MR.png

c.gingerich · « **Reply #6 on:** November 07, 2013, 01:20 PM »

It's ready to try out!

http://cgs.vdsworld.com/click.php?id=64

Easy enough to use. Just run File Groupie.exe, select Source folder, Target Folder, Number of files per sub-folder, Command (Move or Copy), Sort By and press Run!

Also made it my NANY submission - https://www.donation...opic=36494.msg341524

Screen shot: http://i.imm.io/1k9Pg.png

tomos · « **Reply #7 on:** November 07, 2013, 02:33 PM »

I dont even back up my photos to DVD any more but I still sort my photos in groups that *will* fit on a DVD (single-layer, say 4.2GB). That way I have the option. It's also simply a convenient breakdown size to use.

Would you consider adding MB (or GB) per sub-folder as an alternative option to the "Files per sub-folder" option?

TBH, I'm not sure how practical it would be to have so many photos in one folder. But the user could work around this and create folders with e.g. 1GB content. I average around 3 or 4 subfolders in a DVD size folder. In my case they're randomly based on the content of the camera's card when transferred.

Another "TBH" - I'm not sure would I use this, but I still think my suggestion would be a good extenstion of it.

c.gingerich · « **Reply #8 on:** November 07, 2013, 02:55 PM »

I'll add it to the list :-)

c.gingerich · « **Reply #9 on:** November 08, 2013, 11:17 AM »

Update to 1.0.1.0 - Now allows prefix/suffix of files (renamed) and folder.

c.gingerich · « **Reply #10 on:** November 10, 2013, 11:26 AM »

Anyone test it out? @nkormanik Does it work like you wanted?

skwire · « **Reply #11 on:** November 10, 2013, 12:52 PM »

I just finished giving this a fairly thorough testing. Here's my setup:

[Select]

OS: Windows 7 Ultimate (Version 6.1 Build 7601) Service Pack 1
CPU: 4 x Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz
MEM: 16 GB
Source folder: c:\tmp5
Target folder: c:\tmp6

Using zero-byte files named File_#.txt where # is 1-to-however-many test files I generated for a given test.

Test #1:

120,000 test files in c:\tmp5
Files per sub-folder: 1000
Command: Copy

After clicking Run, the statusbar updated to "Processing 0 of 120000" and the CPU usage immediately went to 100% for one of the CPU cores.
I left it there for a few minutes but nothing was created in the target folder.
Killed the process, tried again with a files-per-subfolder of 10000 with the same results.
Tried the various "Sort files by" methods but no changes in behaviour.

Test #2:

10,000 test files in c:\tmp5
Files per sub-folder: 10
Command: Copy

After clicking Run, the statusbar updated to "Processing 0 of 10000" and the CPU usage immediately went to 100% for one of the CPU cores.
File groups began to be created very slowly.
When I say slowly, I mean slowly. After fifteen minutes, the program had only created roughly 300 of the eventual 1000 folders for this test.
Killed the process as clicking the close button didn't work while the app was in its file processing routine.

Test #3:

1,000 test files in c:\tmp5
Files per sub-folder: 100
Command: Copy

File groups were created successfully.

Test #4:

5,000 test files in c:\tmp5
Files per sub-folder: 100
Command: Copy

After clicking Run, the statusbar updated to "Processing 0 of 10000" and the CPU usage immediately went to 100% for one of the CPU cores.
File groups were created successfully. Again, though...much more slowly than I would expect (as a programmer).

I hope this helps some.

c.gingerich · « **Reply #12 on:** November 10, 2013, 01:19 PM »

Wow. That was unexpected... Ok I'll take a look. Thanks for the testing!

TaoPhoenix · « **Reply #13 on:** November 10, 2013, 09:07 PM »

Wow. That was unexpected... ... Thanks for the testing!
-c.gingerich (November 10, 2013, 01:19 PM)

Heh since my tests often produce odd results lemme try a couple of tests two, from a diff angle! More news in a bit!
(Hi Skwire!)

c.gingerich · « **Reply #14 on:** November 10, 2013, 09:40 PM »

Ok sounds good. I think I found what the problem is but waiting to hear what you find.

c.gingerich · « **Reply #15 on:** November 11, 2013, 09:55 AM »

Updated to 1.0.2.0

Fixed bug where it would not process files over 999 very well or at all. Added Abort button. All controls are disabled when processing (except Abort).

Please give this a try. http://cgs.vdsworld.com/click.php?id=64

Thanks

TaoPhoenix · « **Reply #16 on:** November 11, 2013, 11:03 AM »

Heh hmm.

Well, it doesn't work at all for me!

For example:
I extracted it all to a folder on my desktop, same name "File Groupie". So it has a bunch of stuff. That Readme opens in a strange app that wasn't easy to print from. So I copied it and pasted it into a text file.

Then I hit ControlC-ControlV a bunch of times and got stuff like "Copy2 of Readme" "Copy3 of Readme".

I made a folder called "Test MoveTo Folder".

I ran Groupie. Selected "Sort by name". Prefix "Copy". Picked the folders.

Then I ran it.

Not unlike Skwire's, it locked up my entire comp (quad core, but one thread? "CPU Use" went to 100...), and stayed there until I killed the process in Task Manager. Abort didn't work!

Now I know my machine is a little feisty but it generally does everything else, and it's only these kinds of shell-esque AHK tools that really tank!

So there's my use case!

Current Status for me: 100% Fail. Park Until Advised.

Edit: My test is already in the 1.0.2.0 version.

TaoPhoenix · « **Reply #17 on:** November 11, 2013, 11:04 AM »

Updated to 1.0.2.0

Fixed bug where it would not process files over 999 very well or at all. Added Abort button. All controls are disabled when processing (except Abort).

Please give this a try. http://cgs.vdsworld.com/click.php?id=64

Thanks
-c.gingerich (November 11, 2013, 09:55 AM)

My test case only had four files!

My note crossed yours. See what you think!

c.gingerich · « **Reply #18 on:** November 11, 2013, 11:54 AM »

Now give it a try. Version 1.0.2.1

http://cgs.vdsworld.com/click.php?id=64

skwire · « **Reply #19 on:** November 11, 2013, 02:29 PM »

Now give it a try. Version 1.0.2.1
-c.gingerich (November 11, 2013, 11:54 AM)

A quick 10000/100 test worked much better. Very quick...finished in a matter of seconds.

However, the 120000/1000 test did not go so well:

It took 4 minutes to get through the sorting phase. It got slower as the statusbar count went up.
After that, it took 4.5 minutes to start creating the folder groups.
Folder group creation took ~4 minutes to complete.
Oddly, the folder names were created in tens, e.g., 0010, 0020, 0030, etc.

c.gingerich · « **Reply #20 on:** November 11, 2013, 02:37 PM »

Ok. I'll try to recreate that and get it fixed asap. Thanks!

magician62 · « **Reply #21 on:** November 11, 2013, 04:07 PM »

Ok , I will give you some perspective as to how your program should work.

Just using explorer I created a folder with 20000 jpg files totalling 4.56GB

It took 4 minutes to copy to another directory on the same drive 4412 files with a total of 1.13GB

So extrapolating, 20000 files will take around 20 minutes

So 100,000 files will take 100 minutes

And this is before you try and do something clever with the files.

The transfer rate was approximately 4.7MB/s

If you were moving the files rather than copying it would probably be far quicker, especially to a different drive, and preferably clean, as these small files just like filling up any fragmentation in a drive

nkormanik · « **Reply #22 on:** November 12, 2013, 03:01 AM »

Terribly sorry for not responding sooner. For some reason I wasn't notified that anyone did anything on this project request -- must have forgotten to check "Notify me of replies." Just assumed no one was interested, or that it was too hard to accomplish. Wow. Lots of action going on here! How exciting.

Test Results: "File Groupie" worked flawlessly. Hip hip hooray!!!

Thank you very much Chris Gingerich, and all of you that tested and helped bring this useful program about.

My Test:

File Groupie version 1.0.2.1

64,709 files totalling 55.1 GB, all in one single folder

Moving files to Target Folder: c:\FGed

Files per sub-folder: 1000

Sort files by: Size - Smallest to Largest

Left the optional settings blank.

Hit "Run"....

Reading the files took 8 minutes.

Sorting took 26 minutes.

Processing took 5 minutes.

Output Results:
(note that file sort order worked! size-low early on, to size-high later)

Folder File Bytes Files
0 49,522,465 1,000
100 77,547,802 1,000
200 96,510,142 1,000
300 113,185,581 1,000
400 127,619,495 1,000
500 141,009,494 1,000
600 153,392,198 1,000
700 166,393,938 1,000
800 179,206,949 1,000
.
.
.
5800 1,903,439,374 1,000
5900 2,092,646,683 1,000
6000 2,355,084,664 1,000
6100 2,740,121,189 1,000
6200 3,370,157,067 1,000
6300 4,358,001,053 1,000
6400 5,036,581,013 709

64,709 Files

c.gingerich · « **Reply #23 on:** November 12, 2013, 07:07 AM »

Great glad to hear!

you are very welcome.

TaoPhoenix · « **Reply #24 on:** November 12, 2013, 09:11 AM »

Hmm.

Well the good news is it didn't hang!

The "Ugly news" is that you said you wanted it to be NANY and not just a "snack"... that's my famous topic I keep saying about "Coding Snacks vs Coding Lunches"! NANY's are Coding Lunches ... or maybe Cereal?

Skwire, can you weigh in on the differences between this and Files2Folders? They're suddenly sounding similar, except this time Chris's version "did what it does" and I think I recall yours making my system grumpy, so I don't know why because it feels like the same "mechanical end result". So maybe feature differences, and maybe a quick summary of the "backbone".

So to continue the "Ugly News", how many use cases do you want vs "easy for a user to walk into"? Since "no one reads the Readme's anyway", I thought "Optional Setting - File - Prefix" would only move the files *with that prefix* ... imagine my shock "Holy @#$@#$@, it blew up my data??!" So for example a dialog box indicating that data could be totally unrecoverable if you had more than my trivial case carefully file named, like 100 songs with descriptive names, only to have them all become "song1 song2 song3"!

If I ... uh ... hadn't used bvckup like last week, and did that to my music folder...

Another example pops up from what would have been my use case at work. I used to do Change Orders to contracts in construction accounting. So here are my juicy 85 change orders from my hard working manager, all sensibly named "ChangeOrder1 ChangeOrder2"... and off I go... out comes folders 0-84 ...

(the manager grabbed #s 45-84 for the lawyer, who signed the contract... with addendums 44-83!!) (Gets fired)

So this feels like a Bill Murray in Groundhog Day making ice sculptures with a chainsaw! Very powerful, but if you slip on that ice, you might lose a hand!

I think somehow Skwire's mechanic of selecting relevant files to be processed feels a bit safer, so maybe your current UI might be some kind of unlockable "advanced power mode".