Welcome Guest.   Make a donation to an author on the site July 26, 2014, 08:11:35 AM  *

Please login or register.
Or did you miss your validation email?


Login with username and password (forgot your password?)
Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.


You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.
 
Learn about the DonationCoder.com microdonation system (DonationCredits).
   
   Forum Home   Thread Marks Chat! Downloads Search Login Register  
Pages: [1] 2 3 Next   Go Down
  Reply  |  New Topic  |  Print  
Author Topic: DONE: Organize files into folders based on flename wildcard parsing  (Read 15621 times)
mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« on: March 03, 2010, 09:54:38 PM »

I am looking for a very specific organizational ability, namely this:

I have a huge directory (51GB and growing) of images (but I wouldn't want the solution to be image-file specific).

I need to parse the filenames in this directory using standard grep / wildcard rules and then create a series of file folders labeled as per the parsing rule and move all matching files into that subfolder. So for example, if I had a range of files called

BobsPhoto_ABC_0001.jpg    through    BobsPhoto_ABC_0251.jpg

and then similar files but for different names with varying numbers of files, thus

BobsPhoto_AAB_0001.jpg    through    BobsPhoto_AAB_0147.jpg
BobsPhoto_BFG_0011.jpg    through    BobsPhoto_BFG_0025.jpg
BobsPhoto_XMC_0009.jpg    through    BobsPhoto_XMC_4252.jpg
etc.

I would like all the ABC files in the .\ABC folder, all the AAB files in the .\AAB folder etc. or wherever I specified the folders should go.

In an absolutely perfect world there would be a csv file containing

'ABC','Smith, Mary'
'AAC','Brown, Joan"
etc.

and it would create the destination directories according to those rules (if the csv file is present, otherwise name the directories ABC, AAC etc)

If someone knows of a readily available solution (preferably freeware! Thmbsup) I'd love to hear about it. All my Googling has produced numerous solutions, none of which do quite what I want.
Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #1 on: March 03, 2010, 10:50:13 PM »

Are all the original files in one source folder?
Logged

mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #2 on: March 03, 2010, 10:55:42 PM »

Yes they are.
Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #3 on: March 03, 2010, 11:41:38 PM »

By chance, have you checked out Skrommel's MoveOut?

http://www.donationcoder....rommel/index.html#MoveOut

If that doesn't do it for you, I'll write you a custom app.
Logged

mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #4 on: March 04, 2010, 11:56:09 AM »

Hi skwire,

I checked out Moveout and it is indeed closer than much else I have found other than perhaps RoboBasket whose author says it can do the job. He said (In your case you can use the "Move to" action and check the "sort into subfolder" option , insert the "name.part2"  substring into the text box. ), but it certainly isn't intuitive as to why I need to do what he said (nowhere is defined what the name, name.part1, name.part2, name.part3, name.lastpart substrings refer to or even where or how one can define them). The software should (if it's designed properly) not need a manual for all but the intellectually challenged  undecided) or it isn't written well enough, IMHO.

MoveOut (as far as I can see) lacks the ability to create the necessary sub-folders and then move the files that meet the wildcard criteria into the self-named newly-created folders.

Did I miss something?

Mike
Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #5 on: March 04, 2010, 07:54:35 PM »

MoveOut (as far as I can see) lacks the ability to create the necessary sub-folders and then move the files that meet the wildcard criteria into the self-named newly-created folders.
Did I miss something?

Ah, apologies.  I didn't realise that MoveOut didn't create the destination folders.  In that case, would it be possible for you to PM me a simple text listing of your image filenames so I can test against them?  I don't need the full paths...just the names.  However, if there are privacy/sensitivity issues with this request, I certainly understand.  In that case, I'll just generate some random filenames based off the patterns in your initial post.  I've got the main framework of your app already finished;  I just need to do a more work on the flow and then some testing.
Logged

mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #6 on: March 04, 2010, 09:37:43 PM »

OK, I've tried to give you some sense of the file naming conventions which are reasonably strict as long as you appreciate that they will always start with a 2 or 3-letter or number group, then an underscore, then a 2 or 3 letter designator (typically but no hard and fast rule) then an underscore and some more letters and/or numbers and perhaps another underscore or hyphen and some more letters or numbers but this last part is not relevant to the project.

So what we need to create is the following structure:

root (could be anywhere)
then the first grouping creates a major subdir named after it (in the examples shown FJ or MA)
then subdirs off those based on the next grouping
then subdirs off that based on third grouping
then all the files that fit that pattern into those dirs so only three levels deep

Basically Photographer then Model then Project then Image Number

I hope you appreciate that I need an automatic process because (currently) there are 21,128 filers (and growing) so any needed manual intervention makes it a very laborious process indeed!

I forgot to ask you whether this process can be recursive because currently I have a mess of files in this monster dir and (more recently) some measure of subdirs off the monster dir named with project names and their contents need to be processed according to the overall rules, so you don't have to care what the dirs are named, just enter and process (I know, I'm not asking for much...)

I hope this makes it clear?

Mike



* skwire 3.jpg (7.54 KB, 151x136 - viewed 1289 times.)

* skwire 3a.jpg (7.32 KB, 141x138 - viewed 1286 times.)

* skwire 3b.jpg (8.16 KB, 138x133 - viewed 1287 times.)

* skwire 4.jpg (10.19 KB, 170x172 - viewed 1275 times.)
« Last Edit: March 04, 2010, 09:46:06 PM by mickblu » Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #7 on: March 05, 2010, 04:07:50 AM »

OK, I've tried to give you some sense of the file naming conventions which are reasonably strict as long as you appreciate that they will always start with a 2 or 3-letter or number group, then an underscore, then a 2 or 3 letter designator (typically but no hard and fast rule) then an underscore and some more letters and/or numbers and perhaps another underscore or hyphen and some more letters or numbers but this last part is not relevant to the project.

Well, therein lies the rub.  "Reasonably strict" doesn't equate to "strict" so this will be very difficult to do automatically.  However, read on.

So what we need to create is the following structure:
root (could be anywhere)
then the first grouping creates a major subdir named after it (in the examples shown FJ or MA)
then subdirs off those based on the next grouping
then subdirs off that based on third grouping
then all the files that fit that pattern into those dirs so only three levels deep

At its most basic, what you describe above is very easy to do.  Cracking a filename on its underscores, creating folders, and moving said file into said folder is a piece of cake.  However, since some of your files have three groups and others have four, this is much more difficult to do accurately.  Do you understand what I mean?

For instance, your MA_* files mostly have four groups (three that matter and then the irrelevant counter portion) but your FJ_* files mostly have three groups (two that matter and then the irrelevant counter portion).  Normally, I could take num_groups-1 and go with that but I can't, in this case, since there are example in your images where the MA_* files have three groups and the FJ_* files have four groups.

At this point, I don't feel comfortable writing a parser that goes three levels deep since there is no hard standard to the filenames.  There are a few ways to approach this:

1) You strictly standarise your filenames at which point I can make the parser as creative as you would like.

2) You divvy up your files into groups that contains the same amount of "underscore groups" and then I can offer an option to crack the filenames on first, second, or third groups.

3) I write the app so you can form basic *? type of wildcard matching.  Something like "MA_*_*.jpg" gets filtered to some destination folder you specify.  One drawback here is that each match entry can only have one destination folder.  Another drawback, I suppose, is that the entry will need to be entered manually.  On the other hand, the list of entries is easily saved and reused.  This is how I started writing this app before you mentioned your automatic-filename-cracking request.

I hope you appreciate that I need an automatic process because (currently) there are 21,128 filers (and growing) so any needed manual intervention makes it a very laborious process indeed!

Sure, I appreciate it but, from my point of the view as the coder who is messing with your files, I have to feel confident that I'm not going to muck them up.

I forgot to ask you whether this process can be recursive because currently

Yes, recursive processing is easily doable.

Logged

mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #8 on: March 05, 2010, 04:18:09 PM »

Hi Skwire,

I appreciate your dilemma, and I think your solution #3 is the best where you allow me to put in the search criteria in the form of a wild-card string. That wouldn't bother me at all as there are really only the two naming conventions (3 group and 4 group) but at least that way any other eventuality is easily handled.

I look forward to hearing from you,

Mike
« Last Edit: March 08, 2010, 11:53:41 AM by mickblu » Logged
mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #9 on: March 11, 2010, 12:18:06 PM »

Hi Skwire,

I was just wondering where we were at with the "project" as it's been over a week and I hadn't heard back from you.

Sincerely,

Mike
Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #10 on: March 11, 2010, 04:37:11 PM »

Doh...I'll work on this tonight.  Sorry about that.   embarassed
Logged

mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #11 on: March 11, 2010, 04:46:37 PM »

Hey, no worries, I was just curious is all...  embarassed
Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #12 on: March 11, 2010, 04:55:49 PM »

BTW, have you checked out File Sieve 3 at all?  The author is a great coder...I use a few of his apps.  Comical is a favourite.  Anyway, if File Sieve doesn't do what you need, I'll continue work on your custom app.  Please let me know.
« Last Edit: March 11, 2010, 05:12:34 PM by skwire » Logged

mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #13 on: March 11, 2010, 05:31:56 PM »

Nope, haven't come across that one before, and I searched the heck out of the web at the time, but I guess I just didn't hit the right search phrase...

Certainly looks interesting, and probably can do the job, but because of its sophistication it will take some playing with to ascertain whether it will or won't be able to do the sort. I'll play with it a bit and let you know... What really appeals to me is that it supports multiple input directories.

Thanks for the heads-up!
Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #14 on: March 11, 2010, 05:48:32 PM »

For what it's worth, my custom app for you also has multiple input directories.  Here's a preliminary screenshot:

Logged

mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #15 on: March 11, 2010, 05:50:03 PM »

Wow! All I can say so far is that it looks about a million times more intuitive than FileSieve!

Out of curiosity, what language are you using to write it?
« Last Edit: March 11, 2010, 05:59:19 PM by mickblu » Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #16 on: March 11, 2010, 06:30:37 PM »

If FileSieve doesn't end up working out for you, let me know.  All my applications are written in AutoHotkey.  You can find more examples of my work at my website in my sig.
Logged

mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #17 on: March 11, 2010, 07:26:35 PM »

Now you've got me interested in AutoHotKey! It seems like it would, with some learning, enable me to automate a lot of the repetitive things I do.

I don't know how old you are so I don't know how much of what I am about to say will make any sense but, here goes:

I studied programming waaaay back in the 70's (BASIC, COBOL and FORTRAN on an NCR mini coding onto 80-column  punch-cards!) which, while they are certainly not current languages, should give me some structural guidelines. I've also coded in dBase IV and similar (Foxbase etc.) and in  VBasic. In fact, I ran the very first BBS here in town on a 300 baud dialup modem using a Z80 computer running the entire system in interpreted BASIC!

If you were to make up this app for me, would you provide it in AHK source or is that too much to hope for? I'd love to be able to learn from a master!

I've been fiddling with FileSieve and it is a lot less intuitive than I would have hoped, and it's not helped by the almost complete lack of documentation other than a few simple examples in a .chm file.

What would it take to convince you to complete the project for me  Kiss ?
Logged
mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #18 on: March 11, 2010, 08:23:25 PM »

Hi Skwire,

It seems that FileSieve crashes when faced with the 293,717 files  that I have (and I only pointed it to the "fj" subset, "ma" has another 241,098) so, I guess, your mission, should you choose to accept it sir, is to demonstrate how AHK can do the job!

BTW, FWIW in FileSieve's documentation it says: In this version, it is not currently possible for you to create your own method. It may be possible in a later version which will either provide scripting or plugin support to provide you with this functionality.
« Last Edit: March 11, 2010, 08:28:32 PM by mickblu » Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #19 on: March 11, 2010, 09:36:31 PM »

Now you've got me interested in AutoHotKey! It seems like it would, with some learning, enable me to automate a lot of the repetitive things I do.

Oh, my.  I would hate to use a computer without it.  I've built my main "system script" up over the years to the point where vanilla Windows functionality is tough to deal with.

I don't know how old you are

I'm 36.  Not too old...not too young.   tongue

If you were to make up this app for me, would you provide it in AHK source or is that too much to hope for? I'd love to be able to learn from a master!

I'm no master...but you're welcome to the source on this one.

What would it take to convince you to complete the project for me  Kiss ?

I'll work on it tonight.  There really isn't much left to do other than some GUI bits and testing.

It seems that FileSieve crashes when faced with the 293,717 files  that I have (and I only pointed it to the "fj" subset, "ma" has another 241,098) so, I guess, your mission, should you choose to accept it sir, is to demonstrate how AHK can do the job!

I've no doubt AHK can do it.   smiley
Logged

mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #20 on: March 11, 2010, 10:07:59 PM »

That will be awesome! Thank you.

Heh, 36! My daughter is 34 and my son is 32 so that gives you a clue how old I am (I'm 62   !)

Anyway, I'm really looking forward to seeing what AHK can do...
« Last Edit: March 11, 2010, 10:09:59 PM by mickblu » Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #21 on: March 11, 2010, 10:19:14 PM »

To make my testing easier, can you provide me with a text file containing the filenames you're dealing with?
« Last Edit: March 11, 2010, 10:33:51 PM by skwire » Logged

mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #22 on: March 11, 2010, 10:20:10 PM »

Sure, gimme 5...
Logged
mickblu
Supporting Member
**
Posts: 14


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #23 on: March 11, 2010, 11:47:52 PM »

I guess it took somewhat longer than 5  embarassed but I had to find the right utility to produce a recursive directory list...

Anyway, here it is!

* skwire2.txt (4170.49 KB - downloaded 168 times.)
Logged
skwire
Moderator
*****
Posts: 4,016



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #24 on: March 11, 2010, 11:49:43 PM »

Nice, thanks.  I've sent you a PM as well.
Logged

Pages: [1] 2 3 Next   Go Up
  Reply  |  New Topic  |  Print  
 
Jump to:  
   Forum Home   Thread Marks Chat! Downloads Search Login Register  

DonationCoder.com | About Us
DonationCoder.com Forum | Powered by SMF
[ Page time: 0.041s | Server load: 0.08 ]