topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Friday December 13, 2024, 10:06 pm
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: DONE: Organize files into folders based on flename wildcard parsing  (Read 105791 times)

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
I am looking for a very specific organizational ability, namely this:

I have a huge directory (51GB and growing) of images (but I wouldn't want the solution to be image-file specific).

I need to parse the filenames in this directory using standard grep / wildcard rules and then create a series of file folders labeled as per the parsing rule and move all matching files into that subfolder. So for example, if I had a range of files called

BobsPhoto_ABC_0001.jpg    through    BobsPhoto_ABC_0251.jpg

and then similar files but for different names with varying numbers of files, thus

BobsPhoto_AAB_0001.jpg    through    BobsPhoto_AAB_0147.jpg
BobsPhoto_BFG_0011.jpg    through    BobsPhoto_BFG_0025.jpg
BobsPhoto_XMC_0009.jpg    through    BobsPhoto_XMC_4252.jpg
etc.

I would like all the ABC files in the .\ABC folder, all the AAB files in the .\AAB folder etc. or wherever I specified the folders should go.

In an absolutely perfect world there would be a csv file containing

'ABC','Smith, Mary'
'AAC','Brown, Joan"
etc.

and it would create the destination directories according to those rules (if the csv file is present, otherwise name the directories ABC, AAC etc)

If someone knows of a readily available solution (preferably freeware! :Thmbsup:) I'd love to hear about it. All my Googling has produced numerous solutions, none of which do quite what I want.

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
Are all the original files in one source folder?

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
Yes they are.

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
By chance, have you checked out Skrommel's MoveOut?

https://www.donation...l/index.html#MoveOut

If that doesn't do it for you, I'll write you a custom app.

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
Hi skwire,

I checked out Moveout and it is indeed closer than much else I have found other than perhaps RoboBasket whose author says it can do the job. He said (In your case you can use the "Move to" action and check the "sort into subfolder" option , insert the "name.part2"  substring into the text box. ), but it certainly isn't intuitive as to why I need to do what he said (nowhere is defined what the name, name.part1, name.part2, name.part3, name.lastpart substrings refer to or even where or how one can define them). The software should (if it's designed properly) not need a manual for all but the intellectually challenged  :-\) or it isn't written well enough, IMHO.

MoveOut (as far as I can see) lacks the ability to create the necessary sub-folders and then move the files that meet the wildcard criteria into the self-named newly-created folders.

Did I miss something?

Mike

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
MoveOut (as far as I can see) lacks the ability to create the necessary sub-folders and then move the files that meet the wildcard criteria into the self-named newly-created folders.
Did I miss something?

Ah, apologies.  I didn't realise that MoveOut didn't create the destination folders.  In that case, would it be possible for you to PM me a simple text listing of your image filenames so I can test against them?  I don't need the full paths...just the names.  However, if there are privacy/sensitivity issues with this request, I certainly understand.  In that case, I'll just generate some random filenames based off the patterns in your initial post.  I've got the main framework of your app already finished;  I just need to do a more work on the flow and then some testing.

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
OK, I've tried to give you some sense of the file naming conventions which are reasonably strict as long as you appreciate that they will always start with a 2 or 3-letter or number group, then an underscore, then a 2 or 3 letter designator (typically but no hard and fast rule) then an underscore and some more letters and/or numbers and perhaps another underscore or hyphen and some more letters or numbers but this last part is not relevant to the project.

So what we need to create is the following structure:

root (could be anywhere)
then the first grouping creates a major subdir named after it (in the examples shown FJ or MA)
then subdirs off those based on the next grouping
then subdirs off that based on third grouping
then all the files that fit that pattern into those dirs so only three levels deep

Basically Photographer then Model then Project then Image Number

I hope you appreciate that I need an automatic process because (currently) there are 21,128 filers (and growing) so any needed manual intervention makes it a very laborious process indeed!

I forgot to ask you whether this process can be recursive because currently I have a mess of files in this monster dir and (more recently) some measure of subdirs off the monster dir named with project names and their contents need to be processed according to the overall rules, so you don't have to care what the dirs are named, just enter and process (I know, I'm not asking for much...)

I hope this makes it clear?

Mike

« Last Edit: March 04, 2010, 09:46 PM by mickblu »

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
OK, I've tried to give you some sense of the file naming conventions which are reasonably strict as long as you appreciate that they will always start with a 2 or 3-letter or number group, then an underscore, then a 2 or 3 letter designator (typically but no hard and fast rule) then an underscore and some more letters and/or numbers and perhaps another underscore or hyphen and some more letters or numbers but this last part is not relevant to the project.

Well, therein lies the rub.  "Reasonably strict" doesn't equate to "strict" so this will be very difficult to do automatically.  However, read on.

So what we need to create is the following structure:
root (could be anywhere)
then the first grouping creates a major subdir named after it (in the examples shown FJ or MA)
then subdirs off those based on the next grouping
then subdirs off that based on third grouping
then all the files that fit that pattern into those dirs so only three levels deep

At its most basic, what you describe above is very easy to do.  Cracking a filename on its underscores, creating folders, and moving said file into said folder is a piece of cake.  However, since some of your files have three groups and others have four, this is much more difficult to do accurately.  Do you understand what I mean?

For instance, your MA_* files mostly have four groups (three that matter and then the irrelevant counter portion) but your FJ_* files mostly have three groups (two that matter and then the irrelevant counter portion).  Normally, I could take num_groups-1 and go with that but I can't, in this case, since there are example in your images where the MA_* files have three groups and the FJ_* files have four groups.

At this point, I don't feel comfortable writing a parser that goes three levels deep since there is no hard standard to the filenames.  There are a few ways to approach this:

1) You strictly standarise your filenames at which point I can make the parser as creative as you would like.

2) You divvy up your files into groups that contains the same amount of "underscore groups" and then I can offer an option to crack the filenames on first, second, or third groups.

3) I write the app so you can form basic *? type of wildcard matching.  Something like "MA_*_*.jpg" gets filtered to some destination folder you specify.  One drawback here is that each match entry can only have one destination folder.  Another drawback, I suppose, is that the entry will need to be entered manually.  On the other hand, the list of entries is easily saved and reused.  This is how I started writing this app before you mentioned your automatic-filename-cracking request.

I hope you appreciate that I need an automatic process because (currently) there are 21,128 filers (and growing) so any needed manual intervention makes it a very laborious process indeed!

Sure, I appreciate it but, from my point of the view as the coder who is messing with your files, I have to feel confident that I'm not going to muck them up.

I forgot to ask you whether this process can be recursive because currently

Yes, recursive processing is easily doable.


mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
Hi Skwire,

I appreciate your dilemma, and I think your solution #3 is the best where you allow me to put in the search criteria in the form of a wild-card string. That wouldn't bother me at all as there are really only the two naming conventions (3 group and 4 group) but at least that way any other eventuality is easily handled.

I look forward to hearing from you,

Mike
« Last Edit: March 08, 2010, 11:53 AM by mickblu »

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
Hi Skwire,

I was just wondering where we were at with the "project" as it's been over a week and I hadn't heard back from you.

Sincerely,

Mike

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
Doh...I'll work on this tonight.  Sorry about that.   :-[

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
Hey, no worries, I was just curious is all...  :-[

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
BTW, have you checked out File Sieve 3 at all?  The author is a great coder...I use a few of his apps.  Comical is a favourite.  Anyway, if File Sieve doesn't do what you need, I'll continue work on your custom app.  Please let me know.
« Last Edit: March 11, 2010, 05:12 PM by skwire »

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
Nope, haven't come across that one before, and I searched the heck out of the web at the time, but I guess I just didn't hit the right search phrase...

Certainly looks interesting, and probably can do the job, but because of its sophistication it will take some playing with to ascertain whether it will or won't be able to do the sort. I'll play with it a bit and let you know... What really appeals to me is that it supports multiple input directories.

Thanks for the heads-up!

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
For what it's worth, my custom app for you also has multiple input directories.  Here's a preliminary screenshot:

2010-03-11_174623.pngDONE: Organize files into folders based on flename wildcard parsing

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
Wow! All I can say so far is that it looks about a million times more intuitive than FileSieve!

Out of curiosity, what language are you using to write it?
« Last Edit: March 11, 2010, 05:59 PM by mickblu »

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
If FileSieve doesn't end up working out for you, let me know.  All my applications are written in AutoHotkey.  You can find more examples of my work at my website in my sig.

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
Now you've got me interested in AutoHotKey! It seems like it would, with some learning, enable me to automate a lot of the repetitive things I do.

I don't know how old you are so I don't know how much of what I am about to say will make any sense but, here goes:

I studied programming waaaay back in the 70's (BASIC, COBOL and FORTRAN on an NCR mini coding onto 80-column  punch-cards!) which, while they are certainly not current languages, should give me some structural guidelines. I've also coded in dBase IV and similar (Foxbase etc.) and in  VBasic. In fact, I ran the very first BBS here in town on a 300 baud dialup modem using a Z80 computer running the entire system in interpreted BASIC!

If you were to make up this app for me, would you provide it in AHK source or is that too much to hope for? I'd love to be able to learn from a master!

I've been fiddling with FileSieve and it is a lot less intuitive than I would have hoped, and it's not helped by the almost complete lack of documentation other than a few simple examples in a .chm file.

What would it take to convince you to complete the project for me  :-* ?

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
Hi Skwire,

It seems that FileSieve crashes when faced with the 293,717 files  that I have (and I only pointed it to the "fj" subset, "ma" has another 241,098) so, I guess, your mission, should you choose to accept it sir, is to demonstrate how AHK can do the job!

BTW, FWIW in FileSieve's documentation it says: In this version, it is not currently possible for you to create your own method. It may be possible in a later version which will either provide scripting or plugin support to provide you with this functionality.
« Last Edit: March 11, 2010, 08:28 PM by mickblu »

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
Now you've got me interested in AutoHotKey! It seems like it would, with some learning, enable me to automate a lot of the repetitive things I do.

Oh, my.  I would hate to use a computer without it.  I've built my main "system script" up over the years to the point where vanilla Windows functionality is tough to deal with.

I don't know how old you are

I'm 36.  Not too old...not too young.   :P

If you were to make up this app for me, would you provide it in AHK source or is that too much to hope for? I'd love to be able to learn from a master!

I'm no master...but you're welcome to the source on this one.

What would it take to convince you to complete the project for me  :-* ?

I'll work on it tonight.  There really isn't much left to do other than some GUI bits and testing.

It seems that FileSieve crashes when faced with the 293,717 files  that I have (and I only pointed it to the "fj" subset, "ma" has another 241,098) so, I guess, your mission, should you choose to accept it sir, is to demonstrate how AHK can do the job!

I've no doubt AHK can do it.   :)

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
That will be awesome! Thank you.

Heh, 36! My daughter is 34 and my son is 32 so that gives you a clue how old I am (I'm 62  bearded emoticon.jpg !)

Anyway, I'm really looking forward to seeing what AHK can do...
« Last Edit: March 11, 2010, 10:09 PM by mickblu »

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
To make my testing easier, can you provide me with a text file containing the filenames you're dealing with?
« Last Edit: March 11, 2010, 10:33 PM by skwire »

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
Sure, gimme 5...

mickblu

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 14
    • View Profile
    • Donate to Member
I guess it took somewhat longer than 5  :-[ but I had to find the right utility to produce a recursive directory list...

Anyway, here it is!

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,287
    • View Profile
    • Donate to Member
Nice, thanks.  I've sent you a PM as well.