Author Topic: SOLVED: Compare locations and remove duplicate folders (Read 29118 times)

dikiz · « **on:** June 10, 2012, 06:55 AM »

Hello! Out of all the existing duplicate finders, none of them really looks for duplicate folders, only files. The closest thing I've found is this http://sourceforge.n...rojects/vfolderdups/

It really performs a duplicate search based on folder names. In addition to that I'm looking for a program that would perform 3 other things :
- A two location mode in order to compare 2 drives/locations
- Possibility to remove all duplicate folders at once (whitout right clicking on all sub folder to delete, impossble to do when there's 2000 duplicates)
- Option to identify a folder as duplicate only if it has not subfolder inside

To understand more the use I would have of it, let me give a concrete example :
I have a folder (A) of mp3 want to use to replace my existing mp3 folder (B) made with different music file type.
   A
   |____Nirvana
   | |_____1991 - Nevermind <= mp3
   | |_____1993 - In Utero <= mp3

   B
   |____Nirvana
   | |_____1991 - Nevermind <= mp3
   | |_____1993 - In Utero <= ogg
   | |_____1992 - Incesticide <= aac

In this example, "B" folder has 2 duplicate sub-folders (Nevermind and In Utero) so they will be removed. NOTE :
   - It doesn't matter if the files inside are different
   - Folders and sub-folders with different names (not existing in "A") won't be touched inside "B" (Incesticide in the example)
   - "Nirvana" folder isn't considered as a duplicate since it contains subfolders
After B folder has been processed :
   B
   |____Nirvana
   | |_____1992 - Incesticide <= acc

It could look a bit like this :

http://img210.imageshack.us/img210/3636/comparefolders.jpg

SOLVED: Compare locations and remove duplicate folders

Thanks to all !

yksyks · « **Reply #1 on:** June 10, 2012, 11:06 AM »

ZTreeWin can do this regardless of the folder contents, it can compare only the folder names. You'll need some of the beta version though (the new final release will be available soon), which work with the directories.

Here you can see the results after the comparison. The blue lines means that the folders are "tagged", so you can do any global operation with them, like deleting, moving, or renaming.

If you're interested and need more info, let me know.

dikiz · « **Reply #2 on:** June 10, 2012, 12:47 PM »

Yes! It looks very interesting in regards to what I'm looking for..

I'd like to compare folders "A" and "B", and I cannot find how in the example bellow:

http://img600.imageshack.us/img600/5303/subtree.jpg

SOLVED: Compare locations and remove duplicate folders

It says directory empty with the "Compare" function

yksyks · « **Reply #3 on:** June 10, 2012, 03:02 PM »

First, you're right, there's a problem with directories with no files in them—I've just reported this to author.

Second, to compare the directories, you have to sWitch to Directory mode (press W, and on one side is enough), because you don't want to work with files, but directories only. Then press Enter to display the contents. Then Alt-F4 for compare, then Enter, and key "I" repeatedly until you reach "tag: Identical (name )". That should do, provided the relative roots contain any files. This for sure will be fixed soon, the author is very active in fixing problems.

dikiz · « **Reply #4 on:** June 10, 2012, 03:20 PM »

Thanks
Well it is actually a good thing for me if this program doesn't compare a folder without files, but it should if it contains subfolders.

In the example above, "Autechre" should not be considered as a duplicate (no files inside, only folders)
Do you think it is possible to acheive this with Ztree?

4wd · « **Reply #5 on:** June 11, 2012, 12:10 AM »

There's also FreeCommanderXE, (which looks better IMHO

):

Open Extras->Settings->Compare Folders and choose your options:

SOLVED: Compare locations and remove duplicate folders

Navigate to the folders in the Left and Right pane:

SOLVED: Compare locations and remove duplicate folders

Select Folder->Compare (Alt + V by default) and the folders that are different will be highlighted on both sides but only the right side will be active:

SOLVED: Compare locations and remove duplicate folders

Select Edit->Invert Selection (Num * by default):

SOLVED: Compare locations and remove duplicate folders

Then you can delete, move, etc.

yksyks · « **Reply #6 on:** June 11, 2012, 02:13 AM »

Thanks
Well it is actually a good thing for me if this program doesn't compare a folder without files, but it should if it contains subfolders.
-dikiz (June 10, 2012, 03:20 PM)

Exactly. In essence this is what I reported to the author.

In the example above, "Autechre" should not be considered as a duplicate (no files inside, only folders)
Do you think it is possible to acheive this with Ztree?
-dikiz (June 10, 2012, 03:20 PM)

I don't get it here. In my opinion both the "Autechre" folders are duplicates, according to theirs names, aren't they? They even contain the exactly same subfolders. Anyway, for the time being you can experiment with ZTW with a workaround by putting one small file into each empty "Autechre" folder.

There's also FreeCommanderXE, (which looks better IMHO ):
-4wd (June 11, 2012, 12:10 AM)

Never heard of it, looks interesting. Not that I'm going to even try, though. ZTree is addictive. It's not an eye candy application, it's targeted on keyboard maniacs like myself, who prefer productivity and speed. I bet nothing can beat it in this regard.

dikiz · « **Reply #7 on:** June 11, 2012, 04:26 AM »

I don't get it here. In my opinion both the "Autechre" folders are duplicates, according to theirs names, aren't they? They even contain the exactly same subfolders. Anyway, for the time being you can experiment with ZTW with a workaround by putting one small file into each empty "Autechre" folder.
-yksyks (June 11, 2012, 02:13 AM)

Autechre folders are different in "A" and "B", the subfolders are different (Amber vs Anti EP). So imagine if theses "Autechre" were marked as duplicates, they would be removed in "B", no matter if the subfolders are different.
This is why I say folders without files shouldn't be compared.

I'll try FreeCommanderXE too. I don't know where you found these, I've spent hours searching and I could only come up with files comparison tools!

4wd · « **Reply #8 on:** June 11, 2012, 05:38 AM »

I'll try FreeCommanderXE too. I don't know where you found these, I've spent hours searching and I could only come up with files comparison tools!
-dikiz (June 11, 2012, 04:26 AM)

A lot of filemanagers, (especially dual pane), have some kind of comparison function, just depends how far they can be tweaked to get what you want.

Chinese site but here's a list of 60 - knock yourself out

The site is out of date so some of the links don't work but a search should find most of the no longer linked ones.

dikiz · « **Reply #9 on:** June 11, 2012, 01:14 PM »

4wd, do you know if there's a way so that FreeCommanderXE will scan and compare subfolders too?

yksyks · « **Reply #10 on:** June 11, 2012, 01:16 PM »

Autechre folders are different in "A" and "B", the subfolders are different (Amber vs Anti EP). So imagine if theses "Autechre" were marked as duplicates, they would be removed in "B", no matter if the subfolders are different.
This is why I say folders without files shouldn't be compared.
-dikiz (June 11, 2012, 04:26 AM)

Oh, I see. I didn't notice the difference. I'm afraid this is not possible in ZTree, because in the above described case you compare the names of directories only, regardless of theirs contents. I though that's what you wanted. If you need to compare the complete subdirectories structure, it would require a recursive approach, or whatnot. Or just one level would be enough? However, there are other comparing options in ZTree, try them out.

dikiz · « **Reply #11 on:** June 11, 2012, 01:48 PM »

To avoid that "Autechre" folders and such are marked as duplicate, I thought it would work if the possibility was given to mark a folder as duplicate only if it contains a file or more.
Folders with only subfolders inside would not count.

But I guess it would work also to work at a +1 level ! Most of the folders I have to handle are at this level:
http://dl.dropbox.com/u/251882/mp3.txt

4wd · « **Reply #12 on:** June 11, 2012, 09:37 PM »

4wd, do you know if there's a way so that FreeCommanderXE will scan and compare subfolders too?
-dikiz (June 11, 2012, 01:14 PM)

Not AFAIK, sorry.

It's the filemanager I use when my normal one isn't available, (Directory Opus), but the XE version is technically still beta ATM - so you could pass along a request to the developer.

dikiz · « **Reply #13 on:** June 12, 2012, 04:13 PM »

Well my need is so specific that I doubt something actually exists. And since it's one shot need, I've started to do it manually. A long way to go, several months I guess, of 2511 folders to check :-)

skwire · « **Reply #14 on:** June 12, 2012, 05:18 PM »

You know, it wouldn't take much at all to write a script that would spit out a text listing of the differences between two folder's folder names. That being said, I'd really rather not make the script automatically delete anything, etc. You'd then, at least, have a list of folder names to reference instead of going about it manually. Would something like that be acceptable?

IainB · « **Reply #15 on:** June 12, 2012, 09:39 PM »

Well my need is so specific that I doubt something actually exists. And since it's one shot need, I've started to do it manually. A long way to go, several months I guess, of 2511 folders to check :-)
-dikiz (June 12, 2012, 04:13 PM)

Apologies for this belated input. I only just saw this comment. I could have replied sooner if I had been looking more closely at this discussion.

What you want to do appears to be a really dead simple exercise that will probably not take you more than a few minutes at most to do using the standard features already built in to xplorer² - that includes sorting and filtering for folders in 2 panes side-by-side, identifying and deleting all the unwanted duplicate folders in B, as per your example above. You can do this for any two folder structures of the Parent-Child sort A + B as per your example. No coding required.
I think you should be able to do this using an evaluation copy - or maybe the free version - of xplorer².
(I'm not sure as I have had the PRO (paid) version for years.)

xplorer² makes this easy due to:

(a) It being able to sort and display "flat" file/folder lists of your nested folder tree structure ("local search"), and you can filter out files so you only see the folders - which is what you want, because the actual file contents of those folders are apparently irrelevant to what you require to do. Just tested it now. Time: probably take about 5 to 10 sec. for each - (A and then B) - to set the search filter, have the results (the flattened, filtered folders list) displayed in one panel (side) of the "container" window. So both folders' lists flattened, sorted, filtered and displayed side-by-side, showing just the bottom-level folders that you want to compare/delete. Time: (say) 20 sec. max. (New users of xplorer² will probably be slower.)
(b) It having a sync wizard (invoked using Ctrl-F9) where you select a target panel (I selected the B folder list side first) to highlight all those folder names that are the same in A. Time: about 3 sec.

At which point you press the delete key and all those highlighted folders in B are deleted. Time: about 3 sec.

Finished/Done.

The only things that I could think of that might slow you down are:

1. Multiple duplicates: If, on seeing the flattened folder listing, you then discover that you have identical duplicated nested folder/subfolder names in either of the A or B display panels, because you will then have to manually identify them from the flattened folder listing (path can be displayed to help) to establish which ones you may want to rename/keep/delete.
2. Varying depths of nested folders: If your folders nest more deeply than the consistent Parent-Child example you give above, in which case deleting folders could be dangerous and not what you want, as it will also delete all subfolders and files of the folders deleted.
3. Inconsistent naming standard: If you don't have a consistent naming standard, and especially for the bottom-level folders that you are wanting to compare/delete. This will make the list view filtering trickier.

dikiz · « **Reply #16 on:** June 14, 2012, 03:34 PM »

Thank you IainB. I will look into it this weekend, looks very promising!

dikiz · « **Reply #17 on:** June 16, 2012, 01:06 PM »

Well it really looks like xplorer² can make it! Will probably save my summer from a very stupid and long job

Many many thanks !

The only remaining time-consuming task is to un-select all folders (artists) to have only subfolders (releases) selected

SOLVED: Compare locations and remove duplicate folders
Sorting by "emplacement" easily seperate folders from subfolders. But is there a way to quickly unselect folders?

IainB · « **Reply #18 on:** June 16, 2012, 05:54 PM »

That's great! I thought it might be what you probably needed.

In answer to your Q:
I don't think you mentioned subfolders for "releases" in this thread before. Do you mean you have the situation as above in 3. Inconsistent naming standard, or is it 2. Varying depths of nested folders?

Otherwise, if you want to delete some of the folders selected, the quickest way is to turn on Edit-->Sticky selection (hotkey is Alt+S), then the files you have selected can be unselected by just going down (cursor DOWN) them and pressing the spacebar to deselect an already-selected item. The rest of the selected items will remain selected ("sticky select", you see). This saves you from a lot of trouble/mucking about with Shift or Ctrl keys.

If you make a mistake and inadvertently press the spacebar on an unselected item, it will merely become a selected item (the spacebar is a toggle), so you can deselect it the same way.

xplorer² is a seriosuly powerful file management tool. Please do a backup of all your relevant files before doing this, in case you have to repeat it because you made a mistake and deleted something you wanted. It's often easier to start over in such cases.

If there is anything complex you need to do with files, a read through the xplorer² User Manual will generally surprise you with what it can do. I regard it as being rather like a database management system, where the files and folders are components in the database. I probably only know how to use about 20% of the tool's functionality, and learn something new about it every couple of weeks.

dikiz · « **Reply #19 on:** July 08, 2012, 10:10 AM »

Ok it's done! It took me so little time that I felt almost frustrated

To compare only subfolders (releases) and not folders (artists), I sorted them by column location. It groups all folders together and subfolders together. On the pic you can see the limit.

SOLVED: Compare locations and remove duplicate folders
I selected all folders with mouse, then press "delete" on my keyboard. I did that on the right panel too, THEN (and only then) I launched the comparison with CTRL + F9, so that only subfolders were compared. And it worked great!
Thanks a ton IainB, it was definitetly the most helpful assistance I've ever received on a internet forum!

skwire · « **Reply #20 on:** July 08, 2012, 11:07 AM »

Great to hear, dikiz, and thanks to IainB for the tip. Do you mind if I mark this thread as Solved?

dikiz · « **Reply #21 on:** July 08, 2012, 11:37 AM »

Sure it's solved

Can be marked

IainB · « **Reply #22 on:** July 08, 2012, 11:58 AM »

Ok it's done! It took me so little time that I felt almost frustrated
-dikiz (July 08, 2012, 10:10 AM)

Glad it solved the problem!
xplorer² really does seem to be a uniquely powerful file/folder explorer/manager.