Compare folders with identical files, but .. different file names

[dcwul62]

Have vainly been searching for a tool that compares folders that contain in fact the same files, but the files in one folders are renamed.
Meaning to say that hashes, date time stamp and sizes of the files are the same.

All compare tools take the file names into account, by design, there is no option to just compare based on MD5 or so.

I have two relatively big folders, with probably over 90% the same files (different names though), but the folder sizes are different and I would like to figure out what files are missing in the smaller folder.
The tool should therefore 'hide' identical files.

WinDiff, WinMerge, FreeFileSync, GoodSync, Beyond Compare, ExamDiff Pro: not possible.

Thanks!

[Ath]

I've done smth similar, many moons ago, and then found a PowerShell script that found me duplicates based on the hash, so finding non-dulicates seems trivial after that
Didn't find the script I used back then, but this link should be what you want to find, IMHO.

[IainB]

xplorer² <https://www.zabkat.com/> has offered this handy facility (compare CONTENT of two folders or filter panes) for ages. I use it quite often, and it works a treat. All the files matching the compare criteria are automatically selected so that they can be operated on easily.

I read that the compare process calculates and compares checksums. Checksums can also be dynamically auto-calculated and listed for files and displayed in a view pane column, which can be sorted for visual comparison of checksums. Since checksums take more CPU time/resource to calculate for larger files, the upper limit of file size for auto-calculating cheksums can be set by the user. This could make an appreciable time difference when comparing folders or filter panes containing thousands of files.

Example of content matching pane settings:

[tomos]

Dopus claims to do a byte comparison when syncing -- I've never used that, but just tried it:
Copied a file to a new folder and renamed
the sync is not recognising them as the same, and is prompting to copy over the original (again).

[tomos]

xplorer² <https://www.zabkat.com/> has offered this handy facility (compare CONTENT of two folders or filter panes) for ages. I use it quite often, and it works a treat. All the files matching the compare criteria are automatically selected so that they can be operated on easily.

-IainB (May 19, 2018, 12:22 PM)

I'm not clear from your description:
does it recognise that files are the same even if named differently?

Xyplorer does not appear to do this (but it not my default file manager, so I not 100% sure)

[IainB]

does it recognise that files are the same even if named differently?

-tomos (May 19, 2018, 02:08 PM)

Good Q. I had thought that it did, but I tested it just now to make sure and it seems to do a comparison of Filename AND content (checksum) - i.e., NOT just a comparison of the checksums.
So that's not much use to you, I guess.

Bother! Sorry to mislead you. I should have tested it before commenting.

Actually, I might post this as a query on the xplorer² user forum. Will let you know if I get a useful response.

EDIT:
I just checked at zabkat.com (xplorer²) user forum.
There I searched and found an old discussion where they recommended Beyond Compare which has Features that include: Compare files or clipboard contents.
So that might be worth a trial. It's not $FREE though and not particularly cheap either...

[skwire]

@dcwul62: If none of the other solutions pan out, I could probably add this functionality to my SFV Ninja program.

[mwb1100]

There are quite a few duplicate file finder programs out there designed to find files that are the same even though the names and/or time stamps are different.  I'd imagine that most of them would be able to report 'orphans' as well.

Unfortunately, I haven't used any of them in ages so I'm no position to suggest one, but I think googling for "duplicate file finder" would net you something worth trying.

[antekgla]

Fast Duplicate File Finder claims to do the taks you want.
I dont try myself, found by google in this article from Trouble fixers

Edit: after read your FULL post I find what this software dont show the unique files besides only shows the identical ones. So i dont know if suits yours needs.

[4wd]

The tool should therefore 'hide' identical files.

-dcwul62 (May 19, 2018, 07:59 AM)

They gave you the answer on the DOpus forum.

The first seven files on the right are renamed copies of the last seven on the left plus an extra file thrown in on the right to show it wasn't in the left side.

Start:
Compare folders with identical files, but .. different file names


Files that have identical MD5 hashes are selected on both sides.

Result:
Compare folders with identical files, but .. different file names

A modified script from here hides selected files, (simple search of the forum), leaving the files that aren't identical to any on the other side.

Compare folders with identical files, but .. different file names

Compare folders with identical files, but .. different file names

Hide button:

Code: Javascript [Select]

1. function OnClick(clickData)
2. {
3.         var cmdSource = clickData.func.command;
4. //      cmdSource.ClearFiles();
5.  
6.         var cmdDest = DOpus.create.command;
7.         cmdDest.SetSourceTab(clickData.func.desttab);
8.  
9. //      cmdSource.RunCommand("Select DESELECTNOMATCH SOURCETODEST=in");
10. //      cmdDest.RunCommand("Select DESELECTNOMATCH SOURCETODEST=in");
11.         cmdSource.RunCommand("Select NOPATTERN HIDESEL");
12.         cmdDest.RunCommand("Select NOPATTERN HIDESEL");
13. }

UnHide function is identical to the one on the forum that Leo posted.

[dcwul62]

Sorry for the delay, one way or the other I didn't get any notifications anymore.

IanB

I tried following
-
Compare folders with identical files, but .. different file names
-

But then what? When I click on [OK]  nothing seems to happen.
For yr info: I selected the two folders in the right window panel and then Sync Wizard.

@antekgla
Indeed Duplicate File Finder could not do what I wanted (I -did- gave it a try anyhow, but bad luck)

@skwire
Well, as said, I have tried quite a few of those file comparison tools, but haven't found any of them to offer what I want.
Guess it is quite unique then.

@4wd
the script didn't work, at least not with me. Reason why I am posting here...
I had the toolbar button alright (meanwhile removed), but got a Windows error.
Seems the script should be stored in some folder that 'my' Directory Opus does not have. I have no 'script' folder, nor Dopusdata, nor something that even looks like that. After trying over and over and vainly checking on Internet and reading long texts, well, I gave up.
(Felt I shouldn't keep on nagging there in the Opus forum)
Also please note that I am just an 'end-user' - I am not familiar with scripts. Sorry.

The thread 'Hide identical file names' - that one is new to me. Thanks.
However, actually the issue is that the file names are not identical, the hashes are identical.
It would be nice if it was 'hide files with identical hashes'

[4wd]

the script didn't work, at least not with me. Reason why I am posting here...
I had the toolbar button alright (meanwhile removed), but got a Windows error.

-dcwul62 (May 21, 2018, 09:02 AM)

As was mentioned, it's a script addin, you need to install it: Preferences->Toolbars->Scripts and drop it on the interface.
Compare folders with identical files, but .. different file names

Also please note that I am just an 'end-user' ...

As am I but I do know how to use the forum search and it's also in the manual.

The thread 'Hide identical file names' - that one is new to me. Thanks.
However, actually the issue is that the file names are not identical, the hashes are identical.

You need to reread what I wrote:

A modified script from here hides selected files, ...

-4wd (May 20, 2018, 01:35 AM)

It's got nothing to do with hiding identical file names other than only keeping the parts of the script that were necessary to hide selected files.

[dcwul62]

the script didn't work, at least not with me. Reason why I am posting here...
I had the toolbar button alright (meanwhile removed), but got a Windows error.

-dcwul62 (May 21, 2018, 09:02 AM)

As was mentioned, it's a script plugin, you need to install it: Preferences->Toolbars->Scripts and drop it on the interface.
[ Invalid Attachment ]

-4wd (May 21, 2018, 09:54 AM)

it was there alright.

Also please note that I am just an 'end-user' ...

As am I but I do know how to use the forum search.

you are great!!

As for the rest: forget it, let it rest.

I was asking for a simple tool, but end up with scripts that don't work with me as well as allegations for not looking good enough.

again, forget it!

[antekgla]

Well I search a solution for this and after some research a find a very primitive one...only a list of the unique files

You need the program HashMyFiles from Nirsoft.

Conventions in this explanation: Big_Folder (the folder with more files) Small_Folder (the folder with fewer files)

The procedure is this:

• Open HashMyFiles and in Options --> Hash Types only select MD5 hash (you only need a hash to identify the files)
• In HashMyFiles drag the Big_Folder wait a little and the program would calculate the MD5 hashes.
• In View --> Choose Colums only select MD5 & Full Path (in that order)
• Select All (Ctrl+A)
• Save Selected Items (Ctrl+S) . Choose Tab Delimited text file in Type. Name the file Big.txt
• You ended with a file like this:

[Select]

e520af771051085a0d88f681b1e3aa07 C:\Users\USER\Portable.Programs\Splayer\ir41_32.ax
652809bf6fc8ff180094b069f0612188 C:\Users\USER\Portable.Programs\Splayer\ir50_32.dll
727641a2c4b80374aa03da42a62cc6d9 C:\Users\USER\Portable.Programs\Splayer\ivm.dll
261ceeafb613f8ae1ec96814a0a4ed8f C:\Users\USER\Portable.Programs\Splayer\IVMSource.ax
9c4fe9c4f64947a8778fa9f588aae4a0 C:\Users\USER\Portable.Programs\Splayer\mc.dll
5a4cb17bfcba5d7d78f8296408dfacda C:\Users\USER\Portable.Programs\Splayer\media.db
b7f43e6dc5b7199f5dc12e3a6ca00848 C:\Users\USER\Portable.Programs\Splayer\MediaInfo.dll
8d803ebe525991e6c85ac047d39b569a C:\Users\USER\Portable.Programs\Splayer\mkunicode.dll
51424ca9507cfb6eaf19ab4c78273b88 C:\Users\USER\Portable.Programs\Splayer\mkx.dll
9df0f8c0acc5548f32906f6ea4d222b1 C:\Users\USER\Portable.Programs\Splayer\mkzlib.dll
e7d1fed458491c4963da4529756d46dc C:\Users\USER\Portable.Programs\Splayer\mmamrdmx.ax
17cf953ae7ea3128f1a8d44a39746011 C:\Users\USER\Portable.Programs\Splayer\mp4.dll
43316f8a3072ce9ba9a82526e7f94987 C:\Users\USER\Portable.Programs\Splayer\ogm.dll
dc1defde4f0b51bd17332586d0962786 C:\Users\USER\Portable.Programs\Splayer\PmpSplitter.ax
7668248c3101e6cca0b88fc9ea99f6a3 C:\Users\USER\Portable.Programs\Splayer\RadGtSplitter.ax
f8dd535c7c145b18d31e00d40f1ffef8 C:\Users\USER\Portable.Programs\Splayer\rlapedec.ax
263e1422ef9fb6a21922dc6282fe0431 C:\Users\USER\Portable.Programs\Splayer\settings.db
79d5c71c74566beb2bb18e392f2403bf C:\Users\USER\Portable.Programs\Splayer\sinet.dll
9dcf8871a1c8fbf20fbd9cd8b332cea4 C:\Users\USER\Portable.Programs\Splayer\smackw32.dll

• Again in HashmyFiles Clear All (Ctrl+X).
• Now drag Small_Folder in HashMyFiles
• In View --> Choose Colums only select MD5
• Select All (Ctrl+A)
• Save Selected Items (Ctrl+S) . Choose Tab Delimited text file in Type. Name the file Small.txt
• You ended with a file like this:

[Select]

c6771e032b4085968e8e081c57c49f0b
002cdf612509807b33e4ab09c686a966
b6cfb690fe5997da0f07506c8982334f
3fa06cf5079b84155d18b05c08f7131b
7a8a8919feeeab8f7dbda713510418ab
e7cf8ddd9941ba6e08e212d91759b174
0dc0734ba778ef05933cd8a3d9a2fff1
c37f106cba9a726e5689eb610c5b1c77

Now open a command window and change to the folder you saved Big.txt and Small.txt and type this command:

[Select]

findstr /v /g:small.txt big.txt > diff.txt

In the file diff.txt you have only the unique files (and his MD5 hashes).
Edit in Notepad++ or any other plain text editor what permit block (or column) selection (In Notepad++ alt+arrow keys or alt+mouse drag) to remove the MD5 hashes and you end with a list of unique files only.

If you use Teracopy or Fastcopy you can use my Software DualCopy to paste that list and copy that files. 

Edit: Make sure what your file names dont cointains UNICODE chars because the resulting txt would be Unicode and the command findstr DONT WORK WITH UNICODE STRINGS.

[tomos]

Seems the script should be stored in some folder that 'my' Directory Opus does not have. I have no 'script' folder, nor Dopusdata, nor something that even looks like that.

-dcwul62 (May 21, 2018, 09:02 AM)

hi dcwul62, I was hoping to try installing this and was wondering what version of dopus are you using? (I have 11 on this machine and 12 on my desktop -- I will try it anyways, but will probably only get to it later in the week.)

the script didn't work, at least not with me. Reason why I am posting here...
I had the toolbar button alright (meanwhile removed), but got a Windows error.
Seems the script should be stored in some folder that 'my' Directory Opus does not have. I have no 'script' folder, nor Dopusdata, nor something that even looks like that. After trying over and over and vainly checking on Internet and reading long texts, well, I gave up.
(Felt I shouldn't keep on nagging there in the Opus forum)

-dcwul62 (May 21, 2018, 09:02 AM)

I only see one reply in the thread 4wd links to (?) FWIW I'm not trying to give you a hard time -- just trying to understand.

[4wd]

Seems the script should be stored in some folder that 'my' Directory Opus does not have. I have no 'script' folder, nor Dopusdata, nor something that even looks like that.

-dcwul62 (May 21, 2018, 09:02 AM)

hi dcwul62, I was hoping to try installing this and was wondering what version of dopus are you using? (I have 11 on this machine and 12 on my desktop -- I will try it anyways, but will probably only get to it later in the week.)

-tomos (May 22, 2018, 02:31 PM)

The shortcut /scripts also takes you to the relevant directory, type it in the lister path.

[skwire]

@dcwul62: Apologies for the delay.  Please give this a try:

http://www.dcmembers...ja/SFV_Ninja_Mod.zip

• Download, unpack, and run SFVNinja_Mod.exe
• Load your folders/files.  You can drag-and-drop or use the toolbar buttons.
• Click the "All" button in the toolbar.  Depending on number and size of the files, this might take a while.
• Once SFV Ninja is done generating the checksums, press F12 to reduce the list to unique files.

Let me know if this works for you.

[antekgla]

@dcwul62: Apologies for the delay.  Please give this a try:

http://www.dcmembers...ja/SFV_Ninja_Mod.zip

• Download, unpack, and run SFVNinja_Mod.exe
• Load your folders/files.  You can drag-and-drop or use the toolbar buttons.
• Click the "All" button in the toolbar.  Depending on number and size of the files, this might take a while.
• Once SFV Ninja is done generating the checksums, press F12 to reduce the list to unique files.

Let me know if this works for you.

-skwire (June 04, 2018, 05:40 PM)

Seems like @dcwul62 dont participate more of the thread. Maybe offended... I dont known...

Some feedback about SFV_Ninja: The F12 trick is something quickly you do specifically for @dcwul62? ...Because dont seems documented in any form and dont correspond to any menu in the UI. Besides once F12 is pressed there is not way to return to the original list.
Another sugestion would be the posibility of copy/cut to the clipboard the files in the list (the files dont the filenames). That way would be possible copy in SF_Ninja list and paste in a new folder.
Great work in the app...skwire I use several of your programs... there are great!

[skwire]

Seems like @dcwul62 dont participate more of the thread. Maybe offended... I dont known...

-antekgla (June 05, 2018, 02:00 PM)

I sent him a personal message as well in hopes that he sees this.

Some feedback about SFV_Ninja: The F12 trick is something quickly you do specifically for @dcwul62? ...Because dont seems documented in any form and dont correspond to any menu in the UI.

-antekgla (June 05, 2018, 02:00 PM)

You are correct.  I created a custom build just for dcwul62's request.

Besides once F12 is pressed there is not way to return to the original list.

-antekgla (June 05, 2018, 02:00 PM)

Yep.  Again, just a quick mod for his needs.  I wasn't sure if the need extended to other users.

Another sugestion would be the posibility of copy/cut to the clipboard the files in the list (the files dont the filenames). That way would be possible copy in SF_Ninja list and paste in a new folder.

-antekgla (June 05, 2018, 02:00 PM)

That's a good idea.  I'll keep it in mind.

Great work in the app...skwire I use several of your programs... there are great!

-antekgla (June 05, 2018, 02:00 PM)

Thank you for your kind words.  I'm glad you find the applications useful.