detect duplicates for 100% sure

[kalos]

hello!

I want to detect and delete duplicate files, but is there a way to be 100% that they are identical?

thanks!

[Renegade]

hello!

I want to detect and delete duplicate files, but is there a way to be 100% that they are identical?

thanks!

-kalos (March 12, 2012, 09:50 AM)

The ONLY way is to do a byte for byte check, and to also check for streams.

This can be done by first checking file sizes then comparing identically sized files, then checking for streams.

You CANNOT rely on hashes as they can collide.

Streams MUST be checked, because even if the main part of the files are identical, they could have streams attached, which are normally invisible to most software.

I do not know of any software that does this, but I'm sure there is some. Perhaps someone else can chime in there.

[flamerz]

i use duplicate file detective.

do you want to check it manually?

[kalos]

hello!

I want to detect and delete duplicate files, but is there a way to be 100% that they are identical?

thanks!

-kalos (March 12, 2012, 09:50 AM)

The ONLY way is to do a byte for byte check, and to also check for streams.

This can be done by first checking file sizes then comparing identically sized files, then checking for streams.

You CANNOT rely on hashes as they can collide.

Streams MUST be checked, because even if the main part of the files are identical, they could have streams attached, which are normally invisible to most software.

I do not know of any software that does this, but I'm sure there is some. Perhaps someone else can chime in there.

-Renegade (March 12, 2012, 09:56 AM)

congrats for the depth of knowledge, where you learned these?

indeed, it seems I need a software like this

[Renegade]

congrats for the depth of knowledge, where you learned these?

indeed, it seems I need a software like this

-kalos (March 12, 2012, 06:02 PM)

That's just some basic knowledge about programming and file systems. Nothing special. There are tonnes more people here at DC than can tell you a lot more than I can.

Now, hopefully someone will chime in about this with a software recommendation...

[rjbull]

DoubleKillerPro has plenty of other options, though it doesn't mention streams.  There's a lighter free-for-personal-use version as well.

[xtabber]

Unless a file is open, I don't believe streams have any bearing on its content, so they probably can be safely ignored unless you are operating in an  environment where files to be compared are constantly being accessed.

I use Beyond Compare to do byte-for-byte comparisons of any two files, or all the files in two different directory trees.  However, Beyond Compare does not search for duplicates.

I use TreeSize Pro to find duplicates and optionally delete them. TreeSize Pro has a number of options for comparing files, incluuding MD5 and SHA256 checksums, which I feel is more than adequate. If you don't feel that's enough, you can always check based only on size and use Beyond Compare to do a byte-for-byte comparison of all matches found.

[SKA]

Check out Delete Duplicate Files (USD 19/-):
http://www.drivehq.com/web/brana/ddf.htm

Ska