Main Area and Open Discussion > General Software Discussion

detect duplicates for 100% sure

(1/2) > >>

kalos:
hello!

I want to detect and delete duplicate files, but is there a way to be 100% that they are identical?

thanks!

Renegade:
hello!

I want to detect and delete duplicate files, but is there a way to be 100% that they are identical?

thanks!
-kalos (March 12, 2012, 09:50 AM)
--- End quote ---

The ONLY way is to do a byte for byte check, and to also check for streams.

This can be done by first checking file sizes then comparing identically sized files, then checking for streams.

You CANNOT rely on hashes as they can collide.

Streams MUST be checked, because even if the main part of the files are identical, they could have streams attached, which are normally invisible to most software.

I do not know of any software that does this, but I'm sure there is some. Perhaps someone else can chime in there.

flamerz:
i use duplicate file detective.

do you want to check it manually?

kalos:
hello!

I want to detect and delete duplicate files, but is there a way to be 100% that they are identical?

thanks!
-kalos (March 12, 2012, 09:50 AM)
--- End quote ---

The ONLY way is to do a byte for byte check, and to also check for streams.

This can be done by first checking file sizes then comparing identically sized files, then checking for streams.

You CANNOT rely on hashes as they can collide.

Streams MUST be checked, because even if the main part of the files are identical, they could have streams attached, which are normally invisible to most software.

I do not know of any software that does this, but I'm sure there is some. Perhaps someone else can chime in there.


-Renegade (March 12, 2012, 09:56 AM)
--- End quote ---

congrats for the depth of knowledge, where you learned these?

indeed, it seems I need a software like this

Renegade:
congrats for the depth of knowledge, where you learned these?

indeed, it seems I need a software like this
-kalos (March 12, 2012, 06:02 PM)
--- End quote ---

That's just some basic knowledge about programming and file systems. Nothing special. There are tonnes more people here at DC than can tell you a lot more than I can. :)

Now, hopefully someone will chime in about this with a software recommendation...

Navigation

[0] Message Index

[#] Next page