Author Topic: NANY 2011 Release: Duplicate Photo Finder (Read 83929 times)

Renegade · « **on:** December 20, 2010, 07:28 PM »

NANY 2011 Entry Information

Application Name	Duplicate Photo Finder
Version	1.3
Short Description	Find and delete duplicate photos.
Supported OSes	Windows 2000/XP/
Web Page	Duplicate Photo Finder
Download Link	Download Installer - Download Source Code
System Requirements	Windows XP Home or higher / Windows 2003 / Windows Vista / Windows 2008 / Windows 7 .NET 4.0 200 MB of free RAM
Version History	v1.3 - Update - Increased speed in edge case, now impossible to delete 'originals' in edge cases v1.2 - Update - Improved hashing methods v1.1 - Update - Source released v1.0 - Initial release
Author	Renegade

Description
Duplicate Photo Finder is a photo file management tool to find photos by folder and let you delete duplicates. It aims to be minimalistic with no significant learning curve.

Features
Duplicate Photo Finder:

Lets you choose an "original" and "duplicate" folder to search in
Finds duplicates by file size, file signature (file hash), or photo signature (pixel data hash)
Lets you delete individual photos or many photos at once
Deletes to the Recycle Bin (just in case your brain falls out and you delete the wrong photos)
Only JPG files are scanned
Has no complex features to confuse you

Duplicate Photo Finder is for "photos", and not for other graphics or files. As such, it only scans JPG files because digital cameras take JPG photos and not PNG or GIF or XLS or DOC...

It is also aimed at regular people and hobbyists, and not professional photographers. As such, there is no RAW image support and no professional-type features, e.g. no undelete, no database support. Again, this is aimed at being minimalistic and easy to use.

Planned Features
Planned features include speed optimizations, multi-threading for additional speed, and other image comparison methods (exact image comparisons).

Screenshots

Yes. That is me and my wife~!

And yes, that is me sticking my tongue out~!

2x~!

Usage
Installation
Installation is trivial. Run the installer and follow the directions. The application is installed to:

<drive>:\Program Files (x86)\Renegade Minds\Duplicate Photo Finder

Running the application the first time has no surprises and there is no configuration required.

Using the Application
The installer puts an application shortcut under the Start Menu. i.e. Start > Duplicate Photo Finder. There is no desktop shortcut unless you create one yourself.

1) Browse to 2 folders using the folder browsers.
2) Click the "Compare" button.
3) Decide which photos you want to delete.
4) Click a delete button.
5) Repeat steps 1~4 for other folders that you wish to compare images in.

TIP 1: Use the "Simple" mode first, then do the more thorough "Image Signature" scan afterwards. It will save you time and produce the best results.

TIP 2: Take your time. The program is relatively idiot-proof, and deletes photos to the Recycle Bin, so you can still retrieve deleted photos. However, this is only proof up to "Idiot v5.0"; please do not try to upgrade yourself to "Idiot v6.0" as you will win~!

Uninstallation
Go to the Windows Control Panel applet for uninstalling programs. Click the program's entry, then click "Uninstall".

Known Issues
The application does not make use of threading/multi-threading, and as such requires that the user wait for tasks to complete before starting a new task. Multi-threading is planned.

There is no help file, documentation, or credits yet.

While the program deletes duplicates to the Recycle Bin, and they can be retrieved, there is no substitute for not doing stupid things in the first place.

Remember, this is a bulk deleting utility and can delete large swaths of data quickly. Take your time. There's no rush.

Enjoy~!

skwire · « **Reply #1 on:** December 20, 2010, 07:49 PM »

That's perfect for a NANY.

Renegade · « **Reply #2 on:** December 20, 2010, 08:21 PM »

That's perfect for a NANY.
-skwire (December 20, 2010, 07:49 PM)

Cool.

I'll get stuff sorted shortly. I need to run out the door to get that new mouse now.

I've setup a page for Duplicate Photo Deleter. (Again, need to clean that, but it's up.)

Perry Mowbray · « **Reply #3 on:** December 20, 2010, 10:10 PM »

Not NANY 2011???

Renegade · « **Reply #4 on:** December 20, 2010, 11:28 PM »

Not NANY 2011???
-Perry Mowbray (December 20, 2010, 10:10 PM)

Yeah... Can you move the thread?

Perry Mowbray · « **Reply #5 on:** December 21, 2010, 11:54 PM »

All done!

Deozaan · « **Reply #6 on:** December 22, 2010, 12:31 AM »

CARP! I wish I'd known about this yesterday. I just moved 3,000+ photos into the same directory on someone's computer because they were spread all over, and there were plenty of duplicates, but sometimes the filesize didn't match. So instead of going through every one comparing filesize and timestamp, I just had TeraCopy automatically rename the naming conflicts.

Is there any way this app can find duplicates if they don't have exactly the same name? e.g. PIC-1028.jpg & PIC-1028_2.jpg.

Renegade · « **Reply #7 on:** December 22, 2010, 12:39 AM »

CARP! I wish I'd known about this yesterday. I just moved 3,000+ photos into the same directory on someone's computer because they were spread all over, and there were plenty of duplicates, but sometimes the filesize didn't match. So instead of going through every one comparing filesize and timestamp, I just had TeraCopy automatically rename the naming conflicts.

Is there any way this app can find duplicates if they don't have exactly the same name? e.g. PIC-1028.jpg & PIC-1028_2.jpg.
-Deozaan (December 22, 2010, 12:31 AM)

At the moment I'm only comparing file sizes. File names are irrelevant and I don't look at them at all.

I plan on adding in better comparisons by loading the actual image then comparing that data.

I won't be doing anything like ratios in image comparison, e.g. these pictures are 80% similar. I just think that's a stupid comparison method that's completely unreliable. e.g. You take a picture of someone you are with against a backdrop of some nice scenery, then they take your picture. It's a small amount of difference, but it's important. Since the amount of coding to do facial recognition and all that is really far beyond what I'm prepared to put in right now, I'm not even going to try because I don't see any point in doing a crappy job.

The software right now is stable, but alpha (feature incomplete).

skwire · « **Reply #8 on:** December 22, 2010, 12:02 PM »

Is there any way this app can find duplicates if they don't have exactly the same name? e.g. PIC-1028.jpg & PIC-1028_2.jpg.
-Deozaan (December 22, 2010, 12:31 AM)

You can easily do this using file checksums.

Deozaan · « **Reply #9 on:** December 22, 2010, 12:18 PM »

Is there any way this app can find duplicates if they don't have exactly the same name? e.g. PIC-1028.jpg & PIC-1028_2.jpg.
-Deozaan (December 22, 2010, 12:31 AM)

You can easily do this using file checksums.
-skwire (December 22, 2010, 12:02 PM)

How do you easily compare 3,000+ checksums?

skwire · « **Reply #10 on:** December 22, 2010, 12:20 PM »

Compute them, build a list, report duplicates.

Renegade · « **Reply #11 on:** December 30, 2010, 05:48 AM »

<clears throat /> Ahem...

Ooops.

Got a memory issue popping up that affects large images and don't know if I'll be able to solve it in time.

Sorry... I think this is dead.

Perry Mowbray · « **Reply #12 on:** December 30, 2010, 08:33 PM »

It's never dead until it's dead... but I changed the title to suit

Renegade · « **Reply #13 on:** December 30, 2010, 08:46 PM »

It's never dead until it's dead... but I changed the title to suit
-Perry Mowbray (December 30, 2010, 08:33 PM)

Still working on it... Managed to hack around one super-nasty issue, but with what I want to do in there, it's tough. (IO is a problem as I'm taking an entirely bulk approach and working with my own photos, which are all 10 megapixels, so it's GB at a time and even with optimized super-fast hashing it's slow.)

Perry Mowbray · « **Reply #14 on:** December 30, 2010, 08:52 PM »

Still working on it...
-Renegade (December 30, 2010, 08:46 PM)

Just change the title and the OP to suit if it gets to Release

Renegade · « **Reply #15 on:** December 31, 2010, 02:56 AM »

Made it in time.

Perry Mowbray · « **Reply #16 on:** December 31, 2010, 03:05 AM »

Made it in time.
-Renegade (December 31, 2010, 02:56 AM)

Renegade · « **Reply #17 on:** December 31, 2010, 11:04 PM »

I've updated the software to v1.1.

Changes:

* Image previews now 25~35% faster
* Image load times displayed
* New JPG icon for images
* Selected folders now highlighted in folder browsers
* Press F1 for a nice "About" screen

* Refactored some code
* Added massive amounts of comments to code, complete with stupid humor :p
* Released source code YAY~!

I've commented the source extensively for anyone that wants to have a look or perhaps learn some stuff in C#.

Please note that I've done some very bad things in there (e.g. Application.DoEvents() to maintain a responsive UI). I know that. However, in the interests of getting things done I had to sacrifice in places. e.g. No threading there.

However, there are enough comments there and the really bad things are mostly marked.

Perry Mowbray · « **Reply #18 on:** January 01, 2011, 08:45 AM »

If you're going to continue adding features... one I'd like to see is the ability to test just the image data, not the whole file (which will include EXIF data too)... once I've geotagged my images it messes the matching up

Renegade · « **Reply #19 on:** January 01, 2011, 11:04 AM »

If you're going to continue adding features... one I'd like to see is the ability to test just the image data, not the whole file (which will include EXIF data too)... once I've geotagged my images it messes the matching up
-Perry Mowbray (January 01, 2011, 08:45 AM)

The current feature set hashes the image, and not the file.

Here's the source:

Code: C# [Select]

List<UInt32> hash1 = new List<uint>();
SuperFastHashUnsafe sf1 = new SuperFastHashUnsafe();
Bitmap bmp1;
ImageConverter converter = new ImageConverter();
 
...
 
bmp1 = (Bitmap)Bitmap.FromFile(fi1.FullName);
byte[] bytes = new byte[1];
bytes = (byte[])converter.ConvertTo(bmp1, bytes.GetType());
 
// Add the has value to the list. 
hash1.Add(sf1.Hash(bytes));

And the relevant MSDN article:

http://msdn.microsof....imageconverter.aspx

If I'm not mistaken, only the pixel data is considered, which results in a safe comparison.

I COULD be mistaken though.

If you could send me 2 files to test, I'd appreciate it.

Renegade · « **Reply #20 on:** January 01, 2011, 11:05 AM »

Oh, I should mention that's for the second comparison method. The first is file size only. Check the top combobox there.

Perry Mowbray · « **Reply #21 on:** January 01, 2011, 11:08 AM »

I thought I did it right... but didn't match (can you remind me later for the files?)

Renegade · « **Reply #22 on:** January 01, 2011, 12:20 PM »

I thought I did it right... but didn't match (can you remind me later for the files?)
-Perry Mowbray (January 01, 2011, 11:08 AM)

It matches folders only. Not individual images... Design flaw considering the UI. I'll need to update that. But I will remind you tomorrow to send me the pics.

mouser · « **Reply #23 on:** January 01, 2011, 05:17 PM »

Written up on ghacks

http://www.ghacks.ne...licate-photo-finder/

Renegade · « **Reply #24 on:** January 01, 2011, 09:29 PM »

Written up on ghacks

http://www.ghacks.ne...licate-photo-finder/
-mouser (January 01, 2011, 05:17 PM)

Cool! That's a pleasant surprise~!