topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Tuesday March 19, 2024, 1:49 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: NANY 2011 Release: Duplicate Photo Finder  (Read 80696 times)

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
NANY 2011 Release: Duplicate Photo Finder
« on: December 20, 2010, 07:28 PM »
NANY 2011 Entry Information

Application Name Duplicate Photo Finder
Version 1.3
Short Description Find and delete duplicate photos.
Supported OSes Windows 2000/XP/
Web Page Duplicate Photo Finder
Download Link Download Installer - Download Source Code
System Requirements
  • Windows XP Home or higher / Windows 2003 / Windows Vista / Windows 2008 / Windows 7
  • .NET 4.0
  • 200 MB of free RAM
Version History
  • v1.3 - Update - Increased speed in edge case, now impossible to delete 'originals' in edge cases
  • v1.2 - Update - Improved hashing methods
  • v1.1 - Update - Source released
  • v1.0 - Initial release
Author Renegade


Description
Duplicate Photo Finder is a photo file management tool to find photos by folder and let you delete duplicates. It aims to be minimalistic with no significant learning curve.


Features
Duplicate Photo Finder:

  • Lets you choose an "original" and "duplicate" folder to search in
  • Finds duplicates by file size, file signature (file hash), or photo signature (pixel data hash)
  • Lets you delete individual photos or many photos at once
  • Deletes to the Recycle Bin (just in case your brain falls out and you delete the wrong photos)
  • Only JPG files are scanned
  • Has no complex features to confuse you

Duplicate Photo Finder is for "photos", and not for other graphics or files. As such, it only scans JPG files because digital cameras take JPG photos and not PNG or GIF or XLS or DOC... ;)

It is also aimed at regular people and hobbyists, and not professional photographers. As such, there is no RAW image support and no professional-type features, e.g. no undelete, no database support. Again, this is aimed at being minimalistic and easy to use.


Planned Features
Planned features include speed optimizations, multi-threading for additional speed, and other image comparison methods (exact image comparisons).


Screenshots



Yes. That is me and my wife~! :D

And yes, that is me sticking my tongue out~! :P 2x~! :P


Usage
Installation
Installation is trivial. Run the installer and follow the directions. The application is installed to:

<drive>:\Program Files (x86)\Renegade Minds\Duplicate Photo Finder

Running the application the first time has no surprises and there is no configuration required.  


Using the Application
The installer puts an application shortcut under the Start Menu. i.e. Start > Duplicate Photo Finder. There is no desktop shortcut unless you create one yourself.

1) Browse to 2 folders using the folder browsers.
2) Click the "Compare" button.
3) Decide which photos you want to delete.
4) Click a delete button.
5) Repeat steps 1~4 for other folders that you wish to compare images in.

TIP 1: Use the "Simple" mode first, then do the more thorough "Image Signature" scan afterwards. It will save you time and produce the best results.

TIP 2: Take your time. The program is relatively idiot-proof, and deletes photos to the Recycle Bin, so you can still retrieve deleted photos. However, this is only proof up to "Idiot v5.0"; please do not try to upgrade yourself to "Idiot v6.0" as you will win~! :D


Uninstallation
Go to the Windows Control Panel applet for uninstalling programs. Click the program's entry, then click "Uninstall".


Known Issues
The application does not make use of threading/multi-threading, and as such requires that the user wait for tasks to complete before starting a new task. Multi-threading is planned.

There is no help file, documentation, or credits yet.

While the program deletes duplicates to the Recycle Bin, and they can be retrieved, there is no substitute for not doing stupid things in the first place. ;) Remember, this is a bulk deleting utility and can delete large swaths of data quickly. Take your time. There's no rush. :)




Enjoy~! :D
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker
« Last Edit: January 24, 2014, 09:24 PM by Renegade, Reason: Updated links. »

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
Re: NANY 2010 Proposal - Duplicate Photo Deleter
« Reply #1 on: December 20, 2010, 07:49 PM »
That's perfect for a NANY.   :up:

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2010 Proposal - Duplicate Photo Deleter
« Reply #2 on: December 20, 2010, 08:21 PM »
That's perfect for a NANY.   :up:

Cool.

I'll get stuff sorted shortly. I need to run out the door to get that new mouse now. :)

I've setup a page for Duplicate Photo Deleter. (Again, need to clean that, but it's up.)
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Perry Mowbray

  • N.A.N.Y. Organizer
  • Moderator
  • Joined in 2005
  • *****
  • Posts: 1,817
    • View Profile
    • Donate to Member
Re: NANY 2010 Proposal - Duplicate Photo Deleter
« Reply #3 on: December 20, 2010, 10:10 PM »
 :-[ Not NANY 2011???
« Last Edit: December 20, 2010, 10:25 PM by Perry Mowbray »

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2010 Proposal - Duplicate Photo Deleter
« Reply #4 on: December 20, 2010, 11:28 PM »
:-[ Not NANY 2011???

Yeah... Can you move the thread?
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Perry Mowbray

  • N.A.N.Y. Organizer
  • Moderator
  • Joined in 2005
  • *****
  • Posts: 1,817
    • View Profile
    • Donate to Member
Re: NANY 2011 Pledge: Duplicate Photo Deleter
« Reply #5 on: December 21, 2010, 11:54 PM »
All done!

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Points: 1
  • Posts: 9,746
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: NANY 2011 Pledge: Duplicate Photo Deleter
« Reply #6 on: December 22, 2010, 12:31 AM »
CARP! I wish I'd known about this yesterday. I just moved 3,000+ photos into the same directory on someone's computer because they were spread all over, and there were plenty of duplicates, but sometimes the filesize didn't match. So instead of going through every one comparing filesize and timestamp, I just had TeraCopy automatically rename the naming conflicts.

Is there any way this app can find duplicates if they don't have exactly the same name?  e.g. PIC-1028.jpg & PIC-1028_2.jpg.

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2011 Pledge: Duplicate Photo Deleter
« Reply #7 on: December 22, 2010, 12:39 AM »
CARP! I wish I'd known about this yesterday. I just moved 3,000+ photos into the same directory on someone's computer because they were spread all over, and there were plenty of duplicates, but sometimes the filesize didn't match. So instead of going through every one comparing filesize and timestamp, I just had TeraCopy automatically rename the naming conflicts.

Is there any way this app can find duplicates if they don't have exactly the same name?  e.g. PIC-1028.jpg & PIC-1028_2.jpg.

At the moment I'm only comparing file sizes. File names are irrelevant and I don't look at them at all.

I plan on adding in better comparisons by loading the actual image then comparing that data.

I won't be doing anything like ratios in image comparison, e.g. these pictures are 80% similar. I just think that's a stupid comparison method that's completely unreliable. e.g. You take a picture of someone you are with against a backdrop  of some nice scenery, then they take your picture. It's a small amount of difference, but it's important. Since the amount of coding to do facial recognition and all that is really far beyond what I'm prepared to put in right now, I'm not even going to try because I don't see any point in doing a crappy job. 

The software right now is stable, but alpha (feature incomplete).
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
Re: NANY 2011 Pledge: Duplicate Photo Deleter
« Reply #8 on: December 22, 2010, 12:02 PM »
Is there any way this app can find duplicates if they don't have exactly the same name?  e.g. PIC-1028.jpg & PIC-1028_2.jpg.

You can easily do this using file checksums.

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Points: 1
  • Posts: 9,746
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: NANY 2011 Pledge: Duplicate Photo Deleter
« Reply #9 on: December 22, 2010, 12:18 PM »
Is there any way this app can find duplicates if they don't have exactly the same name?  e.g. PIC-1028.jpg & PIC-1028_2.jpg.

You can easily do this using file checksums.

How do you easily compare 3,000+ checksums?

skwire

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 5,286
    • View Profile
    • Donate to Member
Re: NANY 2011 Pledge: Duplicate Photo Deleter
« Reply #10 on: December 22, 2010, 12:20 PM »
Compute them, build a list, report duplicates.   :D

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2011 Pledge: Duplicate Photo Deleter
« Reply #11 on: December 30, 2010, 05:48 AM »
<clears throat /> Ahem...

Ooops.

Got a memory issue popping up that affects large images and don't know if I'll be able to solve it in time.

Sorry... I think this is dead.
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Perry Mowbray

  • N.A.N.Y. Organizer
  • Moderator
  • Joined in 2005
  • *****
  • Posts: 1,817
    • View Profile
    • Donate to Member
Re: NANY 2011 Withdrawn: Duplicate Photo Deleter
« Reply #12 on: December 30, 2010, 08:33 PM »
It's never dead until it's dead... but I changed the title to suit  ;)

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2011 Withdrawn: Duplicate Photo Deleter
« Reply #13 on: December 30, 2010, 08:46 PM »
It's never dead until it's dead... but I changed the title to suit  ;)

Still working on it... Managed to hack around one super-nasty issue, but with what I want to do in there, it's tough. (IO is a problem as I'm taking an entirely bulk approach and working with my own photos, which are all 10 megapixels, so it's GB at a time and even with optimized super-fast hashing it's slow.)
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Perry Mowbray

  • N.A.N.Y. Organizer
  • Moderator
  • Joined in 2005
  • *****
  • Posts: 1,817
    • View Profile
    • Donate to Member
Re: NANY 2011 Withdrawn: Duplicate Photo Deleter
« Reply #14 on: December 30, 2010, 08:52 PM »
Still working on it...

 :Thmbsup:  Just change the title and the OP to suit if it gets to Release

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2011 Release: Duplicate Photo Deleter
« Reply #15 on: December 31, 2010, 02:56 AM »
Made it in time. :)

Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Perry Mowbray

  • N.A.N.Y. Organizer
  • Moderator
  • Joined in 2005
  • *****
  • Posts: 1,817
    • View Profile
    • Donate to Member
Re: NANY 2011 Release: Duplicate Photo Deleter
« Reply #16 on: December 31, 2010, 03:05 AM »
Made it in time. :)

 :Thmbsup:

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2011 Release: Duplicate Photo Deleter
« Reply #17 on: December 31, 2010, 11:04 PM »
I've updated the software to v1.1.

Changes:

* Image previews now 25~35% faster
* Image load times displayed
* New JPG icon for images
* Selected folders now highlighted in folder browsers
* Press F1 for a nice "About" screen :)
* Refactored some code
* Added massive amounts of comments to code, complete with stupid humor :p
* Released source code YAY~!

I've commented the source extensively for anyone that wants to have a look or perhaps learn some stuff in C#.

Please note that I've done some very bad things in there (e.g. Application.DoEvents() to maintain a responsive UI). I know that. However, in the interests of getting things done I had to sacrifice in places. e.g. No threading there.

However, there are enough comments there and the really bad things are mostly marked.

Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Perry Mowbray

  • N.A.N.Y. Organizer
  • Moderator
  • Joined in 2005
  • *****
  • Posts: 1,817
    • View Profile
    • Donate to Member
Re: NANY 2011 Release: Duplicate Photo Deleter
« Reply #18 on: January 01, 2011, 08:45 AM »
If you're going to continue adding features... one I'd like to see is the ability to test just the image data, not the whole file (which will include EXIF data too)... once I've geotagged my images it messes the matching up  :(

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2011 Release: Duplicate Photo Deleter
« Reply #19 on: January 01, 2011, 11:04 AM »
If you're going to continue adding features... one I'd like to see is the ability to test just the image data, not the whole file (which will include EXIF data too)... once I've geotagged my images it messes the matching up  :(

The current feature set hashes the image, and not the file.

Here's the source:

Code: C# [Select]
  1. List<UInt32> hash1 = new List<uint>();
  2. SuperFastHashUnsafe sf1 = new SuperFastHashUnsafe();
  3. Bitmap bmp1;
  4. ImageConverter converter = new ImageConverter();
  5.  
  6. ...
  7.  
  8. bmp1 = (Bitmap)Bitmap.FromFile(fi1.FullName);
  9. byte[] bytes = new byte[1];
  10. bytes = (byte[])converter.ConvertTo(bmp1, bytes.GetType());
  11.  
  12. // Add the has value to the list.
  13. hash1.Add(sf1.Hash(bytes));

And the relevant MSDN article:

http://msdn.microsof....imageconverter.aspx

If I'm not mistaken, only the pixel data is considered, which results in a safe comparison.

I COULD be mistaken though.

If you could send me 2 files to test, I'd appreciate it.



Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2011 Release: Duplicate Photo Deleter
« Reply #20 on: January 01, 2011, 11:05 AM »
Oh, I should mention that's for the second comparison method. The first is file size only. Check the top combobox there.
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Perry Mowbray

  • N.A.N.Y. Organizer
  • Moderator
  • Joined in 2005
  • *****
  • Posts: 1,817
    • View Profile
    • Donate to Member
Re: NANY 2011 Release: Duplicate Photo Deleter
« Reply #21 on: January 01, 2011, 11:08 AM »
I thought I did it right... but didn't match (can you remind me later for the files?)

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2011 Release: Duplicate Photo Deleter
« Reply #22 on: January 01, 2011, 12:20 PM »
I thought I did it right... but didn't match (can you remind me later for the files?)

It matches folders only. Not individual images... Design flaw considering the UI. I'll need to update that. But I will remind you tomorrow to send me the pics.
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: NANY 2011 Release: Duplicate Photo Deleter
« Reply #23 on: January 01, 2011, 05:17 PM »

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: NANY 2011 Release: Duplicate Photo Deleter
« Reply #24 on: January 01, 2011, 09:29 PM »
Written up on ghacks :Thmbsup:

http://www.ghacks.ne...licate-photo-finder/

Cool! That's a pleasant surprise~! :D
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker