photo duplicate scanner

[Jammo the OrganizedFellow]

Between my wife and I, we take lots of pics using our nice little point-n-shoot Panasonic Lumix.
Living in a new city, new friends, many new sites to see, and a new baby girl (5 months). So about once a week we transfer all our photos and many .MOV files from our SD cards to our desktop.

Usually, files would retain the cameras default naming system. EX: P1010368.jpg, or P1000952.mov.
But for the past few weeks, my naming system is messed up. Now I have dozens, if not hundreds of hi-res photos that are duplicates (average size of 2-3MB per photo).

I'm not sure if this is something caused by my constant app install/uninstalls or not, but I think it can be solved.
Rather than manually scan through my photos, I recall having seen or heard of an application that does it automagically? and removes/moves/renames/copies duplicate photos?

If there is an app like this, what are its features?!
hhmm ... I just came from this thread. But it didn't offer anything I need.

[tomos]

I used to have one called similar images finder but I never used it
so I googled -
similar images finder

one out of the first three listed is free, one has a trial, I didnt look further

if you do try any of them let us know how you get on!

[jgpaiva]

From what i understand, you don't need a "similar images finder", all you need is a "duplicate file finder", which is included in serveral alternative file explorers (like dopus) or, alternativelly, you could use a standalone tool, like this one.

disclaimer: i've never used dupfilefinder. It does look good, though

[tomos]

From what i understand, you don't need a "similar images finder", all you need is a "duplicate file finder", which is included in serveral alternative file explorers (like dopus) or, alternativelly, you could use a standalone tool, like this one.

disclaimer: i've never used dupfilefinder. It does look good, though

-jgpaiva (January 14, 2008, 12:11 PM)

would that find them if they have different names - dont think dopus would do that
would have to compare all files for same size and then do a bit comparision

[iphigenie]

the ones that deal with images specifically have the nice feature of showing potential matches side by side, with a quick "yes/no" way to cycle and delete them.

Comparison usually starts with size, meta information (image info) then binary checksum

[iphigenie]

I think you could also use a rename-images-based-on-meta-information feature, i.e. use exif date/time to organise all the images, making spotting duplicates easier.

[nosh]

Easy Duplicate Finder is free and it's also the fastest dupe finder I've used. From the info you provided it seems perfect for the job.
http://www.easyduplicatefinder.com

DoubleKiller is a more advanced dupe finder
http://www.bigbangen....de/en/doublekiller/

If you're looking for an image dupe finder, something that finds dupes even if pics have been resized, try Unique Filer
http://www.uniquefiler.com/

[Falcon_four]

One I've found, although it's not free: PhotoSort <http://www.asmdev.ne.../products/photosort/>

I use this myself and it is quite good - it has support for various RAW images and is quite easy to use.

>>>>F-4>>>>

[katykaty]

I use Duplicate File Finder by Rashid Hoda which has plenty of options and will find duplicates even if they have different names and/or are in different folders.

See screengrab attached.

Development seems to have ceased though, his website is down and has been for months, and it's tricky to find on the web.

I found a download link to version 1.1.0.2 - I use 1.1.0.3

It is/was freeware but I don't think it was open source - I wonder when it starts to count as abandonware?

I could post the installer but I'd probably be in breach of the license, even if it was freeware and the developer awol.

[PhilB66]

Duplicate File Finder (Freeware, Version 1.1.0.3, Direct Download). Found on this page http://web.archive.o...//dff.nazrashid.com/

[Kimchee53]

D'Peg - by somewareonthe.net - I think. Have used D'Peg for many years, and it has improved continually. It now also handles movies & sound files. IMHO it has about a 98% acurracy (did I spell that right?) rate. It's a bit tricky to get the settings for what you want it to do (on first try), but author is VERY helpful (as is the help built in). Price is pretty reasonable, and sometimes you'll get another program as a "gift" from the author (not always).

Hope this helps a bit.

http://www.somewareonthe.net/dpeg.htm

link to site is above.

[Jammo the OrganizedFellow]

Thanks for the many many suggestions everyone!
I'll give them each a try, and see where I get!

[mwb1100]

In a few days Bits du Jour will be having an offer on some software called "Image Comparer" which claims to be able to find similar photos - not just exact file copies.  I have no idea how well it works, I only noticed the advertisement:

http://www.bitsdujou....aspx%3freview%3d161

So for anyone who is looking to cull out a photo library, this might be an interesting bit of software to test.

[MikeMcLoughlin]

jammo

I've tried many and the best one I've come across is DupDetector, and it's free.

It was/is from a company called Prismatic Software

http://www.prismaticsoftware.com

Although it looks like they sadly no longer exist.

It's worth Googling to see if you can find it elsewhere or I'll gladly email it to you.

It's not very pretty to look at but works really well.

[PhilB66]

jammo

I've tried many and the best one I've come across is DupDetector, and it's free.

It was/is from a company called Prismatic Software

http://www.prismaticsoftware.com

Although it looks like they sadly no longer exist.

It's worth Googling to see if you can find it elsewhere or I'll gladly email it to you.

It's not very pretty to look at but works really well.

-MikeMcLoughlin (January 16, 2008, 05:41 AM)

Dup Detector freeware version 3.201 still available via this link:

http://web.archive.o...Utility/Utility.html

Here's a screenshot and a brief description:



"Dup Detector finds duplicate and near duplicate images by opening and reading image pixel data. Large collections and large image file sizes can slow Dup Detector but it works well running in the background. Large image collections often are filled with duplicate images under different names. Use Dup Detector to clean them. DupDetector reads 9 image file types from a folder or its subfolders or from an image list."

[J-Mac]

I use Duplicate Finder by Ashisoft Software.  Very nice - I've used it for a few years now.  Uses several different methods to search for duplicates and lets you view them in thumbnails to compare.

Jim

[lanux128]

yet another tool for this task, Finddupe from the makers of Jhead.

Finddupe is a tool for quick detecting of duplicate files on a hard drive under Windows. Duplicate files can be just detected, hardlinked, or deleted. Finddupe has several possible uses: Deleting duplicate files When working thru somebody else's photo collection, or MP3 collection, this tool is useful for deleting the files that are duplicate. Depending on how the media is organized, there can be a lot of duplicate files in a collection.

• http://www.sentex.net/~mwandel/finddupe/

[Jammo the OrganizedFellow]

"Dup Detector finds duplicate and near duplicate images by opening and reading image pixel data. Large collections and large image file sizes can slow Dup Detector but it works well running in the background. Large image collections often are filled with duplicate images under different names. Use Dup Detector to clean them. DupDetector reads 9 image file types from a folder or its subfolders or from an image list."

-PhilB66 (January 16, 2008, 07:11 AM)

THANKS ALL!

This is the one I chose to use for my needs.
I really liked how it displays the 'suspected duplicate' next to the original.

Now I just need to find a method to RENAME all my files by the creation date. Something that reads the EXIF data?
I hate DOpus, if anyone knows how. I use DOpus as a newbie, haha, I know it has super powerful features. I'm reading up on it right now, see if that's something it can do for me.

[Jammo the OrganizedFellow]

*AMEND TO ABOVE*

After a ONE MINUTE SEARCH, hahaha, silly me, right?
I went to LifeHacker and found this article Namexif Batch Renames Digital Photos by Date

But as always, I read the comments and came across:  JoeJoe's freeware utilities, Rename Master.
It worked like a charm.

I didn't want JUST a renamer for EXIF data, but an all-around EVERYthing renamer, and this fits the ticket!

[tomos]

Now I just need to find a method to RENAME all my files by the creation date. Something that reads the EXIF data?
I hate DOpus, if anyone knows how. I use DOpus as a newbie, haha, I know it has super powerful features. I'm reading up on it right now, see if that's something it can do for me.

-jammo (January 19, 2008, 10:28 PM)

jammo
[edit] got my wires crossed - see edit below [/edit]
have a look at this (but read following post too - small bug here but I would still recommend it)

here are two more tools that will be useful in this task.

• http://www.sentex.net/~mwandel/jhead/
• http://jpgtime.learsy.com/

-lanux128 (January 19, 2008, 12:30 PM)

that Learsy one is a thing of beauty
 (see attachment in previous post)
# no install
# simple to use -

• select files (all if you want) / click Set time from EXIF - it show the Exif time in the "New Time" column
• compare
• select those you want to change
• click Apply changes

then go off and donate to the author (donationware )
http://www.muralpix.com/donation.html

-tomos (January 19, 2008, 01:08 PM)

BIG EDIT:- this one here changes modified date back to creation/shooting date -
doesnt change the name - sorry bout that confusion

[PhilB66]

Now I just need to find a method to RENAME all my files by the creation date. Something that reads the EXIF data?

-jammo (January 19, 2008, 10:28 PM)

AmoK Exif Sorter

Features:

+ Fast renaming of pictures based Exif- and IPTC data
+ Support of all Exif / IPTC data ( date picture taken, year, month, week, day, hour, second, size of photo, camera model, camera manufacturer, exposure, aperture, original filename, description, place, keywords, and many more)
+ Easy definition of rules for renaming via Tags (%year%, %month%, %day%, ...)
+ Photos may be renamed, moved or copied
+ Preview of new filenames in real-time
+ Assistant for renaming photos
+ Processing of single files or directories including sub directories
+ Integrated picture viewer
+ Handles thousand of pictures at once
+ Drag & Drop support
+ Automatic update check
+ Profiles for different cameras, users and renaming rules
+ 40% faster than AmoK Exif Sorter 1.x
+ Independent of operating system (runs on Windows, Linux and Mac)
+ Free English support via email
+ 100% free for private and commercial use
+ With and without setup/installer available

Screenshots

[Jammo the OrganizedFellow]

So far, I'm sticking to using JoeJoes Renamer for everyday usage. It works well with my methods of categorizing my photos.

But now, I've got a new dilemma: my camera outputs video files as *.mov
And as pointed out by my original post, the file names are all wonky. Is there any way to batch rename all based on creation date? There is no exif type data on these.

[4wd]

But now, I've got a new dilemma: my camera outputs video files as *.mov
And as pointed out by my original post, the file names are all wonky. Is there any way to batch rename all based on creation date? There is no exif type data on these.

-jammo (January 20, 2008, 11:41 AM)

You mentioned DOpus above, so I'll assume you have it.

The easy way:

1) Select files/movies/etc.
2) Hit the Rename button.
3) Hit the Advanced button - assuming it's in Simple mode.
4) Tick 'Enable file information fields' - you'll now notice a selection button on the right side
    of the 'New name' input field.
5) Hit that selection button, (a down arrow), and select 'General->Date (Created)'.
6) The new filenames will be displayed in the preview area - hit the OK button.
7) Job Done.

The 'Enable file information fields' also gives you access to a lot more renaming info, including EXIF data.

Cheers,
4wd (ex Amiga DOpus beta-tester

[brahman]

Hello,

my problem is similar to the threadstarter, yet needs a very different approach:

I often rename video etc. files and then still have the unrenamed files on my backup drive.

I use SynchronizeIt! (great app from grigsoft.com - highly recommended) to keep my backups in synch.

However, after my renaming, it wants to copy the entire file and delete the same unrenamed file on the backup volume, because it cannot detect that I renamed the file. This can be very lengthy on gigabyte sized files.

So now I am searching for an app, which does not only find dupes and deletes them, but which I can tell: compare these two folders, the first folder is source, the second target, files have different names but same time and size (so no need to do lengthy byte by byte or checksum compare) and rename all same files from target name to source name.

Has anybody found something like this?

Regards,

Brahman

[Jammo the OrganizedFellow]

You mentioned DOpus above, so I'll assume you have it.

The easy way:

1-7

-4wd (January 23, 2008, 03:17 AM)

THanks for that info.
I tried that too. That will work well for further organizing of my files.
Might even eliminate my file renamer that I downloaded from JoeJoe software!