Home | Blog | Software | Reviews and Features | Forum | Help | Donate | About us
topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • December 09, 2016, 01:27:57 PM
  • Proudly celebrating 10 years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: photo duplicate scanner  (Read 60190 times)

Jammo the OrganizedFellow

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 212
    • View Profile
    • OrganizedFellow
    • Donate to Member
photo duplicate scanner
« on: January 14, 2008, 11:46:50 AM »
Between my wife and I, we take lots of pics using our nice little point-n-shoot Panasonic Lumix.
Living in a new city, new friends, many new sites to see, and a new baby girl (5 months). So about once a week we transfer all our photos and many .MOV files from our SD cards to our desktop.

Usually, files would retain the cameras default naming system. EX: P1010368.jpg, or P1000952.mov.
But for the past few weeks, my naming system is messed up. Now I have dozens, if not hundreds of hi-res photos that are duplicates (average size of 2-3MB per photo).

I'm not sure if this is something caused by my constant app install/uninstalls or not, but I think it can be solved.
Rather than manually scan through my photos, I recall having seen or heard of an application that does it automagically? and removes/moves/renames/copies duplicate photos?

If there is an app like this, what are its features?!
hhmm ... I just came from this thread. But it didn't offer anything I need.
As an aspiring web developer/designer, it is a constant struggle to cope with my ADHD + Hypomania/Bipolar Disorder.

The slow growth of my web dev projects is eclipsed by my patience, understanding and desire to learn AS MUCH AS POSSIBLE as I slowly progress.

X_____jamjammo_____

tomos

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 10,341
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #1 on: January 14, 2008, 12:08:04 PM »
I used to have one called similar images finder but I never used it :-\
so I googled -
similar images finder

one out of the first three listed is free, one has a trial, I didnt look further

if you do try any of them let us know how you get on!
Tom

jgpaiva

  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 4,727
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #2 on: January 14, 2008, 12:11:40 PM »
From what i understand, you don't need a "similar images finder", all you need is a "duplicate file finder", which is included in serveral alternative file explorers (like dopus) or, alternativelly, you could use a standalone tool, like this one.

disclaimer: i've never used dupfilefinder. It does look good, though :P

tomos

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 10,341
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #3 on: January 14, 2008, 12:14:41 PM »
From what i understand, you don't need a "similar images finder", all you need is a "duplicate file finder", which is included in serveral alternative file explorers (like dopus) or, alternativelly, you could use a standalone tool, like this one.

disclaimer: i've never used dupfilefinder. It does look good, though :P

would that find them if they have different names - dont think dopus would do that
would have to compare all files for same size and then do a bit comparision
Tom

iphigenie

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 1,169
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #4 on: January 14, 2008, 12:15:34 PM »
the ones that deal with images specifically have the nice feature of showing potential matches side by side, with a quick "yes/no" way to cycle and delete them.

Comparison usually starts with size, meta information (image info) then binary checksum

iphigenie

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 1,169
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #5 on: January 14, 2008, 12:17:01 PM »
I think you could also use a rename-images-based-on-meta-information feature, i.e. use exif date/time to organise all the images, making spotting duplicates easier.

nosh

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 1,426
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #6 on: January 14, 2008, 01:30:25 PM »
Easy Duplicate Finder is free and it's also the fastest dupe finder I've used. From the info you provided it seems perfect for the job.
http://www.easyduplicatefinder.com

DoubleKiller is a more advanced dupe finder
http://www.bigbangen....de/en/doublekiller/

If you're looking for an image dupe finder, something that finds dupes even if pics have been resized, try Unique Filer
http://www.uniquefiler.com/



Falcon_four

  • Supporting Member
  • Joined in 2007
  • **
  • default avatar
  • Posts: 9
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #7 on: January 14, 2008, 01:51:52 PM »
One I've found, although it's not free: PhotoSort <http://www.asmdev.ne.../products/photosort/>

I use this myself and it is quite good - it has support for various RAW images and is quite easy to use.

>>>>F-4>>>>

>>>>F-4>>>>

katykaty

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 221
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #8 on: January 14, 2008, 07:31:10 PM »
I use Duplicate File Finder by Rashid Hoda which has plenty of options and will find duplicates even if they have different names and/or are in different folders.

See screengrab attached.

Development seems to have ceased though, his website is down and has been for months, and it's tricky to find on the web.

I found a download link to version 1.1.0.2 - I use 1.1.0.3

It is/was freeware but I don't think it was open source - I wonder when it starts to count as abandonware?

I could post the installer but I'd probably be in breach of the license, even if it was freeware and the developer awol.

PhilB66

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 1,522
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #9 on: January 14, 2008, 07:47:00 PM »
Duplicate File Finder (Freeware, Version 1.1.0.3, Direct Download). Found on this page http://web.archive.o...//dff.nazrashid.com/

Kimchee53

  • Participant
  • Joined in 2006
  • *
  • default avatar
  • Posts: 4
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #10 on: January 14, 2008, 09:35:28 PM »
D'Peg - by somewareonthe.net - I think. Have used D'Peg for many years, and it has improved continually. It now also handles movies & sound files. IMHO it has about a 98% acurracy (did I spell that right?) rate. It's a bit tricky to get the settings for what you want it to do (on first try), but author is VERY helpful (as is the help built in). Price is pretty reasonable, and sometimes you'll get another program as a "gift" from the author (not always).

Hope this helps a bit.

http://www.somewareonthe.net/dpeg.htm

link to site is above.

Jammo the OrganizedFellow

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 212
    • View Profile
    • OrganizedFellow
    • Donate to Member
Re: photo duplicate scanner
« Reply #11 on: January 14, 2008, 10:36:51 PM »
Thanks for the many many suggestions everyone!
I'll give them each a try, and see where I get!
As an aspiring web developer/designer, it is a constant struggle to cope with my ADHD + Hypomania/Bipolar Disorder.

The slow growth of my web dev projects is eclipsed by my patience, understanding and desire to learn AS MUCH AS POSSIBLE as I slowly progress.

X_____jamjammo_____

mwb1100

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 1,522
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #12 on: January 15, 2008, 01:20:18 PM »
In a few days Bits du Jour will be having an offer on some software called "Image Comparer" which claims to be able to find similar photos - not just exact file copies.  I have no idea how well it works, I only noticed the advertisement:

http://www.bitsdujou....aspx%3freview%3d161

So for anyone who is looking to cull out a photo library, this might be an interesting bit of software to test.

MikeMcLoughlin

  • Supporting Member
  • Joined in 2007
  • **
  • default avatar
  • Posts: 31
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #13 on: January 16, 2008, 05:41:36 AM »
jammo

I've tried many and the best one I've come across is DupDetector, and it's free.

It was/is from a company called Prismatic Software

http://www.prismaticsoftware.com

Although it looks like they sadly no longer exist.

It's worth Googling to see if you can find it elsewhere or I'll gladly email it to you.

It's not very pretty to look at but works really well.

PhilB66

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 1,522
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #14 on: January 16, 2008, 07:11:26 AM »
jammo

I've tried many and the best one I've come across is DupDetector, and it's free.

It was/is from a company called Prismatic Software

http://www.prismaticsoftware.com

Although it looks like they sadly no longer exist.

It's worth Googling to see if you can find it elsewhere or I'll gladly email it to you.

It's not very pretty to look at but works really well.

Dup Detector freeware version 3.201 still available via this link:

http://web.archive.o...Utility/Utility.html

Here's a screenshot and a brief description:

2008-01-16_211516.jpg

"Dup Detector finds duplicate and near duplicate images by opening and reading image pixel data. Large collections and large image file sizes can slow Dup Detector but it works well running in the background. Large image collections often are filled with duplicate images under different names. Use Dup Detector to clean them. DupDetector reads 9 image file types from a folder or its subfolders or from an image list."
« Last Edit: January 16, 2008, 07:17:43 AM by PhilB66 »

J-Mac

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 2,913
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #15 on: January 17, 2008, 03:04:51 AM »
I use Duplicate Finder by Ashisoft Software.  Very nice - I've used it for a few years now.  Uses several different methods to search for duplicates and lets you view them in thumbnails to compare.

Jim

lanux128

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 6,258
    • View Profile
    • Coding Snacks by Lanux128
    • Read more about this member.
    • Donate to Member
Re: photo duplicate scanner
« Reply #16 on: January 19, 2008, 12:37:18 PM »
yet another tool for this task, Finddupe from the makers of Jhead.

Finddupe is a tool for quick detecting of duplicate files on a hard drive under Windows. Duplicate files can be just detected, hardlinked, or deleted.

Finddupe has several possible uses:

Deleting duplicate files
When working thru somebody else's photo collection, or MP3 collection, this tool is useful for deleting the files that are duplicate. Depending on how the media is organized, there can be a lot of duplicate files in a collection.
http://www.sentex.net/~mwandel/finddupe/

Jammo the OrganizedFellow

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 212
    • View Profile
    • OrganizedFellow
    • Donate to Member
Re: photo duplicate scanner
« Reply #17 on: January 19, 2008, 10:28:06 PM »
"Dup Detector finds duplicate and near duplicate images by opening and reading image pixel data. Large collections and large image file sizes can slow Dup Detector but it works well running in the background. Large image collections often are filled with duplicate images under different names. Use Dup Detector to clean them. DupDetector reads 9 image file types from a folder or its subfolders or from an image list."
THANKS ALL!

This is the one I chose to use for my needs.
I really liked how it displays the 'suspected duplicate' next to the original.

Now I just need to find a method to RENAME all my files by the creation date. Something that reads the EXIF data?
I hate DOpus, if anyone knows how. I use DOpus as a newbie, haha, I know it has super powerful features. I'm reading up on it right now, see if that's something it can do for me.
As an aspiring web developer/designer, it is a constant struggle to cope with my ADHD + Hypomania/Bipolar Disorder.

The slow growth of my web dev projects is eclipsed by my patience, understanding and desire to learn AS MUCH AS POSSIBLE as I slowly progress.

X_____jamjammo_____

Jammo the OrganizedFellow

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 212
    • View Profile
    • OrganizedFellow
    • Donate to Member
Re: photo duplicate scanner
« Reply #18 on: January 19, 2008, 11:06:21 PM »
*AMEND TO ABOVE*

After a ONE MINUTE SEARCH, hahaha, silly me, right?
I went to LifeHacker and found this article Namexif Batch Renames Digital Photos by Date

But as always, I read the comments and came across:  JoeJoe's freeware utilities, Rename Master.
It worked like a charm.

I didn't want JUST a renamer for EXIF data, but an all-around EVERYthing renamer, and this fits the ticket!
:)
As an aspiring web developer/designer, it is a constant struggle to cope with my ADHD + Hypomania/Bipolar Disorder.

The slow growth of my web dev projects is eclipsed by my patience, understanding and desire to learn AS MUCH AS POSSIBLE as I slowly progress.

X_____jamjammo_____

tomos

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 10,341
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #19 on: January 20, 2008, 04:04:54 AM »
Now I just need to find a method to RENAME all my files by the creation date. Something that reads the EXIF data?
I hate DOpus, if anyone knows how. I use DOpus as a newbie, haha, I know it has super powerful features. I'm reading up on it right now, see if that's something it can do for me.

jammo
[edit] got my wires crossed - see edit below [/edit]
have a look at this (but read following post too - small bug here but I would still recommend it)

here are two more tools that will be useful in this task.

http://www.sentex.net/~mwandel/jhead/
http://jpgtime.learsy.com/
that Learsy one is a thing of beauty :Thmbsup:
 (see attachment in previous post)
# no install
# simple to use -
  • select files (all if you want) / click Set time from EXIF - it show the Exif time in the "New Time" column
  • compare
  • select those you want to change
  • click Apply changes

then go off and donate to the author (donationware :up:)
http://www.muralpix.com/donation.html

BIG EDIT:- :-[ this one here changes modified date back to creation/shooting date -
doesnt change the name - sorry bout that confusion :)
Tom
« Last Edit: January 26, 2008, 03:23:16 AM by tomos »

PhilB66

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 1,522
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #20 on: January 20, 2008, 06:12:13 AM »
Now I just need to find a method to RENAME all my files by the creation date. Something that reads the EXIF data?

AmoK Exif Sorter

Features:

+ Fast renaming of pictures based Exif- and IPTC data
+ Support of all Exif / IPTC data ( date picture taken, year, month, week, day, hour, second, size of photo, camera model, camera manufacturer, exposure, aperture, original filename, description, place, keywords, and many more)
+ Easy definition of rules for renaming via Tags (%year%, %month%, %day%, ...)
+ Photos may be renamed, moved or copied
+ Preview of new filenames in real-time
+ Assistant for renaming photos
+ Processing of single files or directories including sub directories
+ Integrated picture viewer
+ Handles thousand of pictures at once
+ Drag & Drop support
+ Automatic update check
+ Profiles for different cameras, users and renaming rules
+ 40% faster than AmoK Exif Sorter 1.x
+ Independent of operating system (runs on Windows, Linux and Mac)
+ Free English support via email
+ 100% free for private and commercial use
+ With and without setup/installer available

Screenshots

Jammo the OrganizedFellow

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 212
    • View Profile
    • OrganizedFellow
    • Donate to Member
Re: photo duplicate scanner
« Reply #21 on: January 20, 2008, 11:41:48 AM »
So far, I'm sticking to using JoeJoes Renamer for everyday usage. It works well with my methods of categorizing my photos.

But now, I've got a new dilemma: my camera outputs video files as *.mov
And as pointed out by my original post, the file names are all wonky. Is there any way to batch rename all based on creation date? There is no exif type data on these.
As an aspiring web developer/designer, it is a constant struggle to cope with my ADHD + Hypomania/Bipolar Disorder.

The slow growth of my web dev projects is eclipsed by my patience, understanding and desire to learn AS MUCH AS POSSIBLE as I slowly progress.

X_____jamjammo_____

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,475
    • View Profile
    • Donate to Member
Re: photo duplicate scanner
« Reply #22 on: January 23, 2008, 03:17:44 AM »
But now, I've got a new dilemma: my camera outputs video files as *.mov
And as pointed out by my original post, the file names are all wonky. Is there any way to batch rename all based on creation date? There is no exif type data on these.

You mentioned DOpus above, so I'll assume you have it.

The easy way:

1) Select files/movies/etc.
2) Hit the Rename button.
3) Hit the Advanced button - assuming it's in Simple mode.
4) Tick 'Enable file information fields' - you'll now notice a selection button on the right side
    of the 'New name' input field.
5) Hit that selection button, (a down arrow), and select 'General->Date (Created)'.
6) The new filenames will be displayed in the preview area - hit the OK button.
7) Job Done.

The 'Enable file information fields' also gives you access to a lot more renaming info, including EXIF data.

Cheers,
4wd (ex Amiga DOpus beta-tester :)
« Last Edit: August 10, 2010, 07:31:09 PM by 4wd »

brahman

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 171
    • View Profile
    • Donate to Member
Re: file duplicate scanner
« Reply #23 on: January 26, 2008, 12:14:31 AM »
Hello,

my problem is similar to the threadstarter, yet needs a very different approach:

I often rename video etc. files and then still have the unrenamed files on my backup drive.

I use SynchronizeIt! (great app from grigsoft.com - highly recommended) to keep my backups in synch.

However, after my renaming, it wants to copy the entire file and delete the same unrenamed file on the backup volume, because it cannot detect that I renamed the file. This can be very lengthy on gigabyte sized files.

So now I am searching for an app, which does not only find dupes and deletes them, but which I can tell: compare these two folders, the first folder is source, the second target, files have different names but same time and size (so no need to do lengthy byte by byte or checksum compare) and rename all same files from target name to source name.

Has anybody found something like this?

Regards,

Brahman
Regards, Brahman

Jammo the OrganizedFellow

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 212
    • View Profile
    • OrganizedFellow
    • Donate to Member
Re: photo duplicate scanner
« Reply #24 on: January 26, 2008, 01:55:14 AM »
You mentioned DOpus above, so I'll assume you have it.

The easy way:

1-7
:)
THanks for that info.
I tried that too. That will work well for further organizing of my files.
Might even eliminate my file renamer that I downloaded from JoeJoe software!
As an aspiring web developer/designer, it is a constant struggle to cope with my ADHD + Hypomania/Bipolar Disorder.

The slow growth of my web dev projects is eclipsed by my patience, understanding and desire to learn AS MUCH AS POSSIBLE as I slowly progress.

X_____jamjammo_____