Author Topic: New duplicate file finder... I am open to features suggestion (Read 39647 times)

AsmDev · « **on:** December 04, 2008, 03:59 PM »

First of all let me say hello to all users in this great community of which I am shamefully not a very active member. Some of you I already know as customers from the support mail exchange and there are also other guys like f0dder, who I know from other software developing forums.

My name is Milos and I am the author of DupeTrasher (http://www.asmdev.ne...products/dupetrasher).

I would like to announce the development of the new version of this application which will be, as I hope, the next association for the software for removing duplicate files. First version had some success but there is still much room for improvement. I carefully noted user requests and along with some new ideas of mine and with new Vista technology it is almost done.
I will be giving some free license codes for people in this community and also discounts once it is realised.

I'd like to ask you guys what are the common scenarios you have when dealing with duplicate files (beside searching for all drives). For example one feature that I implemented is to find all duplicate files in hard drive which are already available in CD/DVD. After search is done all dupes in located in hard drive will be automatically marked for removal. Then, user can review the selection and proceed with deleting if needed. This saves user's time and does the job with minimal effort.
So this is just one common scenario that crossed my mind, do you have other similar? I will gladly implement them if they seem to be useful.
Also feel free to write any other feature you think would be useful in an application like this.

Thanks

Darwin · « **Reply #1 on:** December 04, 2008, 04:59 PM »

For me, the biggest issue when using Duplicate finders is VISIBILITY! I haven't tried version one of your application, so I apologize if this is already implemented in it, but I'd like results to be returned "nested" like this:

1. File name
----------->Location One
----------->Location Two
Etc.

I'd also like to see some sort of colour banding utilized so that differentiating between files is easier. Finally, I'd like results to be fully sortable - name, date, type, size, etc.

That's it off the top of my head - I'll post "deeper" thoughts if they occur to me after thinking about this in more depth

Kamel · « **Reply #2 on:** December 04, 2008, 05:07 PM »

I don't know how much I can help, but I'd be glad to beta test etc if you like me to.

I have a problem with duplicate files myself due to having a large network of computers and constantly backing up and reformatting machines etc. Sometimes I can get lots of residue of older files built up that is hard to wade through.

Let me know if you're interested. Just a heads up though, i seriously doubt I will buy your product

. I am under very heavy financial stress being out of a job and all.

city_zen · « **Reply #3 on:** December 04, 2008, 11:16 PM »

AsmDev

I'm not a frequent user of duplicate finders, but here are a few suggestions about features that I'd consider useful in such a program. Maybe some of then are obvious/standard for software in this category and some others are already implemented in your program in particular (sorry, I haven't tried it yet). And Darwin has already made a couple of very good suggestions. Nevertheless, here they are:

- Possibility of detailed selection of drives and folders to scan/watch
- Possibility of several "levels" of similarity between files. For example: a) exactly the same, b) same name, different size, c) similar name, same size, same date, and so on
- Ability to set that the older files in a pair/group are the ones marked for deletion by default
- Ability to select only certain file types to be scanned (i.e. only music files, only video files, etc.)

I hope it helps

mouser · « **Reply #4 on:** December 04, 2008, 11:45 PM »

city_zen's suggestions are all good.
i also always like various options for setting how the program should select the default one to keep.

rjbull · « **Reply #5 on:** December 05, 2008, 05:16 AM »

I'd like results to be returned "nested" like this:

1. File name
----------->Location One
----------->Location Two
Etc.

I'd also like to see some sort of colour banding utilized so that differentiating between files is easier.
-Darwin (December 04, 2008, 04:59 PM)

You mean like DoubleKiller

rjbull · « **Reply #6 on:** December 05, 2008, 05:18 AM »

- Possibility of several "levels" of similarity between files. For example: a) exactly the same, b) same name, different size, c) similar name, same size, same date, and so on
-city_zen (December 04, 2008, 11:16 PM)

Add to that, same root name but different extension, so you can flag e.g. ZIP and RAR archives that are really the same.

Curt · « **Reply #7 on:** December 05, 2008, 06:56 AM »

1. File name
----------->Location One
----------->Location Two
Etc.

-Darwin (December 04, 2008, 04:59 PM)

- maybe, but I would rather ask for two columns:

(clickable thumb):

New duplicate file finder... I am open to features suggestion

It is important that the files are sorted according to their location, as in the photo, because in this example it was possible for me to just click once in the top of one of the columns and mass-choose doublets from one and the same folder only.

Darwin · « **Reply #8 on:** December 05, 2008, 08:27 AM »

I'd like results to be returned "nested" like this:

1. File name
----------->Location One
----------->Location Two
Etc.

I'd also like to see some sort of colour banding utilized so that differentiating between files is easier.
-Darwin (December 04, 2008, 04:59 PM)

You mean like DoubleKiller

-rjbull (December 05, 2008, 05:16 AM)

Yes - but I'd like the format to be as I typed it in above, ie with the filename presented as a header and the duplicate locations listed benath it, indented. It would be nice to have checkboxes next to the file location, too. If DoubleKiller Pro can already do this, I need to go back and RTFM

AsmDev · « **Reply #9 on:** December 05, 2008, 08:29 AM »

Hey guys,

Darwin, all your ideas have already been implemented. Here is the screenshoot at current stage:

http://img68.imageshack.us/img68/2169/duplicate1oo6.png

New duplicate file finder... I am open to features suggestion

Kamel, I'll let you know once beta version is ready for testing, thanks!

1. Possibility of detailed selection of drives and folders to scan/watch - Implemented
2. Possibility of several "levels" of similarity between files. For example: a) exactly the same, b) same name, different size, c) similar name, same size, same date, and so on - Implemented a) and b), as for c) it will be left for future version as I need to tune the algorithm for detecting similarities between file names
3. Ability to set that the older files in a pair/group are the ones marked for deletion by default - Implemented
4. Ability to select only certain file types to be scanned (i.e. only music files, only video files, etc.) - Implemented
-city_zen (December 04, 2008, 11:16 PM)

Btw, about point 3. here, I was thinking more about deleting all files in group and leaving just the oldest by default since it is most probable that the oldest file is original and the newer files were copied later on. But in any case I think I will add an option for user to select what file is to be considered original "the oldest" or "the newest".

city_zen's suggestions are all good.
i also always like various options for setting how the program should select the default one to keep.
-mouser (December 04, 2008, 11:45 PM)

So beside city_zen's suggestions do you have any other ideas on how to help the software to best determine what to select?
Btw there will be option to select all dupe files in specific folder, here is the screenshot:

http://img146.imageshack.us/img146/1298/duplicate2nd5.png

New duplicate file finder... I am open to features suggestion

Here you can click the the checkbox near each folder and all duplicate files (not regular files) will be automatically selected.

- Possibility of several "levels" of similarity between files. For example: a) exactly the same, b) same name, different size, c) similar name, same size, same date, and so on
-city_zen (December 04, 2008, 11:16 PM)
Add to that, same root name but different extension, so you can flag e.g. ZIP and RAR archives that are really the same.
-rjbull (December 05, 2008, 05:18 AM)

So you are basically saying app should be able to detect 2 different archives with same content? Ok thats interesting idea, I will see what can I do about it.

Curt, if I understand right that photo is from app for duplicate pictures right? Well that kind of software is somehow different that regular duplicate file finders because photos with completely different name and content (eg. jpg, bmp) can be the same.
Also about columns, you said you'd like to see two columns but what if there are group of duplicates that have 3 or more same files? Well in that case I think a large number of columns would decrease minuteness.
Sorting is implemented of course, that is a must in any data handling software today.

tsaint · « **Reply #10 on:** December 05, 2008, 08:38 AM »

Thinking pairs of duplicates is a worry....
If you have duplicates A, B and C and matches are presented in pairs, it is possible to get this:
Match A B ..... delete A
other matches .....
Match B C ..... delete B
other matches ....
Match C A ..... delete C
End result = no duplicates, but no single copy either.

If these pairings are seperated by several other matches it's too easy to lose track of what you're doing. To my mind much better to see a list with, for example, check boxes, with all matching duplicates grouped together

Darwin · « **Reply #11 on:** December 05, 2008, 09:01 AM »

If these pairings are seperated by several other matches it's too easy to lose track of what you're doing. To my mind much better to see a list with, for example, check boxes, with all matching duplicates grouped together
-tsaint (December 05, 2008, 08:38 AM)

This is what I'm suggesting as well - unfortunate that chose to limit my example to two files.

File Name
-------->Location One
-------->Location Two
-------->Location Three
-------->Location Four
...........Etc.

If you look at the first group in Vitaly's screen shot, it looks like he's already implemented this. Of course, it's 7 am here and I've only had one cup of coffee, so...

BTW, I've been reading through TFM for DoubleKiller Pro and I can't see how to do this there. Caffeine caveat applies here, too, though...

PS WRT file grouping, it's the file name as group header that I'm looking for (in DoubleKiller)

tsaint · « **Reply #12 on:** December 05, 2008, 09:05 AM »

Sorry Darwin, I meant to say I supported your idea but somehow I left that bit out. My apologies.

Crush · « **Reply #13 on:** December 05, 2008, 09:18 AM »

Till now I miss in this kind of software automatic symlink or hardlink creation to clones. (hardlinks should be retestable in case the original file has been deleted)

Darwin · « **Reply #14 on:** December 05, 2008, 10:33 AM »

Sorry Darwin, I meant to say I supported your idea but somehow I left that bit out. My apologies.
-tsaint (December 05, 2008, 09:05 AM)

No worries!

mouser · « **Reply #15 on:** December 05, 2008, 02:50 PM »

So beside city_zen's suggestions do you have any other ideas on how to help the software to best determine what to select?

i think in general some good options might be:
always prefer the parent folder, or always prefer child folder
or when i specify a list of directories to search for dupes, always default to removing the files in later folders and keeping earlier folders.

mouser · « **Reply #16 on:** December 05, 2008, 02:56 PM »

As rjbull pointed out, DoubleKiller is a really good duplicate tool, you might check it out for inspiration:
http://www.bigbangenterprises.de/en/

Jan-S, the developer of DoubleKiller, also hangs out at DonationCoder.

4wd · « **Reply #17 on:** December 05, 2008, 06:31 PM »

1. Possibility of detailed selection of drives and folders to scan/watch - Implemented
-city_zen (December 04, 2008, 11:16 PM)
-AsmDev (December 05, 2008, 08:29 AM)

A bit late but....

The best interface I ever used for a duplicate finder, (as far as setting up for a search), was Duplicate File Finder 1.1.0.3 by Rashid Hoda, sadly no longer developed but still my main dupe finder.

New duplicate file finder... I am open to features suggestion

Everything you needed to set up a search was on the one window, you could select the folders on the left and they would be listed as you added them in the center-top, filters were immediately selectable below, basic comparison options on top-right, more involved accessed via buttons just below. Sub-directories excluded/included and to what depth, hidden files/folders, etc, etc - all accessed from what I regard as a very well laid out interface.

4. Ability to select only certain file types to be scanned (i.e. only music files, only video files, etc.) - Implemented

Having only played with DupeTrasher 1.2 Demo, my suggestion is make a way to be able to deselect ALL filters. It's a lot easier to deselect all and then select one than it is to deselect 20 just to leave one.

In Duplicate File Finder you can have multiple wildcards in a filter, (eg. *.jp* will filter *.jpg and *.jpeg), saves having to create multiple filters for similar extensions.

PS. OK, dumb question, what's the magic for adding an image as a thumbnail?
Mine are always attached full sized

mouser · « **Reply #18 on:** December 05, 2008, 06:54 PM »

PS. OK, dumb question, what's the magic for adding an image as a thumbnail?

see here:
https://www.donation...dex.php?topic=1996.0

Darwin · « **Reply #19 on:** December 05, 2008, 07:03 PM »

4wd - love that GUI myself

I like DoubleKiller, but setting it up is a bit of a PITA! I use it *just* infrequently enough to forget its idiosyncracies and have to fiddle around with it for a while before I can get it to return the result in which I am interested.

4wd · « **Reply #20 on:** December 05, 2008, 07:09 PM »

PS. OK, dumb question, what's the magic for adding an image as a thumbnail?
see here:
https://www.donation...dex.php?topic=1996.0
-mouser (December 05, 2008, 06:54 PM)

Woohoo! Thanks mouser.

And now back to our regularly scheduled program.....

f0dder · « **Reply #21 on:** December 05, 2008, 11:59 PM »

Hey Milos,

I don't use dupe finders myself, but just wanted to give a heads-up and say nice seeing you here

city_zen · « **Reply #22 on:** December 06, 2008, 11:14 PM »

Till now I miss in this kind of software automatic symlink or hardlink creation to clones. (hardlinks should be retestable in case the original file has been deleted)
-Crush (December 05, 2008, 09:18 AM)

Yes, that's a good suggestion. By the way, Duplicate Cleaner has that feature (this thread made me start researching the subject a bit

)

Crush · « **Reply #23 on:** December 07, 2008, 08:16 AM »

@city_zen
Hey, thanx for your research! Cool Freeware!

MrCrispy · « **Reply #24 on:** December 07, 2008, 08:20 PM »

Hi Milos,

Welcome to this site, I am very glad to see you here as your programs are really fast and appear to be well coded! One of the features I'd like in duplicate scanners is the ability to define exactly what constitutes a match. Most have builtin filters such as match by name/size/contents but that's usually it. I'd like some way to specify similarity searches based on fuzzy logic, or e.g. be able to match on html titles instead of file names. This may be more than a regular duplicates finder though.