Author Topic: New duplicate file finder... I am open to features suggestion (Read 36984 times)

rjbull · « **Reply #25 on:** December 08, 2008, 04:03 AM »

I like DoubleKiller, but setting it up is a bit of a PITA! I use it *just* infrequently enough to forget its idiosyncracies and have to fiddle around with it for a while before I can get it to return the result in which I am interested.
-Darwin (December 05, 2008, 07:03 PM)

I don't think I quite understood your original point about indenting

You're more or less voicing a "problem" I have with any dupe deleter in that (a) I don't use it very often anyway, so I have to semi-relearn it each time; and (b) since its main purpose is destructive, it's hard work to test exhaustively because your target disappears and you have to keep re-copying test set originals.

DoubleKiller's interface is somewhat unusual, but it's very good at what it does, and features like being able to look inside archives for files they contain that are duplicated on disk, can be a real bonus.

nharding · « **Reply #26 on:** December 22, 2008, 07:49 PM »

One thing I was planning on writing on a DC++ modification (so that the same is not downloaded more than once) is to investigate the contents of a rar / zip file. So it would ignore the files, directory names, compression settings inside a rar/zip file, by getting all of the crc's & filelengths inside the rar/zip and then would be able to say Downloaded.zip is the same as Recompressed.rar.

I normally use the Duplicate File Finder mentioned above.

hpearce · « **Reply #27 on:** December 23, 2008, 05:51 AM »

I'd like the option to specify source and destination (for lack of a better word) ... search for duplicates of those files in folder A where they are found ion B,C, or D .... BUT not between the latter (I.E. between B and C).

This allows for quick checks without having to re-do the entire thing.

rjbull · « **Reply #28 on:** December 23, 2008, 09:07 AM »

search for duplicates of those files in folder A where they are found ion B,C, or D .... BUT not between the latter (I.E. between B and C).

This allows for quick checks without having to re-do the entire thing.
-hpearce (December 23, 2008, 05:51 AM)

If I understand it correctly, DoubleKiller does this with the concept of "fresh" and "library." Files in the "library" areas are assumed to be free of duplicates (i.e., you already checked them), so they don't have to be compared amongst themselves. The "fresh" ones are the new ones you don't know about yet, which have to be checked both against each other, and against the "library."

irh · « **Reply #29 on:** December 23, 2008, 10:29 AM »

Milos

I've only just started using DupeTrasher so I apologize if I'm not familiar with all of it's features...

One thing I'd like is to be able to force a deletion from a particular drive rather than rely on oldest etc. For example I have 2 trees (or drives) MASTER and COPY. I only want to remove duplicates from COPY and never from MASTER.

Is this possible?

Also one of the best tools I've used http://www.funduc.com/dupfiles.htm had an option to calculate MD5 or CRC32 hashes for a binary level comparison. Have you considered adding this as an option?

Finally, I spend a lot of time working on de-duping all sorts of files. So far no one has developed a tool that will reliably de-dupe Outlook MSG files. (The problem is that every MSG file is slightly different even when the same message is exported multiple times.) What is needed is a tool that reads some of the internal meta-data such as FROM, TO, SUBJECT, MESSAGE BODY etc. and uses this for de-dupe purposes rather than rely on MD5 or CRC32 hashes.

Is this something you would be interested in doing?

I'd be happy to share with you the procedures I've managed to use. Let me know: irh [AT] advancedforensics.com

Happy Holidays!

Simpsoid · « **Reply #30 on:** January 20, 2009, 03:28 AM »

I am using duplicate file removal,and i think it's better...U dont think so?

skywalka · « **Reply #31 on:** January 27, 2009, 11:01 PM »

The most important feature for me is being able to access the program from the Windows Explorer right click context menu. I hate having to open a program & rebrowse to the directory that I'm already working with. That goes for any application.

I'd like to be able to merge duplicates with hardlinks.

Curt · « **Reply #32 on:** January 28, 2009, 02:45 AM »

The most important feature for me is being able to access the program from the Windows Explorer right click context menu. I hate having to open a program & rebrowse to the directory that I'm already working with. That goes for any application.
-skywalka (January 27, 2009, 11:01 PM)

- ditto

AsmDev · « **Reply #33 on:** February 27, 2009, 07:24 AM »

Hello everyone,

I've been working on this software last two months and now it is almost done. I reviewed your requests and most of them have been implemented. I've also added support for archive files (.zip and .rar for now, as they are most common on the internet) and here are some new screenshots:

Duplicate files in archives are presented in several ways and one of them is following: each archive file that has duplicates outside itself is added to this list. You can expand/collapse to see them and mark the ones you would like to remove:

http://img142.imageshack.us/img142/2497/archiveswithdupes.png

New duplicate file finder... I am open to features suggestion

Also, there is a feature that will find archives with same content regardless of their name or extension. For example here are two archives with different name and type but with same content:

http://img142.imageshack.us/img142/3051/archiveswithsamecontent.png

New duplicate file finder... I am open to features suggestion

And the last but not the least is the feature that will find all folders where archives have been extracted. This happens often as you download some .zip from the internet and extract it somewhere but later you forget about it completely. Each archive has list of folders attached to it in the list, where it has been extracted:

http://img142.imageshack.us/img142/4908/archiveswithextractedfo.png

New duplicate file finder... I am open to features suggestion

If you still have some ideas feel free to let me know.

By the way, I am looking for BETA testers to whom I will be giving a free license after this software is released so if you'd like to contribute I'd really appreciate that. Please note that I need users who really need this kind of software and who would really like to give it a real-world test, and not just guys looking for free license. Those people should not worry either as I will be giving free licenses and discounts to this community when this project is done.

So send me a PM if you are interested, with your e-mail and operating system you use so that I can send you instructions and test build.

Thanks

Lutz_ · « **Reply #34 on:** February 28, 2009, 12:02 PM »

AsmDev,
these sound like very useful search options, no doubt. Hopefully the UI remains relatively straightforward.

mouser · « **Reply #35 on:** February 28, 2009, 01:54 PM »

we prefer people attach images directly to posts instead of using the image hosting sites; for info on how see: https://www.donation...dex.php?topic=1996.0

Taqxim · « **Reply #36 on:** February 28, 2009, 08:54 PM »

The one feature I really need and have never found in gazillion of dupe hunters is this: It's worthless displaying just the dupes without the contents and location of their container folders. A duplicate file is typically part of a package consisting of multiple files in parent and child folders which need to be included in the action. This is especially problematic when dupes are hunted based on a filter. Yes, you can sometimes open the containing folder in Explorer, but it's annoying to keep switching back and forth between windows.

I'd like to see everything in one window - a single pane showing groupings of dupe files, and off to the right, separate panes that display the containing directory folder(s) in tree structure. Since comparison is the goal, there should be separate panes for each selected dupe within a group. Tabs could be created displaying the folder tree for the selected dupe, with the ability to undock the tabs and arrange or tile the panes. An entire folder family can be wiped out or dragged to a special holding container to await their fate -- hold 'em or fold 'em. Layouts can be saved.

Oh, I would give up chocolate for Lent if someone made this! Otherwise, I'll be forced to. Hmmmmm.....

- Taqxim

AsmDev · « **Reply #37 on:** March 01, 2009, 05:46 AM »

Hi Taqxim,
I didnt completely understand your idea but here is what I already had done:

New duplicate file finder... I am open to features suggestion

In this feature I have tree hierarchy of all folders where at least one duplicate file is found. When you click on some folder you can see all duplicate files in it. Note that regular files are not listed. This way you can mark for removal all duplicate files located in one folder (and its subfolders) by clicking in the checkbox. Let me know if this somewhat aligns with your idea. Thanks!

mouser · « **Reply #38 on:** March 01, 2009, 06:26 AM »

sure is beautiful looking.

Dormouse · « **Reply #39 on:** March 01, 2009, 10:04 AM »

I've tried dupe finders and do have a need for them - especially when trying to deal with a variety of recoveries from crashed disks. But I feel a need to be cautious about deletion, which means that I use them more sparingly than I really should. So my main requirement on top of those above would be for the option to move to a 'de-duped' folder with an ability to undo the move for each individual file (or episode of de-duping). And then the option to totally delete after a user-decided time period (I'd probably use 6 months) when it had become clear whether there was a reason not to have deleted that version of the file.

Dormouse · « **Reply #40 on:** March 01, 2009, 10:04 AM »

sure is beautiful looking.
-mouser (March 01, 2009, 06:26 AM)

Have to agree with that.

AsmDev · « **Reply #41 on:** September 12, 2009, 06:32 AM »

Hello,

I'd just like to let you know that I have finally released DupeTrasher. You can check it out on www.dupetrasher.com

Thanks goes to everyone for their suggestions, and for those who helped me beta test it I will send a free license this weekend. Other members from this community who are interested in this program can get it with the 40% discount. I've posted details and coupon code on this new topic

https://www.donation...ex.php?topic=19901.0

I hope this will become your new favorite duplicate file finder . Thanks again for your posts!

nharding · « **Reply #42 on:** January 01, 2011, 12:17 AM »

In DCDisplay I am planning on adding a report of all archives found, which would check the contents of the archives to find duplicates.

Say you have file.zip which has 1.jpg 2.txt files inside it, it would produce a checksum of the contents of those files, so even if it was resaved with compression options, or even as a rar file it would still be possible to detect the fact it is a duplicate (the filenames of the contents would be ignored, so it would allow for even more cases) as well as excluding certains files inside the archive (thumbs.db in particular)

Neil

dpayment · « **Reply #43 on:** March 22, 2011, 09:11 AM »

Hi Milos, I'd like to add my vote to this feature. I have thousands of ebooks I've been collecting for some time now, but the problem is that while the filenames are the same, I have every ebook extension you can name, which leaves me with dozens of copies of the same book. I'd like to be able to specify a wildcard search for all files where the name is the same but the extension is different. Not sure if I'm being clear enough, but to do this manually would take me months.

Thanks,
Dan

- Possibility of several "levels" of similarity between files. For example: a) exactly the same, b) same name, different size, c) similar name, same size, same date, and so on
-city_zen (December 04, 2008, 11:16 PM)

Add to that, same root name but different extension, so you can flag e.g. ZIP and RAR archives that are really the same.

-rjbull (December 05, 2008, 05:18 AM)

f0dder · « **Reply #44 on:** March 22, 2011, 09:21 AM »

Sounds like you might want to look at an ebook manager instead, dpayment - while it's not perfect and it forces it's workflow upon you, I've found Calibre to be pretty good.

Darwin · « **Reply #45 on:** March 22, 2011, 11:18 AM »

+1 for Calibre. It also converts ebooks between formats while (heh, heh) retaining document formatting. My only quibble is that installing it (or updates) and opening it are dog slow...

rjbull · « **Reply #46 on:** March 22, 2011, 04:53 PM »

the problem is that while the filenames are the same, I have every ebook extension you can name, which leaves me with dozens of copies of the same book. I'd like to be able to specify a wildcard search for all files where the name is the same but the extension is different.
-dpayment (March 22, 2011, 09:11 AM)

Erm, DoubleKiller can do that already, so it looks like a good suggestion to add here

DoubleKiller has one version free for private use, and a more advanced payware version.