How would I use/apply this? :)

-bhuiraj (May 02, 2011, 05:33 PM)

I would hazard a guess that most apps that use a "one word per line" flat text file dictionary just suck the whole file into ram and split on the end of line marker.  For example AutoIt3 has the user defined function _FileReadToArray().  If the file fits in ram it's trivial. Each array element is a line of the file.  Most dictionaries I've used are less than 2 MB in size.

You haven't specified if you're using any particular software most often to access this data.

-MilesAhead (May 02, 2011, 05:41 PM)

I use different pieces of software, so there isn't one specific one that I only use. All of them support this plain dictionary/wordlist/text file format.

Just for grins I started a thread on a db forum:

http://forums.databasejournal.com/showthread.php?p=129101#post129101

If anyone uses the terms "password cracking" or "dictionary attack" you're on your own!!

-MilesAhead (May 02, 2011, 06:13 PM)

lol thanks :)

Out of curiosity, could you see the methodology?  I would think something that big would have to use some type of merge sort.  Esp. if you only have one disk that would be thrash city.

-MilesAhead (April 24, 2011, 04:13 PM)

In cygwin's tmp folder, I noticed that the sort process created several smaller temporary files that were merged together in stages. My thought was that it splits the large dictionary file up into many smaller files of approximately the same size (some dictionaries were split up into multiple 64MB files), compares them, merges some of them together, compares again, and so forth until it ends up with two files that are half the size of the complete unduped file. Then, those two files are merged together to create the final dictionary with the duplicates removed.

Thank you for all the help, guys :). I will continue to look for a solution and follow the w7 thread. Please continue posting if you have any thoughts or new ideas.