DonationCoder.com Software > Finished Programs
SOLVED: Remove duplicate entries from within a single large (text) dictionary
bhuiraj:
sort -u input.txt >output.txt (in Cygwin) is very fast and working great. Thank you for pointing me towards the ported nix apps.
@Moderator: Please close this thread.
MilesAhead:
sort -u input.txt >output.txt (in Cygwin) is very fast and working great. Thank you for pointing me towards the ported nix apps.
@Moderator: Please close this thread.
-bhuiraj (April 16, 2011, 01:34 AM)
--- End quote ---
Glad it worked for you. :)
bhuiraj:
In case anyone was wondering, it would take well in excess of a week to sort a 33GB dictionary. I started sorting my 33GB file on April 15th and finally cancelled it today (after 9 days) not even half done.
f0dder:
In case anyone was wondering, it would take well in excess of a week to sort a 33GB dictionary. I started sorting my 33GB file on April 15th and finally cancelled it today (after 9 days) not even half done.-bhuiraj (April 24, 2011, 11:53 AM)
--- End quote ---
There must be some more efficient software out there.
MilesAhead:
Out of curiosity, could you see the methodology? I would think something that big would have to use some type of merge sort. Esp. if you only have one disk that would be thrash city.
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version