Welcome Guest.   Make a donation to an author on the site October 22, 2014, 06:12:59 PM  *

Please login or register.
Or did you miss your validation email?


Login with username and password (forgot your password?)
Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.


You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.
 
Your Support Funds this Site: View the Supporter Yearbook.
   
   Forum Home   Thread Marks Chat! Downloads Search Login Register  
Pages: [1]   Go Down
  Reply  |  New Topic  |  Print  
Author Topic: Dealing with HUGE text files?  (Read 5579 times)
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« on: August 27, 2009, 01:07:07 AM »

I've seen a few inquiries here and there for people who need to deal with huge text files - we're talking the multi-gigabyte range here. Usually it's for log files and the people really just need viewing (and grepping) and not editing, but edit facilities might be necessary every now and then.

Does any of you guys know a text viewer (or, better, editor) that handles such files without hiccup? We're obviously talking something that doesn't use 32bit variables for file size or line count, and doesn't try to load the entire file at once... I'm pretty sure I bumped into such an editor years ago, but haven't had the need for huge files since.

Freeware would of course be preferred, but any suggestions are welcome. Some people seem to think that you'd need a 64bit editor to handle such big files, but imho it's perfectly possibly to handle with a (smartly programmed) 32bit editor... if you need to handle really big files, it's folly to try and do full-memory loading anyway smiley
Logged

- carpe noctem
housetier
Charter Honorary Member
***
Posts: 1,321


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #1 on: August 27, 2009, 02:56:43 AM »

I have had adequate results with vim and http://www.vim.org/script...script.php?script_id=1506

Naturally I recommend vim because I don't use anything else.
Logged
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #2 on: August 27, 2009, 02:57:59 AM »

Doesn't VIM load the entire file into memory?

EDIT: I know UltraEdit can handle files > 4gig, and doesn't try to load the entire file... but I'd like to see other alternatives smiley
« Last Edit: August 27, 2009, 03:34:20 AM by f0dder » Logged

- carpe noctem
jgpaiva
Global Moderator
*****
Posts: 4,710



Artificial Idiocy

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #3 on: August 27, 2009, 03:48:43 AM »

From what I know, vim does work with large files.
It does take a while to load the file, however.

I've just found this plugin which will probably solve that problem smiley
Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #4 on: August 27, 2009, 04:03:43 AM »

From what I know, vim does work with large files.
It does take a while to load the file, however.

I've just found this plugin which will probably solve that problem smiley
housetier already posted that script smiley - taking a while to load kinda indicates to me that it does full loading? I can't see why disabling an undo system would make initial file opening faster, for instance.
Logged

- carpe noctem
qwibbles
Supporting Member
**
Posts: 22


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #5 on: August 27, 2009, 04:13:40 AM »

http://en.wikipedia.org/w...omparison_of_text_editors

Look for Large file support

EditPad I think was ok for this and LTFViewer - Had to open and search very large website logs once..

LTFViewer was very fast to open as it only open's a section of the file. A few seconds for 10GB file

TextPad and UltraEdit are also generally ok with this size of file.

You would have to test them for your application of course and see which one suits you.
Logged
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #6 on: August 27, 2009, 04:35:28 AM »

Well, I looked at that, but I'm not sure the feature matrix is so useful... what do they mean by "large file support"? Is it 2GB, 4GB, or "available memory"? It says nothing about memory vs. diskbased either smiley


EditPad I think was ok
Their site says: "Edit huge files without breaking a sweat. EditPad Pro will instantly open files up to 2 GB in size, even if your PC has less than 2 GB of RAM." - what is with people using 32bit signed integers for file sizes? As if 32bit itself wasn't bad enough, wtf use SIGNED integers? Ever seen a negative-size file? >_<

TextPad and UltraEdit are also generally ok with this size of file.
TextPad doesn't seem to do the trick, quoting their features: "It can handle file sizes up to the largest contiguous chunk of 32-bit virtual memory."

LTFViewer sounds like something that could be useful smiley

PS: thanks for your effort, even if this post sounds a bit critical overall smiley
« Last Edit: August 27, 2009, 04:49:05 AM by f0dder » Logged

- carpe noctem
jgpaiva
Global Moderator
*****
Posts: 4,710



Artificial Idiocy

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #7 on: August 27, 2009, 04:43:15 AM »

housetier already posted that script smiley
Oops.. Sorry.

I just tested it with a large ISO, but even with the plugin, looks like it'll take a few minutes to load (I've waited for about 1min).
Logged

qwibbles
Supporting Member
**
Posts: 22


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #8 on: August 27, 2009, 05:08:03 AM »

http://www.donationcoder..../index.php?topic=14263.25

I knew I saw this once before :-)
Logged
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #9 on: August 27, 2009, 11:05:46 AM »

...and I even posted in that thread embarassed

It was mainly a thread about viewing though, this thread also hints at editing tongue
Logged

- carpe noctem
wraith808
Supporting Member
**
Posts: 6,417



"In my dreams, I always do it right."

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #10 on: August 27, 2009, 12:41:18 PM »

Well, without the editing there's V.  That's what I use to view.
Logged

urlwolf
Charter Member
***
Posts: 1,784



see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #11 on: August 27, 2009, 03:43:29 PM »

there's an editor that trumps all others for large files: emeditor. No contest
Logged
qwibbles
Supporting Member
**
Posts: 22


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #12 on: August 27, 2009, 04:05:12 PM »

Or you could just split the file with a file splitter so it's smaller  Wink

I number of Hex Editors Support very large files HxD is a free one that works on 5GB files on my machine. Frhed and 010editor may also work for you.

That just about exhausts me ..... I need to go lie down.... Thmbsup

Edit: off to play with emEditor  cheesy
« Last Edit: August 27, 2009, 04:07:16 PM by qwibbles » Logged
lothark
Supporting Member
**
Posts: 13


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #13 on: September 28, 2009, 12:54:36 AM »

You could also try command line less which will instantly open and read files of any size without loading them into memory.

You can get it here.
Logged
Pages: [1]   Go Up
  Reply  |  New Topic  |  Print  
 
Jump to:  
   Forum Home   Thread Marks Chat! Downloads Search Login Register  

DonationCoder.com | About Us
DonationCoder.com Forum | Powered by SMF
[ Page time: 0.038s | Server load: 0.15 ]