Dealing with HUGE text files?

[f0dder]

I've seen a few inquiries here and there for people who need to deal with huge text files - we're talking the multi-gigabyte range here. Usually it's for log files and the people really just need viewing (and grepping) and not editing, but edit facilities might be necessary every now and then.

Does any of you guys know a text viewer (or, better, editor) that handles such files without hiccup? We're obviously talking something that doesn't use 32bit variables for file size or line count, and doesn't try to load the entire file at once... I'm pretty sure I bumped into such an editor years ago, but haven't had the need for huge files since.

Freeware would of course be preferred, but any suggestions are welcome. Some people seem to think that you'd need a 64bit editor to handle such big files, but imho it's perfectly possibly to handle with a (smartly programmed) 32bit editor... if you need to handle really big files, it's folly to try and do full-memory loading anyway

[housetier]

I have had adequate results with vim and http://www.vim.org/s...t.php?script_id=1506

Naturally I recommend vim because I don't use anything else.

[f0dder]

Doesn't VIM load the entire file into memory?

EDIT: I know UltraEdit can handle files > 4gig, and doesn't try to load the entire file... but I'd like to see other alternatives

[jgpaiva]

From what I know, vim does work with large files.
It does take a while to load the file, however.

I've just found this plugin which will probably solve that problem

[f0dder]

From what I know, vim does work with large files.
It does take a while to load the file, however.

I've just found this plugin which will probably solve that problem

-jgpaiva (August 27, 2009, 03:48 AM)

housetier already posted that script - taking a while to load kinda indicates to me that it does full loading? I can't see why disabling an undo system would make initial file opening faster, for instance.

[qwibbles]

http://en.wikipedia....ison_of_text_editors

Look for Large file support

EditPad I think was ok for this and LTFViewer - Had to open and search very large website logs once..

LTFViewer was very fast to open as it only open's a section of the file. A few seconds for 10GB file

TextPad and UltraEdit are also generally ok with this size of file.

You would have to test them for your application of course and see which one suits you.

[f0dder]

http://en.wikipedia....ison_of_text_editors

Look for Large file support

-qwibbles (August 27, 2009, 04:13 AM)

Well, I looked at that, but I'm not sure the feature matrix is so useful... what do they mean by "large file support"? Is it 2GB, 4GB, or "available memory"? It says nothing about memory vs. diskbased either

EditPad I think was ok

-qwibbles (August 27, 2009, 04:13 AM)

Their site says: "Edit huge files without breaking a sweat. EditPad Pro will instantly open files up to 2 GB in size, even if your PC has less than 2 GB of RAM." - what is with people using 32bit signed integers for file sizes? As if 32bit itself wasn't bad enough, wtf use SIGNED integers? Ever seen a negative-size file? >_<

TextPad and UltraEdit are also generally ok with this size of file.

-qwibbles (August 27, 2009, 04:13 AM)

TextPad doesn't seem to do the trick, quoting their features: "It can handle file sizes up to the largest contiguous chunk of 32-bit virtual memory."

LTFViewer sounds like something that could be useful

PS: thanks for your effort, even if this post sounds a bit critical overall

[jgpaiva]

housetier already posted that script

-f0dder (August 27, 2009, 04:03 AM)

Oops.. Sorry.

I just tested it with a large ISO, but even with the plugin, looks like it'll take a few minutes to load (I've waited for about 1min).

[qwibbles]

https://www.donation...x.php?topic=14263.25

I knew I saw this once before :-)

[f0dder]

https://www.donation...x.php?topic=14263.25

I knew I saw this once before :-)

-qwibbles (August 27, 2009, 05:08 AM)

...and I even posted in that thread

It was mainly a thread about viewing though, this thread also hints at editing

[wraith808]

Well, without the editing there's V.  That's what I use to view.

[urlwolf]

there's an editor that trumps all others for large files: emeditor. No contest

[qwibbles]

Or you could just split the file with a file splitter so it's smaller  

I number of Hex Editors Support very large files HxD is a free one that works on 5GB files on my machine. Frhed and 010editor may also work for you.

That just about exhausts me ..... I need to go lie down....

Edit: off to play with emEditor 

[lothark]

You could also try command line less which will instantly open and read files of any size without loading them into memory.

You can get it here.