topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Thursday March 28, 2024, 7:53 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Dealing with HUGE text files?  (Read 14940 times)

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Dealing with HUGE text files?
« on: August 27, 2009, 01:07 AM »
I've seen a few inquiries here and there for people who need to deal with huge text files - we're talking the multi-gigabyte range here. Usually it's for log files and the people really just need viewing (and grepping) and not editing, but edit facilities might be necessary every now and then.

Does any of you guys know a text viewer (or, better, editor) that handles such files without hiccup? We're obviously talking something that doesn't use 32bit variables for file size or line count, and doesn't try to load the entire file at once... I'm pretty sure I bumped into such an editor years ago, but haven't had the need for huge files since.

Freeware would of course be preferred, but any suggestions are welcome. Some people seem to think that you'd need a 64bit editor to handle such big files, but imho it's perfectly possibly to handle with a (smartly programmed) 32bit editor... if you need to handle really big files, it's folly to try and do full-memory loading anyway :)
- carpe noctem

housetier

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • default avatar
  • Posts: 1,321
    • View Profile
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #1 on: August 27, 2009, 02:56 AM »
I have had adequate results with vim and http://www.vim.org/s...t.php?script_id=1506

Naturally I recommend vim because I don't use anything else.

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #2 on: August 27, 2009, 02:57 AM »
Doesn't VIM load the entire file into memory?

EDIT: I know UltraEdit can handle files > 4gig, and doesn't try to load the entire file... but I'd like to see other alternatives :)
- carpe noctem
« Last Edit: August 27, 2009, 03:34 AM by f0dder »

jgpaiva

  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 4,727
    • View Profile
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #3 on: August 27, 2009, 03:48 AM »
From what I know, vim does work with large files.
It does take a while to load the file, however.

I've just found this plugin which will probably solve that problem :)

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #4 on: August 27, 2009, 04:03 AM »
From what I know, vim does work with large files.
It does take a while to load the file, however.

I've just found this plugin which will probably solve that problem :)
housetier already posted that script :) - taking a while to load kinda indicates to me that it does full loading? I can't see why disabling an undo system would make initial file opening faster, for instance.
- carpe noctem

qwibbles

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 22
    • View Profile
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #5 on: August 27, 2009, 04:13 AM »
http://en.wikipedia....ison_of_text_editors

Look for Large file support

EditPad I think was ok for this and LTFViewer - Had to open and search very large website logs once..

LTFViewer was very fast to open as it only open's a section of the file. A few seconds for 10GB file

TextPad and UltraEdit are also generally ok with this size of file.

You would have to test them for your application of course and see which one suits you.

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #6 on: August 27, 2009, 04:35 AM »
http://en.wikipedia....ison_of_text_editors

Look for Large file support
Well, I looked at that, but I'm not sure the feature matrix is so useful... what do they mean by "large file support"? Is it 2GB, 4GB, or "available memory"? It says nothing about memory vs. diskbased either :)


EditPad I think was ok
Their site says: "Edit huge files without breaking a sweat. EditPad Pro will instantly open files up to 2 GB in size, even if your PC has less than 2 GB of RAM." - what is with people using 32bit signed integers for file sizes? As if 32bit itself wasn't bad enough, wtf use SIGNED integers? Ever seen a negative-size file? >_<

TextPad and UltraEdit are also generally ok with this size of file.
TextPad doesn't seem to do the trick, quoting their features: "It can handle file sizes up to the largest contiguous chunk of 32-bit virtual memory."

LTFViewer sounds like something that could be useful :)

PS: thanks for your effort, even if this post sounds a bit critical overall :)
- carpe noctem
« Last Edit: August 27, 2009, 04:49 AM by f0dder »

jgpaiva

  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 4,727
    • View Profile
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #7 on: August 27, 2009, 04:43 AM »
housetier already posted that script :)
Oops.. Sorry.

I just tested it with a large ISO, but even with the plugin, looks like it'll take a few minutes to load (I've waited for about 1min).

qwibbles

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 22
    • View Profile
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #8 on: August 27, 2009, 05:08 AM »
https://www.donation...x.php?topic=14263.25

I knew I saw this once before :-)

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #9 on: August 27, 2009, 11:05 AM »
https://www.donation...x.php?topic=14263.25

I knew I saw this once before :-)
...and I even posted in that thread :-[

It was mainly a thread about viewing though, this thread also hints at editing :P
- carpe noctem

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 11,186
    • View Profile
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #10 on: August 27, 2009, 12:41 PM »
Well, without the editing there's V.  That's what I use to view.

urlwolf

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 1,837
    • View Profile
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #11 on: August 27, 2009, 03:43 PM »
there's an editor that trumps all others for large files: emeditor. No contest

qwibbles

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 22
    • View Profile
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #12 on: August 27, 2009, 04:05 PM »
Or you could just split the file with a file splitter so it's smaller  ;)

I number of Hex Editors Support very large files HxD is a free one that works on 5GB files on my machine. Frhed and 010editor may also work for you.

That just about exhausts me ..... I need to go lie down.... :Thmbsup:

Edit: off to play with emEditor  :D
« Last Edit: August 27, 2009, 04:07 PM by qwibbles »

lothark

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 13
    • View Profile
    • Donate to Member
Re: Dealing with HUGE text files?
« Reply #13 on: September 28, 2009, 12:54 AM »
You could also try command line less which will instantly open and read files of any size without loading them into memory.

You can get it here.