Main Area and Open Discussion > General Software Discussion

any spreadsheet-like tool to open tables with millions of rows/columns?

(1/3) > >>

urlwolf:
Do you know of any spreadsheet-like tool to open tables with millions of rows/columns?
I really like emeditor because it can open large files without placing them into memory.
If I could find a spreadsheet tool that did the same, it'd be great.

Thanks

tinjaw:
I think this is a parallel of owning a hammer and everything looking like a nail. If you have that much data, you need a database and a frontend. Now, you can use a spreadsheet as a frontend to the database, but you need that data in a db. Then you only view the results of calculations in the spreadsheet.

But you are a smart guy, so I must be missing something.  :huh:

tide:
I agree with tinjaw. I can't imagine a supermassive spreadsheet with any kind of structure being able to look stuff up in a reasonable time to say nothing of recalulating.

urlwolf:
I do have a database :)
I still miss some things a spreadsheet can do.

I'm right now torn between different modes to analyze data.
Right now I'm R-centric.
(www.r-project.org).
But in R you do everything in the command line. all changes are logged, which is good. Spreadsheets are a big no-no in this community.

The rdbms community has different ideas about how to manage data.
Basically you get what you want a query at a time.

And of course, Joe six-pack does everything with excel.

I wonder if I'm not missing much of the attractive of a visual way to handle data. Color-code cells according to contents, zoom in and out with an slider, ... that kind of feeling. Very exploratory. Compare that to a command line where you do head(data) :)

Since emeditor and R prove that you can do lots of operations without placing things into memory (or screen) I see no reason why one cannot have a streaming spreadsheet that loads the parts you need on the screen really fast.

Doing operations by selecting block would generate the equivalent SELECT for the db... if you need one. There might be very fast alternatives.

I think there are many paradigms for data handling that are not that well explored. For example, comparing OO, RDBMS and RDF (linked data) is eye-opening. This is beyond the point I'm bringing here about a streaming viewer (btw GUIs for RDBMS do suck at this; it's not that hard!)... but still. Let me know if you want more info or if I'm not making sense at all. I have a paper in mind on Object-relational mappings that is very good, if a bit technical.

MikeMcLoughlin:
Have you looked at CSVEd? It may do what you want.

CSVEd

Navigation

[0] Message Index

[#] Next page