Main Area and Open Discussion > General Software Discussion

any spreadsheet-like tool to open tables with millions of rows/columns?

<< < (2/3) > >>

tinjaw:
I am having a difficult time summarizing what I am thinking. That's no excuse for me to just ramble, but it might come out that way.  :-[

You are mixing apples and oranges here. I am going to hope that you understand Model, View, Controller (MVC) architecture. In layman's terms Data, Display, and Calculations. The focus of a spreadsheet is View and Controller, while the focus of a RDBMS is Model. Hence my suggestion of using a RDBMS backend with a spreadsheet frontend.

For lack of a better term (and a lack of coffee on my part) the "missing link" in your perception of the issue is that you are mixing the M, V, and C together in your mind because the spreadsheet appears to be doing them all at once, instantaneously, because of the GUI, while you are better realizing the three steps when working with a RDBMS because the workflow is more distinctly seperated into three steps of data entry, SQL query, and displaying the result.

I don't think the issue is one of keeping the dataset in memory vs on disk. I think it is simply a matter of workflow -- of how you perceive spreadsheet being more conducive to "playing" with data.

urlwolf:
@MikeMcLoughlin: thanks, CSVed looks very good.
@tinjaw: good analogy; never thought of it that way. Right now,
data/model is mysql. which I have to dump into huge text files because I could not get the RMySQL package to work;
View/display The weak point of R. I miss conditional formatting in excel, quick sorts etc. Done in R, dumping to text files or clipboard and pasting into excel
Controller/Calulations R. Doing a good job.

I guess I just want the best of all paradigms and I cannot get that.

kfitting:
One problem I have with databases is that it takes a fair amount of time to design and write a frontend... I like spreadsheets because of the inherent GUI.  I really do wish there was a better way to mix the two ideas.  I don't have time to write a GUI for every little database I would like to make!

Kevin

tinjaw:
I'm not claiming it is a panacea, but it is something you should check out if you haven't already.

Resolver One is a program that blends a familiar spreadsheet-like interface with the powerful Python programming language, giving you a tool with which to better analyze and present your data.

--- End quote ---

urlwolf:
I tried resolver one; not very stable, and chokes on largish files. A pity, because the idea is very good.

Navigation

[0] Message Index

[#] Next page

[*] Previous page