Main Area and Open Discussion > Living Room

Is Excel the most dangerous software in the world?

<< < (3/4) > >>

Shades:
Is Excel the most dangerous software in the world?
Well, whilst calling it the most dangerous software might be stretching it a bit, it could certainly be argued from a historical perspective that they had the potential to be "dangerous" and sometimes had actually proven to be so in fact.
My experience has been that Excel and Access have generally been the de facto financial black holes for a great deal of labour costs, where often non-IT personnel had spent many hours inventively developing one-off or prototype spreadsheet models and access database applications as business solutions to do often quite clever and useful things that simply could not have been done (at the time) in a timely or cost-effective manner using IT otherwise. Often these one-off solutions were tweaked to the stage where they could become so useful that they were commercially indispensable, and might even form a large part of the core legacy solutions for an enterprise.

A great strength of Excel and Access lies in their "tweakability" by the user - the extent of user control. That is also their Achilles Heel - there is a profound scope for human error in developing the logic and formulae in the models/applications, and that is why, from the perspective of stochastic and accounting accuracy, the models need to be independently audited and verified, but certainty in that regard (that all errors have been identified and fixed) is something that was and remains notoriously extremely difficult to achieve - despite the growth of things like spreadsheet auditing tools.

However, having worked on complex models such as ITEM (the UK Independent Treasury Econometric Model) and climate models of the North Sea, I would recommend skepticism towards any plea of incompetence - the "Oh dear, there was an error in the [insert name of scam] financial model" argument. These people aren't necessarily the fools they might have us believe.
For example, in "JPMorgan’s model had not captured this at all...".
Yeah, right.
-IainB (November 12, 2015, 05:03 AM)
--- End quote ---

I completely agree with your last paragraph.

**** rant alert ****
Sounds like you never had the "pleasure" of turning a solution created by such an "inventive" person into a system that actually scales. While those tools might provide a solution for a small company, never think that these work on a bigger scale like MS wants you to believe these tools do. Then again, I am of the opinion that whoever (or whomever) made Access should be taken behind the shed to be dealt with. That alone would be a huge improvement in any gene pool.

Those "inventive" people are allowed to make very(!!!) bad methods of handling whatever data/calculations they are involved with. Methods that would have been ridiculed by any IT pro. Every choice you make when using or creating software has significant consequences later on. Think of the "inventive" person as a butcher and the IT pro as a surgeon. You state that they can do the same job, just because both know how to carve meat?

Excel is also such a "success" story. Most people use it by only having one instance open. In this way I really can't blame you for mistaking it to be a useful tool. However, I work with software that really has to shift lots of data and using this data to calculate/generate much more data. There are many ways to get data into and out of this system. Excel is one of those. Customers use consultants that have the same fondness for Excel as most people seem to have.

All is still nice and dandy at this point. Unfortunately, consultants hardly work together, especially when they are not working in the same area of expertise. Because of the size of the customers and the myriad of disciplines they are active in, they use lots of consultants. The software I work with runs on any version of a Windows PC. From a workstation/single server to big beasts. One customer actually has a Windows system with 64 (multi-core) processors.

This customer was happy using our software system as it performed better as expected, while shifting their data and calculating/generating data. And then came the consultants with their Excel based input. Suddenly that beast of a system ran slower than a 486 PC by taking the input of all that Excel based input at the same time. It doesn't matter how much "horsepower" or RAM you put into a Windows based server...just run 20 instances of Excel at the same time. Doesn't matter which version of Excel or how complicated the Excel file being opened by that Excel instance is. Just open 20 instances.

Excel consumes a shameful amount of resources that messes up the way how Windows handles the available resources. Before I could prove to that customer that Excel was the only software to blame, levels of stress were at unhealthy levels to say the least.

My point is that for serious work Excel and Access are not alternatives and should not even be considered as such by anyone with a mind for IT. Now I fully understand that my comments won't change the opinion of most people and that they keep seeing Excel as the benevolent Dr. Jekyll. While IT pro's, which do need to work with lots of data, see Excel for its true nature, a brutally butchering Mr. Hyde.

Let's just say that it requires a lot of surgical skill to fix the butchering, no matter how "inventive" the butcher thinks he/she is.

Renegade:
^ For Excel connected to a database, maybe opening that many instances might be a problem. I don't do that, so I don't know that case.

But I regularly have 5, 10, 20, 50 instances of Excel running, and I don't have that many problems.

Yes, I do sometimes run into the memory error, but it's not very common. I can usually solve that by closing other applications, with the main culprits being Internet browsers, e.g. Chrome, Firefox, Opera, etc.

Just my own experience.

IainB:
I completely agree with your last paragraph.
**** rant alert ****
Sounds like you never had the "pleasure" of turning a solution created by such an "inventive" person into a system that actually scales. ...
____________________________________
-Shades (November 14, 2015, 09:17 AM)
--- End quote ---
Sorry for misleading you, but I did actually have that "pleasure" many times, and the use of "inventive" was a deliberate euphemism for "[expletive deleted]" - so I can resonate with your rant.
The most polite thing I could say about them in polite company was that they were "inventive", but I would also use that term as it was often quite a politically sensitive matter too, in that management had invariably at some point been responsible for mistakenly allowing these people to wreak their peculiar form of havoc on the company's information systems, so there was a lot of egg-on-face type of concern (no-one wants to get fired for making a genuine mistake in ignorance of the potential risks of that sort of mistake).

In one case, I recall that I was called in as a consultant in the role of project director to help at a company where they did indeed have around their necks the proverbial millstone - a legacy of these kinds of "inventive" solutions - mostly Access databases, and some Excel spreadsheets, and none of them documented, of course. A project had been belatedly set up to address the tremendous mess that had resulted and that now needed to be cleared up. It took quite a while, for which I was paid handsomely, so I couldn't really complain.    :)    $$$

I'd happily sweep the streets if I was paid enough.

TaoPhoenix:

I think what I'm missing here is that "on my better days" I was trained for my couple diff variants of accounting that every now and then you do a "sanity check" - a very fast and ugly-dirty approximate second calculation just to look for these kinds of random weirdness, plus intuition.

So if something begins to feel off, you smash out a "piece of paper and calculator guess" just to see if the results sorta make sense.

Maybe that's beyond the human mind at the big bank level, but "just believing" forever feels risky. So Excel itself isn't exactly the most dangerous software, it's just so generically useful in so many ways and the problem could be the operators maybe stretching the limits of what a mind can intuitively hold.

IainB:
I think what I'm missing here is that "on my better days" I was trained for my couple diff variants of accounting that every now and then you do a "sanity check" - a very fast and ugly-dirty approximate second calculation just to look for these kinds of random weirdness, plus intuition.
So if something begins to feel off, you smash out a "piece of paper and calculator guess" just to see if the results sorta make sense.
Maybe that's beyond the human mind at the big bank level, but "just believing" forever feels risky. So Excel itself isn't exactly the most dangerous software, it's just so generically useful in so many ways and the problem could be the operators maybe stretching the limits of what a mind can intuitively hold.
________________________________
-TaoPhoenix (November 15, 2015, 10:50 AM)
--- End quote ---

Yes, I reckon that you are absolutely right about the sanity checking bit - some spreadsheet "developers" don't even understand the need for or benefit of doing simple cross-casts of column totals in financial spreadsheets.
The worst offenders I have met in that regard ("Why would you need to prove your totals?") were mathematicians who apparently had never heard of that other mathematician - Fra Luca Bartolomeo de Pacioli ("the father of accounting and bookkeeping"). For years, all my spreadsheets have been built to self-check their totals, and to compare other totals where a difference would signify that there has been an error somewhere, and report on same in a dashboard.

However, humans are essentially error-prone (trial-and-error being our fundamental way of learning, after all), and our errors can be notoriously very difficult to identify and locate within spreadsheets - needle-in-a-haystack kind of thing - so they can often remain undiscovered, like a ticking time-bomb. I recall the NZ Treasury reportedly made a huge error in its announced budget forecast several years back, a main total apparently being out by a factor of 10, or something, due to a simple spreadsheet error. This was despite them having had a team of some of their best minds peer-reviewing the forecasting model and its results.

By the way, I would suggest that there's probably no need to have a mind that can "intuitively hold" all of a spreadsheet model, if the thing has been systematically built using good theory and "best" practice and with lots of internal checks and reports on same. If a spreadsheet could not be broken down into logical components, with each able to be checked independently, I would be very skeptical of its output, and if only (say) one man ("George" the developer) understood what the model was doing, I'd be even more skeptical.
As in science, mathematical models need to be accessible to see if they pass an independent test of falsifiability.

Navigation

[0] Message Index

[#] Next page

[*] Previous page