This seems no different than the idea of trying to run a company democratically, which has been tried, or at least conceptualized for a looong time. The idea is interesting but it has lots of problems. Of course so does the current method.
More importantly this is *not* how open source software is usually done - at least not successful projects. Most successful projects *do* have a leader, or several. Decisions are not made by the coders, let alone the users. The coders can only really determine what *they* do, and discuss it with other coders and hope they agree.
It seems there's a fundamental error in associating a distributed group *work* process with a distributed group *decision making* process. Open source entails the former, not necessarily (and in fact rarely) the latter. And in fact I think you could say that, if any given open source project *were* "a company", it essentially would be an "open source company", because everyone is doing what they can when they can and not likely being paid for it. Many open source projects have people who do web pages or marketing or whatever, free of charge and on their own time, whenever they feel like it. This is an "open source" company, really.
Another good example would be a strong mod (game modification) team. Groups of 5, 10, 20 or even more people get together and work on creating a game modification (or often essentially a whole game, a "total conversion", like Natural Selection for Half Life or Day of Defeat for the same). These people are essentially running a company: There are people who do the coding, graphics, scripting, level design, etc.; those who are actually working on the product. But then there are also people doing web design and maintenance, marketing, file hosting, even PR, etc. And, almost inevitably, there are one or more "leaders".
But these are all software-based examples. What about realizing this in a real-world "brick and mortar" kind of way? Well in that sense an "open source company" would essentially consist of a workplace without commitment. Employees to not be paid for their work, but also have no set hours, hardly any rules, etc. It would still led by let's say a CEO or manager, but anyone could do anything to contribute to the company at any time, provided they do it for free. Considering the general attitude toward work and free time in today's society this is unlikely to catch on, not to mention other problems. It really depends quite a lot on what kind of business it is. Imagine a retail store that worked this way - how many people would just volunteer to work so they could pilfer merchandise? This is not usually an issue in the software community because what you are offering or selling is usually not a tangible or at least finite thing - software can be "taken" and still be available for all to use. Taking on a dishonest developer or website maintainer or whatever isn't going to do that much harm, generally speaking, unless it's simple sabotage (a legitimate but rarely realized issue). But there are safeguards against similar undesirable things in open source development just as there could be in such a company or store - don't give the keys to people until they've built up trust, for example. Still, I do think there are certain businesses for which this might be more functional.
The real question is whether there is any real advantage to this work approach for the business. Obviously for the employees there is to some degree because they can work whenever they want. The trick is they don't get paid, so it has to be something they enjoy. Again I think the way our society is structured, and our general attitude toward work and free time, would make this very likely to fail. Fundamental bits of our society would have to change for this to work on a larger scale or, alternatively, you'd have to create a microcosm society - say in a small town - where it just might function. Like open source software it would depend on the good will of everyone involved.
Finally, I leave you with an amusing real-world hypothetical example. It could be said like this: An Open Source business would be hot dog stand where you come to get a hot dog and, if you really like the dog, you just might jump behind the stand and serve the next customer in line behind you if you feel so inclined.
You might mix the ketchup and mustard together for "an exciting new taste treat". Or maybe the stand didn't have mayo because it spoiled too fast in the heat, but you brought along packets and you add them to the condiment selection. And so it goes. And bettere still if no one is manning the thing, you can just jump back there and serve yourself. An open source hot dog stand would be the least organized and least reliable one around, but it just might have the best selection of condiments you could find...