The conflict of interest that is Google

[mouser]

I post about this issue every once in a while because it seems to me to be such an obvious issue of concern, and yet doesn't seem to get much attention.

Google is *the* search engine of choice for much of the world -- and there is an absolutely fevered pitch and entire industry devoted to getting results at the top of the results returned by google for any given search.

While google boasts about being neutral in terms of how they rank results, increasingly, as google expands its business to every product line and industry, they are inserting their own products into their search results in order to promote their other products.

Here's there latest from http://www.downloadsquad.com/2010/05/01/google-now-integrating-its-own-google-health-service-into-your-s/ :

Thanks to a tip from Stuart we learned Google Health is now being integrated into health search results. For instance, if you Google search "hay fever," the top search result is Google Health. The health content is provided by A.D.A.M., an online provider of health content that is physician reviewed for consumers. Taking the number one spot for themselves, Google is making their move to be the source where consumers get their online health information and potentially dominate health search. I guess if you're Google, organic search results are whatever you want them to be.

There is a huge incentive for google to rank the results it returns in order to favor products it is involved in, and pages on which it serves ads (and thus profits from sending people to websites that use their ads).

This has just about reached the breaking point for me.  The bottom line is that I no longer believe that google can be trusted to return results that are most likely to be what i am searching for, but rather the results and rank of information displayed are being leveraged to further their market share.

[superboyac]

This has just about reached the breaking point for me.  The bottom line is that I longer believe that google can be trusted to return results that are most likely to be what i am searching for, but rather the results and rank of information displayed are being leveraged to further their market share.

i agree completely.  I also ranted about this just recently around here.  I was talking about how you can't really find the right thing in Google anymore, unless you know exactly what you want.  But if you know exactly what you want, what's the big benefit of google.  I find I use google more along the lines of a spell checker than a true engine.  For example, if I don't know if it's espn.com or espn.net, i'll just search for espn on google and go the the right site from there.  but in that case, what did google really help with?  I knew it was espn, so i was already 99% of the way there.  But if you were to search for "sports news" and it gave you espn on top, then google helped.  That was a poor example because sports news does go to espn, but that's obvious.  but if you searched for normal things, google is not that helpful.  you'll get a ton of nonsense sites that are purely seo optimized websites with vomit-worthy content.

Google has become the very thing that made google Google in the first place.  That's funny...

[JavaJones]

I disagree in terms of search results; I still find Google gives pretty relevant results in most cases, and certainly as good as or better than other search engines. I think any failing in its searchers is largely due to people trying to game the system. It's in Google's best interest to provide you something that is relevant to your search (ideally, of course, something that they also make money off of - but relevancy is still more important because without that you won't click). So it doesn't make much sense that they'd be intentionally skewing their search results so far in favor of commercial interests that they'd be losing relevancy compared to other search methods. They'd be stabbing themselves in the foot. I've seen a lot of comparisons of Google and other search engines, especially recently with Bing as a heavy competitor, and in the worst case it's a tie with Google, seldom if ever is Google soundly beaten in what is still its core strength.

As for the promotion of its own results, I agree that this is worrisome and a bit unprecedented as far as I know. What I'd like to see is perhaps an additional highlighted "non-organic search results" area, similar to ads, that shows "recommended content". Then, just as with the ads, those of us who don't trust Google's "recommendations", can ignore it and just look at the top search results. This approach would I think be most in keeping with past behavior.

However I'm not entirely convinced that Google isn't just using its own algorithms on its own content to determine relevancy. It's entirely possible that Google's page on hay fever is just full of more relevant content than any other page. Take "flu" for example. You'd think Google Health might come up tops for that too; it comes up at the top for "common cold" and "eczema". But not for flu. It's flu.gov that takes the top spot. Google Health *does* have a page on Flu, https://health.google.com/health/ref/Flu but it doesn't show up in their search results in the 5 pages I checked, and a search engine ranking tool I tested it on said it wouldn't come up at all. Odd, but lending credence to the possibility that it may be an organic ranking. Perhaps there is simply so much other info about the flu out there that ranks higher, that Google's just doesn't show up. Entirely possible given the recent flu hysteria. Then again I can't find any other terms listed in Google's full topic index: https://health.google.com/health/ref/index.html where Google Health is not the top result, but then I only tried a few of the ones in the A section.

Thinking about this further, if you take a look at the actual search results for these terms, it does highlight Google Health, but with almost equal ranking (second horizontally on a list at equal level), you have links to the Mayo Clinic and more. Additionally if you visit the actual page it's almost more of a "meta" thing, with a brief description and then links to lots more resources. More of a "topic" in the search engine that is a dedicated page for more jumping off points. This is arguably a simple enhancement to the search service. In fact Bing is already doing this, and I think has been for longer than Google:

The search:
http://www.bing.com/search?q=common+cold&go=&form=QBLH&qs=n&sk=&sc=8-9
the first result brings you to this page:
http://www.bing.com/health/article/mayo-116994/Common-cold?q=common+cold

Notice Mayo Clinic, and the page has ads all over it (2 at top and a whole bunch down the side), whereas Google's has zero ads (unless you consider the Medline link on the left to be an ad). So I'm starting to wonder, where is the profit motive for Google? Especially when you compare to what Bing is doing. In fact, once again it seems like Google is doing what another engine is doing, but better and cleaner, with less commercial crap. Kind of turns this whole thing on its head...

Also interesting is that if a search result matches for a Google Health topic, you can visit it without having an account. But if you simply visit www.google.com/health you are asked to sign up with your Google Account, indicating it's more about the health records than general health info.

- Oshyan

[mouser]

JJ,

let me see if i can convince you of where i think you are making a mistake in your reasoning.

You say:

It's in Google's best interest to provide you something that is relevant to your search

I think this is generally right.. Or rather, i agree that ONE of their main interests is in providing results that you find more useful than the results from the competing search engines.

If they started returning completely ridiculous results that were clearly all about advertising and no real content, and if their results reallllly became obviously bad, they would be shooting themselves in the foot because people would *eventually* stop using google, despite all of the innate forces of momentum and reputation that help preserve their marketshare.

However, companies are rarely ever so stupid as to leverage their monopolistic or other advantages to this extreme that would obviously drive away customers.  Instead, they simply use their incestuous relationships and leverage and marketshare to put their fingers on the scales a little, enough to make a huge profit without scaring away customers.

Consider this hypothetical thought experiment:
Somewhere in google labs the people trying to improve search do a study and find out that academic pages and pages without advertisements are much much more relevant to almost every search than pages with ads.  And furthermore, that this is especially true for health issues, mortgage and finance and loan issues.  The solution is obvious, penalize the ranking of pages with ads on them.  The researchers running the study propose the change to the google search algorithm, and note that this change will make the google results more relevant and useful to people, but will cost google about 100 million dollars a day in ad revenue.  Now you tell me what you think the google corporation board of directors, etc. would say about this proposed change?  you think they would welcome it with open arms, or you think they would find a way to justify not implementing it?  I suggest the answer is hell no they wouldn't implement that change -- not at long as google doesn't have a viable competitor that people would switch to if they didn't.

This is the conflict of interest that happens when you are a company making oodles money when people visit certain sites and not others.  And it's even worse when some of those sites that you have an interest in sending people to are sites you own that make you even more money if you drive traffic to them.  It's true that google has an incentive to produce "relevant" results, but they also have a very high incentive to promote their own pages and products, and send people to pages with google ads on them.  Two incentives that conflict.. otherwise known as a "conflict of interests".

[JavaJones]

Yes, Google has a profit motive, no doubt about it. That's part of being a publicly traded company in this economy. What is interesting and - I think deserves recognition - about Google is that they do a pretty good job most of the time of balancing the profit with the information and quality of service. More often than not when I see someone pointing the finger saying Google is doing something purely or even largely for profit, when I look at it, it just doesn't seem that way if you consider the whole picture and all evidence. Particularly in this case if you notice the fact that there are no ads on those pages, and compare that to Bing.

If nothing else, Google may not be "as good as it could be", but in most cases it's a lot better than anyone else still, and that's unfortunately all we can expect a lot of the time. Though we can hope for more.

- Oshyan

[mouser]

What is interesting and - I think deserves recognition - about Google is that they do a pretty good job most of the time of balancing the profit with the information and quality of service.

Fair enough, i can't disagree with this (not yet at least.. we shall see what the future holds).

I have never claimed that google was anything but better than the rest of them.. I just don't think that's saying all that much.

[J-Mac]

When Google takes the top spot for themselves, in most cases it seems that all they are doing is knocking Wikipedia down to second. 

Seriously though, I completely agree about most top results being - as superboyac called it - "vomit-worthy" content. I realize that there is more and more data for search engines to sift through but the drek that makes the first page or two of results for just about any topic you search is getting to be tough to take.

Jim

[Renegade]

I agree.

I do a lot of tech searches, and the overwhelming results I get are total dog shit.

And guess what all the dog shit sites are? Screen scraping Google ad sites. They have no original content, and are useless. Again, and again, and again... There are a huge number of sites that scrape MSDN and other sites and just puke out those results. Google serves them up all the time.

And quite often, I will be searching for something that I KNOW is at some web site like the Code Project, but I'm just too lazy to type in "codeproject Ctrl+Enter" then type the search when I can hit "Ctrl+k" then type the search... The results Google returns are mostly crap.

Google ceased to be about relevance a long time ago. They're doing the "not evil" thing for their stock holders ==> maximizing profits.

Mouser, I'm with you! I'm also glad to see that I'm not the only person that isn't enthralled by Google to the point of blindness.

[mitzevo]

For powerusers, finding info on most search engines is a POP. If you think google is not-so-good, check out baidu.

[iphigenie]

As someone who has been involved in search, the only way in which google can be considered "relevant" is because it is the default - people are used to it and its quirks and all the noise in it

[Stoic Joker]

...And here I thought it was just me being to picky. I almost always use quotes & plusses when doing a search to quickly target error messages or APIs that I'm researching. Because less is more (results wise) for that type of search.

But yes, lately the usless crap links even when doing some thing extremely specific is quite annoying - I've even resorted (with great success) to using Bing a few times when Google just gave solid crap for results (e.g. two banner farms & a drive by).

[JavaJones]

Ok, so is Bing demonstrably and consistently better? If not, what is? Is this a systemic problem in the search market, or unique to Google?

- Oshyan

[J-Mac]

Ok, so is Bing demonstrably and consistently better? If not, what is? Is this a systemic problem in the search market, or unique to Google?

- Oshyan

I'd say a little of both. But since Google is the gorilla at the table it has a bigger impact than any others.

Jim

[JavaJones]

All I'm looking for is alternatives. If Google is bad (debatable, and not really true in my experience), then what is good? And if nothing is good, yet Bing and others are spending *100s of millions of dollars* on this problem, then maybe Google isn't doing so bad afterall?

- Oshyan

[zridling]

I suppose it depends on what is linked to what. Google Health along with WebMD are massive link dumps, are they not? The more links, the higher the ranking? Besides, each of those Google Health topics are well outlined without ads unlike the others listed.
____________
I'm no Google guru, but I can consistently find things (answers) that others at work and in my family cannot for some reason. Several times I've diagnosed my own medical problems after several doctors gave me erroneous advice. In each case, I sent them a letter telling them how I found the correct information that cured my problem, and suggesting they use this new thing called "the internet" or get better training. (I couldn't afford a followup visit to tell them in person.)

[Renegade]

A number of years ago Google was being abused by the SEO crowd. They reacted and put an end to a lot of it, it not most.

However, given the amount of effort that Google put into that, and the time since they obliterated it, it is fair to say that if it is in the SERPs now, then it's there because Google wants it there.

Now, given that there are a large number of sites that screenscrape most of their content, I can only assume that this is because Google allows it. They serve Google ads, unlike the sites that they scrape content from.

Let me repeat that.

The screen scraping sites scrape their content from sites that do not serve Google ads, then they use Google ads.

Let me repeat that again...

The screen scraping sites scrape their content from sites that do not serve Google ads, then they use Google ads.

Hmmm... Some things just make you wonder a bit.

For those out there looking to make a fast buck, there's an easy way that not a lot of SEOs have caught onto yet. This is definitely blackhat stuff, and very relevant right now. It works.

Why does it work?

Because Google would rather give you the same content from a site that serves up their ads than from a site that they will not make any money from.

Now, it is perfectly possible that I'm wrong about the motivation and that Google has simply screwed up massively... But the facts there all point to it.

[JavaJones]

I'm curious whether there is conclusive evidence that the "scraped" sites that have Google ads on them are higher ranked because of the ads, or because they *aggregate* content from other sites. If their site content is entirely irrelevant I would not expect them to rank high, and if they do I would definitely view that very suspiciously. But if instead these sites are simply stealing high-ranking content from multiple other sites and aggregating it, it makes sense (although it is a sad perversion of the valuation of content and "relevancy" in Google's algorithm) that these sites would rank higher than the multitude of sites they're stealing from.

I'm interested to see some specific examples of these sites so I can see for myself, both the high-ranking adwords aggregation sites, and the source sites they're apparently stealing from. If all this is going on, then at best it's a weakness in Google's algorithms, and at worse it's downright, straightforward corruption on Google's part. I'd like to know which.

Of course there's always the possibility that Google just doesn't have enough motivation to fix their results since they do in fact make money on these sites, so it could be an error of negligence rather than the even nastier explicit promotion of such sites, but regardless still a morally dubious approach to content ranking and product development/progress (or lack thereof).

Then again if Google started penalizing aggregation sites en mass (assuming there was no easy way to actually single out the true content stealing sites), I'm sure there'd be a big outcry from some major, legitimate aggregation sites. Often times it seems Google is in-between a rock and a hard place as far as how it tunes its search results. They've been sued multiple times by site owners who feel they're entitled to enjoy their high rankings forever, and accuse Google of changing their algorithms just to spite them or to promote something that Google themselves make more money off of. Often times these changes are ostensibly (Google claims) to help combat spammers, so it's always a tricky thing when you start tinkering with "dumb but unbiased" algorithms to try to make them smarter than those gaming the system. There are often civilian "casualties"...

- Oshyan

[mouser]

Of course there's always the possibility that Google just doesn't have enough motivation to fix their results since they do in fact make money on these sites, so it could be an error of negligence rather than the even nastier explicit promotion of such sites, but regardless still a morally dubious approach to content ranking and product development/progress (or lack thereof).

i dont think they have to engage in explicit promotion of these sites -- in fact they probably have a weak effort to occasionally go after the worst offenders.  but that's just standard operating procedure for corporations and government programs and every other organization that has an incentive to look the other way when abuses are occurring.  this isn't a superhero comic with arch villains whose goal is to destroy mankind.

google is making billions off of these sites hosting their ads -- these sites are the geese that are laying the golden eggs.  i return to my hypothetical example above - while it's important to make an occasional public showing of going after such sites, it's also important to their business model to never make any drastic changes that would substantially decrease their revenue stream.

[JavaJones]

I think it's rather presumptuous to conclude that these sites account for a significant portion of Google's revenue. Do you know if there are any statistics about that? Maybe someone has done studies?

- Oshyan

[mouser]

that's a good point, i have no idea what share of their revenue comes from where.  would be nice to know.

[40hz]

I'm always curious why I keep hearing the argument that Google is tweaking its algorithms to favor its own offerings.

Why would they need to?

Since Google (and only Google) knows exactly how their search engine works, who is in a better position to engineer a web page for maximum visibility and ranking than Google?

That's the problem with SEO strategies. Most of them are based  on some combination of common sense, deductive reasoning, and observation. Many also include a healthy dose of wishful thinking.

This leads to a fundamental problem:

For Google, SEO is an exact science. For everybody else - it's an educated guess.

Small wonder Google can precisely place a 'hit' anyplace they want in their rankings.

Mouser said it best. There is a conflict of interest that Google is never going to be able to get completely around as long as their business works both sides of their search engine.

[Stoic Joker]

That's the problem with SEO strategies. Most of them are based on some combination of common sense, deductive reasoning, and observation. Many also include a healthy dose of wishful thinking.

That reminds me of my favorite response when the brass start asking (or pondering aloud) about what if any SEO we should start shoveling money at. I usually reply: How do you recon they get all 400 folks that paid for their services to fit in the Top Ten of the same category?

This leads to a fundamental problem:

For Google, SEO is an exact science. For everybody else - it's an educated guess.

Small wonder Google can precisely place a 'hit' anyplace they want in their rankings.

Now there's one I never thought of, and it does indeed make perfect sense.

[40hz]

that's a good point, i have no idea what share of their revenue comes from where.  would be nice to know.

Advertising accounts for 97% of Google's revenue according to their 2009 annual report. It appears the AdWords and AdSense combo is what's paying the bills.

The full details of how it works can be found under the heading How We Generate Revenue on pages 37-39.

Screen shot of financials below - click to expand:

[attachthumb=#]



Sorry for the GIF. But if you try to copy any of Google's report to the clipboard, it drops all the spaces between words for some reason, making it very hard to read.


---------------------------------
Addendum:

I've got Google's whole 2009 annual report on PDF if anybody wants it. It butchers in at 1.15Mb (for 132 mostly text pages) which is why I hesitated to upload it. It's a mildly interesting read. I learned a thing or two by skimming it. The 'how we make money' section I mentioned above was particularly enlightening. But there's nothing in it that really surprised me other than the size of Google's reported revenue stream.

Gadzooks! I knew they were making money... but I had no idea they were making that much money.

[JavaJones]

I know their revenue comes almost entirely from ads, that's well understood. The bigger question for me is what the revenue split is between various types of pages. That's something I doubt we'd see them publicly disclose though. While the above numbers are interesting, the difference between "Google websites" and "Google network websites" doesn't begin to cover the granularity we need to understand the points through this conversation.

I do however agree with your previous point, and it's a good insight, that Google of course understands their own algorithms better than anyone else could, and so should be able to place a page tops in the rankings if they choose.

I certainly never disagreed that there is huge potential for abuse here. There is huge potential for abuse in having a central government, too, but few are advocating for its total dissolution; just appropriate controls, checks and balances. Theoretically we already have those checks in place with monopoly/antitrust legislation, etc. for corporations.

In any case I'm just not sure how much abuse is actually going on right now. But again the potential is there and is well worth concern.

- Oshyan

- Oshyan

[superboyac]

Wow, they make over $20 billion from advertising??  I would LOVE to see the breakdown of that and learn how it all works.