The Observer Effect and Using Google for Social Science
I thought this was an interesting quick and dirty social study using Google. (via Knowledge Problem)
For any individual study you can validly say that you think the estimate is too low, or indeed, too high, and give reasons for that. For instance, you might say that your sample was mainly young people who tend to be healthier than the general public, or maybe that the diagnostic tools are known to miss some true cases.
But when we look at reporting as a whole, it almost always says the condition is likely to be much more common than the estimate.
For example, have a look at the results of this Google search:
"the true number may be higher" 20,300 hits
"the true number may be lower" 3 hits
I often tell folks that the key to understanding behavior is to understand incentives. The media as institutions have incentives to sensationalize and scare (it sells papers) and as individual reporters have incentives to magnify the importance of whatever story he or she is working on.
But what I found really interesting was how the Observer effect comes into play here. Wikipedia has this brief definition of the observer effect:
In physics, the term observer effect refers to changes that the act of observation will make on the phenomenon being observed. This is often the result of instruments that, by necessity, alter the state of what they measure in some manner.
Click on the Google hit numbers above. I get 42,700 and 5,360 respectively, the increase presumably due in part to this article and links to it. Its impossible to report on patterns in Google searches without the very fact of such reporting affecting what is being measured.
delurking:
This is funny. I put in "the true number may be smaller" and got 2 hits. One said that about a spending growth estimate. The other was a physics paper.
September 28, 2009, 9:52 amRoy:
I think you have more accurately described Heisenberg's discovery than does the view which asserts the universe is uncertain. Put another way: the Uncertainty Principle says that there is a finite limit to the precision of a measurement because the measuring disturbs that which is measured rather than says that that which is measured is governed by chance. Put a still another, perhaps more final way: there exist limits to what people can know.
September 28, 2009, 10:14 amLinda Morgan:
For me, just now, "true number may be lower" got 5180 hits. However, "actual number may be lower" got 127,000 hits.
But get this. Stripping the modifier and entering only "number may be lower" snagged a mere 745 hits.
How the heck could that be? Seriously, wouldn't it have to appear at least 127,000 times, just for the "actual" phrase above? Plus 5180 more times for the "true" version?
My faith is being shaken here. Something more pernicious than the Observer Effect is afoot.
Still, shaken certainties aside, great post, as usual.
September 29, 2009, 5:47 amLinda Morgan:
Nevermind. Now I'm getting 283,000 hits for "number may be lower." I accessed Google from another page.
Faith restored. Sort of. Hmmmmm.
September 29, 2009, 5:58 am