Why You Should Be Very Skeptical of Low-Sample-Size Advocacy Group Polling

A while back, I pointed out this poll from some group called the Commonwealth Fund.  In mid-October, on average about 15-18 days into the exchange process, they polled a group of non-corporate-insured adults (e.g. individual market or uninsured) about whether they had visited an exchange and what had been their experience.

The finding that stood out to me was that 21% of the people they interviewed that said they had visited an exchange reported that they had signed up for a policy (from the wording of the question, this probably includes both private policies and Medicaid signups).

I thought this seemed crazy-high.  And now new data from the Administration is confirming it.   The Administration is reporting about 106,000 "selected a plan" in October -- a very generous definition since it includes people who put a plan in their shopping cart but did not purchase it.  Not a definition of a sale that Amazon.com would ever use.    Further, another 400,000 or so were "found eligible" for Medicaid, whatever that means though it sounds well short of "enrolled."  So call it generously 500,000 people by the end of October.  The other key bit we need is that the Administration is reporting about 27 million unique visitors to these sites.  So at best we are looking at 1.9% of exchange visitors kind-of-sort-of-maybe having done something that approaches being enrolled.

This puts the Commonwealth Fund polling an entire order of magnitude off, at 21% vs. 1.9%.  And remember their survey occurred in the middle of the month, when the web site was not even working, and one can assume that successful enrollments were back-end loaded in the month.  The CF was nice enough to respond to my emails but were unable to explain the discrepancy, other than the sample size was low making the results unreliable.  So why the hell do it, and then put out a press release?

One explanation for part of the discrepancy may be in those who have created user accounts (normally a trivial task on a private site but a Herculean accomplishment on Healthcare.gov) but have not actually purchased a plan.  The Obama Administration says that there are about a million of these, so in addition to the 1.9% that put a plan in their shopping cart, there are another 3.7% that created a user account and gave the Feds enough info to assess subsidy eligibility, but who have not selected a plan.  Remember, that this was the minimum hurdle the Obama Administration originally set even to see insurance plan prices, and is still the minimum hurdle to get a subsidy quote.  It will be interesting to see the conversion rate of people once they find they are not getting free stuff from Uncle Sugar.

Even so, this only adds up to 5.6% of people who visited the exchange and had any sort of success (in most cases far short of enrollment) at all.  Way short of 21%.  Remember that we you see "studies" like this in the future.


  1. Orion Henderson:

    It's worse than your final 5.6% number. 3.7% created an account. 1.9% put an item in their shopping cart. This does not make 5.6%-since 100% of the 1.9% already went through and made an account. Slightly more than 1/2 of the 3.7% actually went so far as to place an item in their cart.

    My niche ecommerce site converts visitors to buyers at a rate of 3.2%+-. Their conversion rate is something well under 1.9%-and it must be A LOT less, or "they" would be crowing about it.

  2. random_eddie:

    The really scary possibility is that the Commonwealth Fund polling is accurate, and that 21 percent of the site visitors THINK they have enrolled for insurance.

    If this is the case, then come Jan / Feb / Mar when some significant portion of these people decide to see a doctor for some reason.... hilarity will ensue, on a scale that even the media won't be able to ignore.

  3. jimbeaux:

    I am one of the people who successfully created a user account on healthcare.gov. I was hoping - silly me - that Obamacare might offer plans which were financially competitive with the insurance offered by my employer. I experienced quite a bit of sticker shock when I saw the prices. Premiums were twice what I'm paying now. The annual deductible was over $10,000. And this was for their bronze plan, which only covers 60% of costs. So the cheapest plan would cost me about $20,000 a year before they paid a dime of my medical costs. Naturally, the prices were based on the presumption of how many healthy people would purchase a plan, which appears to be a wildly inflated and overoptimistic estimate.

    I find the administration's absurd numbers to be yet another indication of how poorly the signups are going. Were the administration truly transparent, they'd release honest numbers along with demographics of who is signing up. That information would be quite telling.

    I find it the Chutzpah of President Obama to be mindboggling, believing as he did (and does?) that people would embrace Obamacare with open arms, and would flock to sign up. He's in the process of a rude awakening, but of course the failure of his Affordable Care Act (affordable for whom? Not me!!) has to be the Republican's fault. And Bush's.