Uncovering Some Really Bad Science
Kevin Drum thinks he has a killer analysis supporting government health care. In a post he titles sarcastically "Best Healthcare In the World, Baby," Drum shares this chart:
The implication is that the US has the worst healthcare system, because, according to this study, the US has the highest rates of "amenable mortality," defined as deaths that are "potentially preventable with timely and effective health care."
I get caught from time to time linking to studies that turn out to have crappy methodology. However, I do try to do a little due diligence each time to at least look at their approach, particularly when the authors are claiming to measure something so non-objective as mortality that was "potentially preventable."
So, when in doubt, let's look at what the author's have to say about their methodology. The press release is here, which gets us nowhere. From there, though, one can link to here and then download the article from Health Affairs via pdf (the site is gated but I found that if you go through the press release site you can get in for free).
The wording of the study and the chart as quoted by Mr. Drum seem to imply that someone has gone through a sampling of medical histories to look at deaths to decide if they were preventable deaths. Some studies like this have been conducted. This is not one of them. The authors do not look at any patient data.
Here is what they actually did: They arbitrarily defined a handful of conditions as "amenable" to care. These are:
Ischemic Heart Disease (IHD)
Other circulatory diseases
Neoplasms (some cancers)
Diabetes
Respiratory diseases
Surgical conditions and medical errors
Infectious Diseases.
Perinatal, congenital, and maternal conditions
Other (very small)
All the study does is show how many people died in each country from this set of diseases and conditions. Period. It doesn't determine if they got care or if they in particular could have been saved, but just that they died of one of the above list of conditions. This study was not an effort to identify people who died when their particular condition should have been preventable or amenable to care; all it measures is the number of people in each country who died from list of conditions. If Joe is talking to me and in the next second flops over instantly dead of a massive heart attack, the author's consider him to have died of a disease amenable to care.
We can learn something by looking at the breakdown of the data. If you can't read the table below, click on it for a larger version
Let's take the data for men. The study makes a big point of saying that France is much better than the US, so we will use those two countries. In 2003, France has an "amenable disease" death rate 56 points lower than the US. But we can see that almost this whole gap, or 42 points of it, comes from heart and circulatory diseases. The incidence of these diseases are highly related to diet and lifestyle. In fact, it is well established that the US has a comparatively high incidence rate of these diseases, much higher than France. This makes it entirely possible that this mortality difference is entirely due to lifestyle differences and disease incidence rates rather than the relative merits of health care systems. In fact, this study is close to meaningless. If they really wanted to make a point about the quality of health care systems, they would compare them on relative mortality with a denominator of the disease incidence rate, not a denominator of total population.
But in their discussion, the study's authors reveal themselves to be, if I am reading them right, complete idiots in terms of statistical methods. The authors acknowledge that lifestyle differences may be a problem in their data. This is how they say they solved this problem:
It is important to recognize that the development of any list of indicators of amenable mortality involves a degree of judgment, as a death from any cause is typically the final event in a complex chain of processes that include issues related to underlying social and economic factors, lifestyles, and preventive and curative health care. As a consequence, interpretation of findings requires an understanding of the natural history and scope for prevention and treatment of the condition in question. Thus, in the case of IHD, we find accumulating evidence that suggests that advances in health care have contributed to declining mortality from this condition in many countries, yet it is equally clear that large international differences in mortality predated the advent of effective health care, reflecting factors such as diet and rates of smoking and physical activity.16 To account for this variation, we included only half of the mortality from IHD, although, based on the available evidence, figures between, say, 25 percent and 70 percent would be equally justifiable.
I have a very smart reader group, so my sense is that many of you already see the gaffe here. The author's posit that 50% of heart disease may be due to lifestyle, though the number might be higher or lower. So to correct for this, they reduce every country's heart disease number (IHD) by a fixed amount of 50%. WTF?? This corrects for NOTHING. All this does is reduce the weighting of IHD in the total measure.
Look, if the problem is that lifestyle contribution to heart disease varies by country, then the percentage of IHD deaths that need to be removed because the deaths are lifestyle related will vary by country. If the US has the "worst" lifestyle, and the number for lifestyle deaths is about 50% there, it is going to be less than 50% in every country. The correction, if an accurate one could be created, needs to be applied to the variance between nations, not to the base numbers. Careful multiple regressions might or might not have sorted the two sets of causes apart, but dividing by 50% doesn't do anything. This mistake is not just wrong, it is LAUGHABLE, and calls into question the author's qualification to say anything on this topic. They may be fine doctors, but they don't know squat about data analysis.
There may be nuggets of concern for the US lurking in this data. I don't know how they measure deaths from surgical conditions and medical errors, but its not good to be higher on this. Though again, you have to be careful. The US has far more surgeries than most other countries per capita, so we have more surgical deaths. Also, medical error data is notoriously difficult to compare country to country because reporting standards and processes are so different. In the US, when the government measures medical errors, it is a neutral third party to the error. In Europe, the government, as healthcare provider, is often the source of the error, calling into question how aggressive these countries may be in defining "an error." Infant mortality data is a good example of such a trap. The US often looks worse than European nations on infant mortality because it is defined as infant deaths as a percentage of live births. But the US has the most advanced neo-natal capabilities in the world. Many pregnancies that would result in a "born dead" in other countries result in a live birth in the US. Since these rescued births are much more problematic, their death rate is much higher.
There is good news for the US in the study. The item on this list most amenable to intensive medical intervention is cancer (neoplasms in the study above). In that category, despite a higher incidence rate than many of these countries, the US has one of the lowest mortality rates as a percentage of the total population, which implies that our cancer mortality in the US as a percentage of cancer incidence is much better than these countries. This shows our much higher 5-year cancer survival rates.
Update: I thought this was pretty clear, but some of the commenters are confused. The halving of IHD numbers was applied to all countries, not just the US. So the actual male US IHD number is about 100 before halving and the actual French number is about 40. Again, this halving only reduces the weighting of IHD in the total index; it in no way corrects for differences in incidence rate.
nicole:
"Perinatal, maternal, and congenital conditions" also have significantly higher mortality rates in the U.S., for both sexes. I would wager most of this is related to the fact that we count much younger premature babies as infant mortalities than do other countries, making our infant mortality rate seem extremely high just because we bother trying to save these infants.
January 8, 2008, 8:55 pmM. Hodak:
I have relatives in about a dozen mostly OECD countries. Most of those relatives consider the U.S. to be their medical destination of last resort. Medical care in France or Canada or Israel doesn't suck. It's actually quite good, from my family's experience. So why do so many of them have this high opinion of American medical treatments, and ready visas for when they might need to access it?
January 8, 2008, 11:11 pmAndy:
15,000 senior citizens dying over the space of a couple weeks due to a heat wave in the midst of the annual Summer exodus in France would be an amenable mortality, n'est pas?
I can tell you from personal experience, you don't want to find yourself in need of medical care in late July or August in France. Lucky for me when I broke my foot, it was near Monte Carlo, where at least the hospitals were operational to treat vacationers.
January 9, 2008, 12:25 amdearieme:
I suspect that it is enormously difficult to recover good information from heaps of international data, gathered, often rather uncritically, using different conventions. Apart from everything else, death is easy to diagnose but cause of death is not, and recorded cause of death is known to be subject to mere fashion. (My own father-in-law's cause of death may well have been misrecorded in fear that we might sue over the infection that he picked up in one of our notoriously lethal NHS hospitals.) I don't really believe it when an analysis shows the US in a good light, or a bad one, compared to other advanced countries. The difference between rich and poor countries would show up, no doubt, but that's of little interest.
January 9, 2008, 5:43 amIronman:
Here's some other data of interest:
Natural Life Expectancy (compares the U.S. with OECD countries)
Since you mentioned five-year cancer survival rates, here's the rankings of OECD countries taking into account the findings of the largest and most comprehensive research study to date, which was released in August 2007.
Finally, here's the relationship between what a country spends on health care vs. the size of its economy. Interesting because it breaks the U.S. into 50 states, which are more similar in economic size and population to most OECD countries.
January 9, 2008, 6:21 ambill-tb:
I agree with dearieme, data collection in some countries is going to be very bad. Math education isn't what it used to be.
It is becoming clear that none of this stuff comes without a political agenda, there are no honest brokers any more.
January 9, 2008, 6:33 amJoe:
"There is good news for the US in the study. The item on this list most amenable to intensive medical intervention is cancer (neoplasms in the study above). In that category, despite a higher incidence rate than many of these countries, the US has one of the lowest mortality rates as a percentage of the total population, which implies that our cancer mortality in the US as a percentage of cancer incidence is much better than these countries."
Survival rate across all cancers is the absolute wrong statistic to use. It is enormously skewed by the US's screening regime for prostate cancer (and colon cancer, to a lesser extent), which catches a ton of cancer that would probably be non-terminal if left untreated (because the patient would die of something else before the cancer became life-threatening). Other health care systems have chosen to "sacrifice" those relatively few patients who would die absent screening to save the costs associated with the screening (mostly financial, but also QOL in the non-terminal patients who avoid an impotence-inducing operation (or a colostomy bag in the case of colon cancer)). I'm not saying that I would make that choice, but I can see the rationale for doing so.
The mortality rate across the population is a much better statistic to use because it effectively weeds out the pseudo-"false positives" (though this admittedly is complicated by factors like smoking rate, diet, average age of childbirth, etc.). And if you look at these figures, you will see that the US does about as well as the better European countries, and better than everybody else.
January 9, 2008, 10:15 amjoe:
I have had heart disease for atleast ten years as far as I know. I had a heart attack at age 38. and that was ten years ago. so far I have had 4 heart attacks.I had a triple bypass when I was 40 and I have 5 cardiac stent implants. Heart disease was very prominant in my family and my mom died when she was 51. still I smoked and ate a lot of junk food. I really didn't know the dangers of what I was doing until I had the heart attack.
I cannot work cause im too sick so i'm spending alot of time promoting Awareness. I have a website if anyone is interested. http://www.living-with-heart-disease.com
Joey
January 9, 2008, 11:09 amLarry Sheldon:
Maybe my question was covered somewhere here, but I missed it if it was.
The question (or maybe is will be a series of questions....
Is it true that people come here for medical care from somewhere else?
If they die here (off the disease or trauma for which they came here) where do they get counted?
Is there ever a case (here or anywhere else) where the official "cause of death" is matched against diagnoses prior to death? Which one is used in the counts? (An example of the problem: Person is admitted for "Respiratory diseases" and dies of an iatrogenic staph infection.)
And of course, there are people who would question the classification of some of those as "amenable".
"Surgical error"? "Diabetes"? Late stage emphysema?
I don't think so.
January 9, 2008, 3:27 pmFrederick Davies:
Everytime I get to read one of these "studies" I am reminded of Benjamin Disraeli's quote: "There are three types of lies: lies, damn lies and statistics." It has been more than a century since Disraeli's days, and we still do not learn!
January 9, 2008, 4:00 pmIronman:
Joe: Fine. If you don't like survival rates across all cancers, perhaps you might find the table at this link which presents the survival rate by type of cancer between the EU and the US more meaningful. You'll need to scroll down, but the payoff is that you can sort the table by clicking the column headings.
Larry: To answer your questions in order:
1. Yes, they do. Most newsworthy in the past year is a prominent (and now former) member of the Canadian parliament, Belinda Stronach, who has been receiving treatment in Los Angeles for breast cancer.
2. Death certificates, and any associated medical reports, are filed in the country where the death occurs. The deceased's home country would only receive copies of the death certificate through the U.S. State Dept. The death would be added to the count in the country in which it occurs.
3. I don't know the answer regarding the full matching of medical diagnosis and cause of death. However, the cause of death would be what is registered in the mortality counts. The only exception I can think to that situation would be if the cause of death was brought on by complications associated with the condition for which they had been diagnosed. For example, nobody dies from having AIDS, but rather from the conditions that they develop because they have AIDS.
January 9, 2008, 4:57 pmRoy Lofquist:
Better a few fewer years in the US than some other places I have been.
January 9, 2008, 10:03 pmChris:
Doesn't the "false positive" screen with cancer go to some length to prove the costs in the US are high, not because its private, but we consume unnecessarily health care - one of the main arguments that opponents are using to oppose single payer in the US?
January 9, 2008, 11:09 pmIronman:
Chris: The main problem with the "false positive tests lead to unnecessary treatment" argument is that the tests themselves are relatively inexpensive, as is the treatment associated with a multitude of cancers found in their early stages. Plus, in many cases, additional testing is available to confirm whether an early screening test result is valid (which means the costs aren't as bad as you might think.)
The primary reason health care is expensive in the U.S. is because health care "consumers" demand it and have demonstrated that they are willing to pay a premium for very specialized care or for rapid programs of treatment. As a result, the range of options for health care and the number of practitioners have increased over time. This contrasts with other countries that have considerably more restrictive regulations and controls over health care, such as those with single payer, or national health care systems.
Most countries with national or "universal" health care systems are able to provide health care "cheaply" since they restrict the kind of care that is available (Canada's single payer system, for instance, has chronically underinvested in the kind of medical imaging technology that has revolutionized the ability of doctors to diagnose and monitor the progression of many serious conditions), or by delaying or slowing treatment (Canada is a good example here, but doesn't hold a candle to the British National Health Service.) The NHS is also unique in that they've come to the point where they're now willing to restrict who is eligible to receive some treatments based on their lifestyle.
In a nutshell, those are the main arguments against a single payer or "universal" health care system in the U.S.
January 10, 2008, 10:07 amJoe:
Thanks for the link. It seems to be fully consistent with my general view that the US about the same as the best European countries, and noticeably better than the rest.
"Doesn't the 'false positive' screen with cancer go to some length to prove the costs in the US are high, not because its private, but we consume unnecessarily health care - one of the main arguments that opponents are using to oppose single payer in the US?"
First, I've never seen the argument that single payer would encourage the consumption of unnecessary health care. Quite the opposite -- it would probably (unpopularly) prevent the consumption of unnecessary health care via population-wide rationing (basically, instead of "luxury" treatments being available to those with great insurance, they would only be available to the extremely wealthy who could afford to pay them out-of-pocket).
Second, the argument ignores that a major non-financial cost of prostate screening is the virtually certainty that most of the cases you diagnose are going to be treated for a non-terminal condition. Meaning impotence in many (if not most) cases. It may be the case that other health care systems assign a higher price to this "side effect" of screening than does the US. Or it could be that the US's screening regime is not cost-effective from a purely financial standpoint. Though given all of the people looking into it, I find the latter explanation a bit hard to believe -- valuing intangible costs differently seems to be a more likely reason.
January 10, 2008, 10:12 amJim Hu:
Looked at the paper last night, and I think there's some sleight of hand going on here. The authors provide the appropriate caveat that the excess deaths from amenable causes is an indicator of possible problems:
Potential means also potentially having nothing to do with effectiveness of health care. But then, it seems to me that they go ahead and discuss the results as if potential =actual. Certainly Drum makes this jump.
A lot of the paper looks at the change in "amenable" death rates by country, and shows that the US is making less progress than other countries. As I point out here, this could be that we are behind other countries or that we are further along the curve of diminishing returns. The data doesn't distinguish between these.
January 10, 2008, 11:37 amKary:
Just thought your readers would like a peek at some criticism of your criticism!
January 11, 2008, 10:02 pmhttp://tinyurl.com/2f28sc
Ari Tai:
If it's not obvious, the Commonwealth Fund and George Sorus are joined at the hip (directorships, people, likely financial, indirect or not, etc.). Doesn't invalidate the study directly, but certainly should be sorted high for review and replication. Dr. McKee has a blog.
http://martinmckeesblog.blogspot.com/
January 12, 2008, 6:27 amkj:
Your criticism is interesting but you fail to explain why the other 18 countries saw greater improvement in amenable mortality than the U.S. I hope you would concede that this relative non-improvement by the U.S. is clearly relevant and quite damning.
And although the 50% reduction in IHD is hardly the best method to correct for lifestyle you make it sound like it has no effect on the results when it quite obviously does effectively making IHD less of a factor in the total number and thus being a correction of sorts for the totals. Regardless, even if you want to dismiss the comparison between countries, you'd be a fool to dismiss the relative improvements which are damning for defenders of the U.S. system and its very high ranking (perhaps #1) in increasing cost over the time period studied.
January 12, 2008, 7:25 pmMark Westling:
Here's another idea: could the reduction in deaths due to IHD be due to improvements in emergency medical response instead of basic health care? I've heard evidence of rapid recent improvement in fire brigade/EMS services in Ireland, which was one of the countries that passed the U.S. Maybe it's the same in the UK?
To address kj's point, something changed in those other countries. I'm not convinced it was the quality of basic health care or the number of people getting health care, though.
January 13, 2008, 2:38 amAri Tai:
The numbers seem to suggest cigarette smoking leads to better health outcomes(?). The integral of per capita tobacco consumption in the U.S. over the last 50 years is 20-30% of Europe's.
January 13, 2008, 6:18 amJason:
kj,
Your criticism is interesting but you fail to explain why the other 18 countries saw greater improvement in amenable mortality than the U.S. I hope you would concede that this relative non-improvement by the U.S. is clearly relevant and quite damning.
It's not damning at all. It could be any number of things that have nothing to do with the health care system. It could largely be a matter of diminishing returns.
And although the 50% reduction in IHD is hardly the best method to correct for lifestyle you make it sound like it has no effect on the results when it quite obviously does effectively making IHD less of a factor in the total number and thus being a correction of sorts for the totals.
No, it's not a correction of any sort. It's nonsense. The whole point of lifestyle and other variables is that they are variable. Their contribution differs between different countries. So just reducing the contribution attributed to health care by the same percentage for each country is meaningless as a correction for other variables influencing mortality.
January 13, 2008, 3:19 pmMahim:
I'm having a hard time understanding your outrage over the adjustment to the IHD numbers. A plausible case can be made about the validity of the adjustment from either side. To wit,
A: 100 people per million die from IHD in the US, and only 20 per million in France.
B: That is bullshit, most of the difference could be from lifestyle issues.
A: Fine, we will attribute 50 deaths per million from the US numbers and 10 deaths per million from the French numbers to lifestyle-related causes, and deduct them from our numbers.
B: This is bullshit! You just reduced both numbers by 50%! Surely a greater proportion of US deaths should be because of lifestyle-related reasons.
C: This is bullshit! You attributed 5 times as many US deaths to lifestyle reasons as you did French deaths! How can you say lifestyle has so big an influence in the US compared to France?
I don't find either argument more compelling than the other. Therefore, I can only conclude that the adjustment is a reasonable thing to do.
January 13, 2008, 5:14 pmJason:
Mahim,
You are confused, perhaps in part because you are talking about numbers of mortalities rather than mortality rates. If the contribution of lifestyle to the U.S. IHD mortality rate is, say, 30% of the total, and the contribution of lifestyle to the French IHD mortality rate is, say, 20% of the total, then to control for the contribution of lifestyle for each country you must reduce the U.S. rate by 30% but the French rate by only 20%. I don't think this is exactly hard to grasp.
January 13, 2008, 6:02 pmAri Tai:
re: IHD reduction.
Perhaps without reduction the results would have been ludicrous, v. just outrageous (difference between being usable by Mr. Friedman v. not).
We should also insist that studies that inform health policy meet, at a minimum, the every higher drug pharma bar. i.e. double blind tests, independent analysis, effectively 10 year cooling off periods. (only half in jest). What's good for the goose, ...
January 14, 2008, 4:23 amRobert Merkel:
How do you explain Australia's excellent performance, then, given that its "lifestyle factors" (along with Canada's) are probably the most similar to the USA amongst all the countries studied?
January 15, 2008, 9:10 pmCoyote:
Mahim -
If you do not understand the problem with the data adjustment for IHD, I am not going to spend my time catching you up on a lifetime of data analysis skills. Desperately wanting this study to be true and of high quality does not make it so. Suffice it to say that the folks who conducted the study, by halving all the numbers, changed only the weighting of IHD in the total index, they did not in any way, shape, or form correct for any variations in incidence rate. I suppose that you could say that reducing the weighting of a faulty variable makes the study better, but then by that logic they should have left it out entirely.
At the end of the day, to draw the conclusions they want to draw, the only way to do this study is to show each country with a numerator of total deaths from set X of diseases and a denominator of total incidents of set X of diseases, ideally corrected for differences in population age, etc.
The study does indeed show that other countries are healthier, say on the heart disease measure. It tells us nothing about why. The conclusion that the difference is in the differences in health care systems is not at all supported by the methodology or the data. In fact, data from other sources should make us strongly suspicious that differences in death rates has MUCH more to do with lifestyles and incidence rates and to some extent even differences in reporting methodologies.
By the way, I challenge anyone who has lived in both the UK or France AND the US to argue that the US health care is of substantially inferior quality to its European counterparts in terms of keeping one alive. It just does not pass the smell test. You might argue that the US system is more of a hassle, or carries more financial risks, but to say that it is worse quality certainly does not fit my experience from being in multiple countries.
January 15, 2008, 11:36 pmcoyote:
By the way Mahim, you are reading the chart wrong. I get it that you are backing out the 50% for the US to get an IHD death rate of 100, but then you have to do the same for France, to make it 40. The whole point is that the authors did not half just the US numbers for IHD, they halved everyone's
January 15, 2008, 11:38 pmguillaume:
Coyote,
January 16, 2008, 3:42 amYou and several others point to anecdotal evidence ("I was treated once 10 years ago at a hospital in Manchester") which, if you really want to stick with your "I'm the statistics expert" line, is crap.
The main difference between European health care systems and the US one is not to be found in the absolute performance of the best care provided by each, but in the ability of each system to treat everybody reasonably well (ie including the uninsured, the poor, the lower middle classes). And, like it or not, from that point of view, the US system sucks.
NickM:
To Ari Tai's question about the Australian health care system, it is not a simple comparison with the US system. When comparisons are done on preventable errors in hospitals, I have read figures which show that the litigous US system actually makes fewer errors than the Australian (and other)systems. Likewise, if you want or need world's best practice surgery using the latest hi-tech methods, you will certainly get it in the US. On the other hand, the US system is relatively expensive, and this might well mean some poorer Americans are not using the high quality services that are in theory available to them. Another important metric to look at is life expectancy, which in Australia is at 80.9 and running third to Japan and Switzerland. (see OECD 2005 figures) I suspect the good outcome in Australia is not so much the hospital system as it is the result of large scale public health programs aimed at prevention and early detection of serious conditions such as AIDS, Heart Disease, hypertension, cancer, diabetes and many others.
January 16, 2008, 8:32 pmJethro:
Plus, it's not just lifestyle that has to be compared across Australia and the US. It's common knowledge that African-Americans have higher rates of heart disease than European-Americans. African-Australians are kind of thin on the ground.
March 7, 2008, 8:34 am