Posts tagged ‘Free Republic’

Useful Reset: What Peer Review Is and Is Not

This is from a post of mine last January, long before the Climategate scandal  (though most of us who spent a lot of time with climate issues knew of the Climategate abuses long before the smoking gun emails were found).  The one thing this article does not mention is what we know today -- that climate scientists were actively working to keep skeptical studies out of the literature, even to the point of getting editors fired.

Peer review is not a guarantee of accuracy or a good housekeeping seal of correctness.  It is a process that insures a work is worthy of publication by a scholarly magazine.  Whether a scientific question is "settled" does not end with peer review, it only begins. It becomes settled after it survives decades of criticism and replication work, a process that was stonewalled by the folks at the CRU, which is really the heart of the scandal.

Yesterday, while I was waiting for my sandwich at the deli downstairs, I was applying about 10% of my consciousness to CNN running on the TV behind the counter.  I saw some woman, presumably in the Obama team, defending some action of the administration being based on "peer reviewed" science.

This may be a legacy of the climate debate.  One of the rhetorical tools climate alarmists have latched onto is to inflate the meaning of peer review.  Often, folks, like the person I saw on TV yesterday, use "peer review" as a synonym for "proven correct and generally accepted in its findings by all right-thinking people who are not anti-scientific wackos."

But in fact peer review has a much narrower function, and certainly is not, either in intent or practice,  any real check or confirmation of the study in question.  The main goals of peer review are:

  • Establish that the article is worthy of publication and consistent with the scope of the publication in question.  They are looking to see if the results are non-trivial, if they are new (ie not duplicative of findings already well-understood), and in some way important.  If you think of peer-reviewers as an ad hoc editorial board for the publication, you get closest to intent
  • Reviewers will check, to the extent they can, to see if the methodology  and its presentation is logical and clear "” not necesarily right, but logical and clear.  Their most frequent comments are for clarification of certain areas of the work or questions that they don't think the authors answered.
  • Peer review is not in any way shape or form a proof that a study is correct, or even likely to be correct.  Enormous numbers of incorrect conclusions have been published in peer-reviewed journals over time.  This is demonstrably true.  For example, at any one time in medicine, for every peer-reviewed study I can usually find another peer-reviewed study with opposite or wildly different findings.
  • Studies are only accepted as likely correct a over time the community tries as hard as it can to poke holes in the findings.  Future studies will try to replicate the findings, or disprove them.  As a result of criticism of the methodology, groups will test the findings in new ways that respond to methodological criticisms.  It is the accretion of this work over time that solidifies confidence  (Ironically, this is exactly the process that climate alarmists want to short-circuit, and even more ironically, they call climate skeptics "anti-scientific" for wanting to follow this typical scientific dispute and replication process).

Further, the quality and sharpness of peer review depends a lot on the reviewers chosen.  For example, a peer review of Rush Limbaugh by the folks at LGF, Free Republic, and Powerline might not be as compelling as a peer review by Kos or Kevin Drum.

But instead of this, peer review is used by folks, particularly in poitical settings, as a shield against criticism, usually for something they don't understand and probably haven't even read themselves.  Here is an example dialog:

Politician or Activist:  "Mann's hockey stick proves humans are warming the planet"

Critic:  "But what about Mann's cherry-picking of proxy groups; or the divergence problem  in the data; or the fact that he routinely uses proxy's as a positive correlation in one period and different correlation in another; or the fact that the results are most driven by proxies that have been manually altered; or the fact that trees really make bad proxies, as they seldom actually display the assumed linear positive relationship between growth and temperature?"

Politician or Activist, who 99% of the time has not even read the study in question and understands nothing of what critic is saying:  "This is peer-reviewed science!  You can't question that."

For those interested in the science of the skeptics position please see my recent movie.

Update: Mark Steyn on Climate and Peer Review

Phrase That Needs to Be Expunged From The Political Lexicon: "Peer Reviewed"

Yesterday, while I was waiting for my sandwich at the deli downstairs, I was applying about 10% of my conciousness to CNN running on the TV behind the counter.  I saw some woman, presumably in the Obama team, defending some action of the administration being based on "peer reviewed" science.

This may be a legacy of the climate debate.  One of the rhetorical tools climate alarmists have latched onto is to inflate the meaning of peer review.  Often, folks, like the person I saw on TV yesterday, use "peer review" as a synonym for "proven correct and generally accepted in its findings by all right-thinking people who are not anti-scientific wackos."

But in fact peer review has a much narrower function, and certainly is not, either in intent or practice,  any real check or confirmation of the study in question.  The main goals of peer review are:

  • Establish that the article is worthy of publication and consistent with the scope of the publication in question.  They are looking to see if the results are non-trivial, if they are new (ie not duplicative of findings already well-understood), and in some way important.  If you think of peer-reviewers as an ad hoc editorial board for the publication, you get closest to intent
  • Reviewers will check, to the extent they can, to see if the methodology  and its presentation is logical and clear -- not necesarily right, but logical and clear.  Their most frequent comments are for clarification of certain areas of the work or questions that they don't think the authors answered.
  • Peer review is not in any way shape or form a proof that a study is correct, or even likely to be correct.  Enormous numbers of incorrect conclusions have been published in peer-reviewed journals over time.  This is demonstrably true.  For example, at any one time in medicine, for every peer-reviewed study I can usually find another peer-reviewed study with opposite or wildly different findings.
  • Studies are only accepted as likely correct a over time the community tries as hard as it can to poke holes in the findings.  Future studies will try to replicate the findings, or disprove them.  As a result of criticism of the methodology, groups will test the findings in new ways that respond to methodological criticisms.  It is the accretion of this work over time that solidifies confidence  (Ironically, this is exactly the process that climate alarmists want to short-circuit, and even more ironically, they call climate skeptics "anti-scientific" for wanting to follow this typical scientific dispute and replication process).

Further, the quality and sharpness of peer review depends a lot on the reviewers chosen.  For example, a peer review of Rush Limbaugh by the folks at LGF, Free Republic, and Powerline might not be as compelling as a peer review by Kos or Kevin Drum.

But instead of this, peer review is used by folks, particularly in poitical settings, as a shield against criticism, usually for something they don't understand and probably haven't even read themselves.  Here is an example dialog:

Politician or Activist:  "Mann's hockey stick proves humans are warming the planet"

Critic:  "But what about Mann's cherry-picking of proxy groups; or the divergence problem  in the data; or the fact that he routinely uses proxy's as a positive correlation in one period and different correlation in another; or the fact that the results are most driven by proxys that have been manually altered; or the fact that trees really make bad proxies, as they seldom actually display the assumed linear positive relationship between growth and temperature?"

Politician or Activist, who 99% of the time has not even read the study in question and understands nothing of what critic is saying:  "This is peer-reviewed science!  You can't question that."

Really a Joke

I am finding Andrew Coyne's live blog of the Canadian hate speech "trial" to be endlessly fascinating.  Imagine taking the the most self-important but dysfunctional local school board you can find, give them a knowledge of court procedure and the rules of evidence mainly through watching People's Court reruns, and put them in charge of enforcing speech and censorship, and you will about have duplicated this proceeding.

Interestingly, the current evidence being entered in the proceeding seems to be blog comments made on non-Canadian blogs.  Every so often, we have to go through an educational process with the MSM to help them understand that commenters on blogs do not necessarily represent the opinion of the blogger.  It may be OK to use blog comments as evidence that the community at the Free Republic or the Democratic Underground are loony, but not to say that blogger X or Y is a racist because racist comments have been posted on his blog.

It appears that the government of Canada needs a similar education, but I can see this being hard to do.  Remember, each of the hearing "judges" are essentially people who make their living as government censors.  Their job is wiping out speech with which they do not agree.  It is therefore quite likely difficult for them to comprehend that many bloggers (like myself) have no desire to edit or control the content of our commenters.

Why I Don't Host This Blog on My Own Servers...

...Because there might come a slow news day in August when Tigerhawk, Hot Air, Pajamas Media, Reddit, the Free Republic, Ace of Spades, and many others all link to the same post at the same time.  In which case my servers here in the office and the poor hamster who powers them by running on his little wheel would be a smoking hole in the ground.

Scary Stuff

Most of you know I tend to avoid the topic of religion like the plague on this blog, but suffice it so say that I am a secular guy.  But that doesn't stop me from being scared of this guy (Chris Hedges at the Nation Institute):

This is the awful paradox of tolerance. There arise moments when
those who would destroy the tolerance that makes an open society
possible should no longer be tolerated. They must be held accountable
by institutions that maintain the free exchange of ideas and liberty.

The radical Christian Right must be forced to include other points
of view to counter their hate talk in their own broadcasts, watched by
tens of millions of Americans. They must be denied the right to
demonize whole segments of American society, saying they are
manipulated by Satan and worthy only of conversion or eradication. They
must be made to treat their opponents with respect and acknowledge the
right of a fair hearing even as they exercise their own freedom to
disagree with their opponents.

Passivity in the face of the rise of the Christian Right threatens
the democratic state. And the movement has targeted the last remaining
obstacles to its systems of indoctrination, mounting a fierce campaign
to defeat hate-crime legislation, fearing the courts could apply it to
them as they spew hate talk over the radio, television and Internet.

Whoa, Nellie.  The "forced to be free" thing never really works out very well, I promise.  I find the outright socialism preached by much of academia to be scary as hell and an incredible threat to me personally as a business owner, but you won't catch me trying to get the government to muzzle them.  Hedges attitude is consistent with opposition to school choice discussed here by Neal McCluskey of Cato:

Another frequent objection to letting parents choose their kids'
schools is that American children need to be steeped in a shared
worldview, lest they be in constant combat as adults. This arose as a
major line of argument in a Free Republic discussion about Why We Fight,
and is very similar to the "Americanization" mission given to
industrial-era public schools, where immigrant students were taught to
reject the customs and values of their parents' lands "” and often their
parents themselves "” and adopt the values political elites deemed
proper.

Now, if one were willing to accept a system that would, by
definition, quash any thoughts not officially sanctioned, then in
theory one would be okay with a public schooling system intended to
force uniform thought. In the context of an otherwise free society,
however, getting such a system to work is impossible, because
it would require that incredibly diverse and constantly combative
adults create and run an education system that somehow produces uniform
and placid graduates. It's no more realistic than hoping a tornado will
drop houses in a more perfect line than it found them.

The practical result of our trying to make uniformity out of diversity has, of course, been constant conflict, as Why We Fight
makes clear. Moreover, there is another by-product of this process that
no one mentions when they weave scenarios about choice producing
schools steeped in ignorance: our schools right now teach very little, especially in the most contentious areas like evolution and history, because they want to avoid conflict.

It all kind of makes a mockery of the left's favorite word "diversity."  One suspects what they want is for people of all color and backgrounds to come together and... think just like they do.  This seems to be part of the same strategy here to bring back the fairness doctrine.

PS- Remember, before you flame me, I am a secularist here defending the right of everyone to speak.  I am not defending Pat Robertson per se, because I almost never agree with the guy, but I am defending his right to say whatever he wants on TV.