Phrase That Needs to Be Expunged From The Political Lexicon: "Peer Reviewed"
Yesterday, while I was waiting for my sandwich at the deli downstairs, I was applying about 10% of my conciousness to CNN running on the TV behind the counter. I saw some woman, presumably in the Obama team, defending some action of the administration being based on "peer reviewed" science.
This may be a legacy of the climate debate. One of the rhetorical tools climate alarmists have latched onto is to inflate the meaning of peer review. Often, folks, like the person I saw on TV yesterday, use "peer review" as a synonym for "proven correct and generally accepted in its findings by all right-thinking people who are not anti-scientific wackos."
But in fact peer review has a much narrower function, and certainly is not, either in intent or practice, any real check or confirmation of the study in question. The main goals of peer review are:
- Establish that the article is worthy of publication and consistent with the scope of the publication in question. They are looking to see if the results are non-trivial, if they are new (ie not duplicative of findings already well-understood), and in some way important. If you think of peer-reviewers as an ad hoc editorial board for the publication, you get closest to intent
- Reviewers will check, to the extent they can, to see if the methodology and its presentation is logical and clear -- not necesarily right, but logical and clear. Their most frequent comments are for clarification of certain areas of the work or questions that they don't think the authors answered.
- Peer review is not in any way shape or form a proof that a study is correct, or even likely to be correct. Enormous numbers of incorrect conclusions have been published in peer-reviewed journals over time. This is demonstrably true. For example, at any one time in medicine, for every peer-reviewed study I can usually find another peer-reviewed study with opposite or wildly different findings.
- Studies are only accepted as likely correct a over time the community tries as hard as it can to poke holes in the findings. Future studies will try to replicate the findings, or disprove them. As a result of criticism of the methodology, groups will test the findings in new ways that respond to methodological criticisms. It is the accretion of this work over time that solidifies confidence (Ironically, this is exactly the process that climate alarmists want to short-circuit, and even more ironically, they call climate skeptics "anti-scientific" for wanting to follow this typical scientific dispute and replication process).
Further, the quality and sharpness of peer review depends a lot on the reviewers chosen. For example, a peer review of Rush Limbaugh by the folks at LGF, Free Republic, and Powerline might not be as compelling as a peer review by Kos or Kevin Drum.
But instead of this, peer review is used by folks, particularly in poitical settings, as a shield against criticism, usually for something they don't understand and probably haven't even read themselves. Here is an example dialog:
Politician or Activist: "Mann's hockey stick proves humans are warming the planet"
Critic: "But what about Mann's cherry-picking of proxy groups; or the divergence problem in the data; or the fact that he routinely uses proxy's as a positive correlation in one period and different correlation in another; or the fact that the results are most driven by proxys that have been manually altered; or the fact that trees really make bad proxies, as they seldom actually display the assumed linear positive relationship between growth and temperature?"
Politician or Activist, who 99% of the time has not even read the study in question and understands nothing of what critic is saying: "This is peer-reviewed science! You can't question that."
Matt:
So you could say that 'peer review' acts as a positive feedback loop, much the same as the the loop involving the degree that GW alarmists claim additional CO2 molecules trap sunlight?
January 29, 2009, 12:25 pmMike C.:
An excellent point. IIRC, the last paper arguing that plate tectonic theory was wrong (in the AAPG Bulletin) was published in 1969. It was a hell of a paper, covering a lot of stuff in great detail, marred only by being wrong.
January 29, 2009, 1:05 pmChuck:
I think it was physicist Paul Dirac who once criticized a presentation by saying "That's not even wrong", and that is the main function of peer review. To ensure that a publication contains sufficient detail and clarity so that it falls into "the set of all things that are either right or wrong", and not into "the set of all things that make no sense whatsoever".
January 29, 2009, 1:53 pmLarry Sheldon:
I am not an Academic, although I did for some years draw a salary from a self-styled University....
I am under the impression that "peer review" has as its primary purpose policing papers to ensure that the dogma of the guild has not been sullied, and that everything said has been said before.
As contrasted with a scientific paper where the information provided is intended to encourage replication and verification by others or, conversely, to encourage strength-testing by attempts to falsify the information presented..
January 29, 2009, 2:17 pmDr. T:
I have been a reviewer for six medical journals. The scientific and medical journal peer-review process doesn't work most of the time. The journal must have great editors for peer-review to work well, and those editors must oversee a stringent peer-review process. The stringent process starts with careful selection of reviewers for different subjects, telling them what their duties are, and ensuring that they follow the reviewer guidelines. Reviewers who don't follow the process must be dropped. Also, most reviewers are not knowledgeable about statistics. Submissions with complex statistics should be reviewed by a journal-employed statistician. Few journals establish or follow this type of process, so peer-review overall is mediocre.
Climatology is a special case. Peer-review is worthless in climatology, because almost no climatologists are scientists. They are anti-scientists. They should observe a phenomenon, make a hypothesis about it, design an experiment to test the hypothesis, evaluate the experimental data, and accept or revise their hypothesis. Instead, they observe a phenomenon (increased mean temperatures on land for 9 years), make an unsupported generalization (global warming), assign a cause (burning of fossil fuels), selectively mine other climate data, create a bad mathematical model, call a press conference, and demand radical lifestyle changes to prevent Waterworld.
What I find most frustrating is the fact that there is no evidence whatsoever that the greenhouse effect applies to the open atmosphere. My hypothesis is that in an open system, increased atmospheric CO2 concentrations have almost no effect on global temperatures. Climatologists hypothesize that the magnitude of the greenhouse effect in the atmosphere is at least as great as the effect in a sealed greenhouse. But, they have never performed experiments to verify their hypothesis. Grrr.
January 29, 2009, 6:24 pmDov Henis:
Peer Review In Science, Whereto?
A. "Google peer review"
http://www.the-scientist.com/community/posts/list/380.page#1497
Seeks comments on "crowd-sourcing the peer review process".
Even though in this case the Google code is just a site that hosts other people's projects, the developers just using Google's hosting, in essence Google enables and promotes this... prudent prospective business development...
B. Enhancing Peer Review (NIH/OD)
Sent:Jan 03, 2009
To: henisdov
Subject:Peer Review And Innovation
Dear Dov Henis,
Thank you for your input. Please be aware that a significant part of
the process to Enhance Peer Review is "Continuous Review of Peer
Review." We will keep your comments under consideration.
The Enhancing Peer Review Team
http://enhancing-peer-review.nih.gov
"...fund the best science, by the best scientists, with the least amount
of administrative burden."
C. From earlier postings of mine:
"Peer Review" is, factually, a tool of a "Subversive Activities Control Board", a tool of the Science Establishment.
The most revolting corrupt aspect of peer review in science is its exploitation by the Science Establishment to tightly clamp its political and financial omni-everything power, rule, control and censorship, including stifling of any shred of scientific innovation not originated or not publicized via The Establishment's network.
The corruption is not inherent in the tool, but in the nature of the Science Establishment and in the characteristics of its obsequious functionaries.
"Implications Of Science And Technology Evolution"
http://blog.360.yahoo.com/blog-P81pQcU1dLBbHgtjQjxG_Q--?cq=1&p=419
The AAAS and its affiliates and equivalent organizations, i.e. all the Science Establishment Guilds, loudly and pseudoscientifically promote and "profoundly respect" the "spiritual religious domain" as a real domain, a domain separate from the real science domain. They do this both because, unbelievably, they actually believe it and as a politically powerful tool in promoting towards themselves public esteem-acceptance and state-public financial support.
The peer review process is a tool of the Establishment. As long as the Science Establishment Guilds are what they now are and as long as they enjoy the public and state esteem they now have, and as long as Science and Technologhy are not seperated from each other conceptually and administratively but considered and handled as one faculty, the corruption of science cannot and will not be overcome. These 21st century technology culture aspects have been well inscribed and entrenched by Science's Big Brothers in the minds and conceptions of the public.
D. The lesson of the wikipedia-encyclopedia's "public-sourcing" product,
and of additional currently on-line "outside-Establishment-sourcing" science publishing start-ups
is discouraging. IMO the GPeerReview is destined to end up similarly, i.e. as a cheaper, more available, second-hand product proudly claimed to be produced with the Establishment's rules-regulations-considerations even if "made in China" for faster and cheaper delivery and for more wide-spread availability.
Thus science publishing delivery will become faster and cheaper but the products will be made and cleared with today's combination of writing rules-regulations and peer-review ecclestiasticism that have been so deeply inscribed in the flock of the 21st century Science Establishment Church.
Dov Henis
(A DH Comment From The 22nd Century)
February 12, 2009, 9:25 amhttp://blog.360.yahoo.com/blog-P81pQcU1dLBbHgtjQjxG_Q--?cq=1