Enshrining Peer Review as Part of the Scientific Method
I have written a lot about problems with over-emphasis on peer review and problems in scientific publishing. This is from a press release by the CRU quoting the highly flawed Muir-Russel review / whitewash of the Climategate emails.
We note that much of the challenge to CRU’s work has not always followed the conventional scientific method of checking and seeking to falsify conclusions or offering alternative hypotheses for peer review and publication. We believe this is necessary if science is to move on, and we hope that all those involved on all sides of the climate science debate will adopt this approach.
Because methodological challenges to scientists work that don't appear in Climate Journals controlled by the scientists in question are not part of the scientific method.
By the way, the statement that "The raw tree-ring data used in our published work are available; anyone is free to use them in any way they wish" is absolutely hilarious for anyone who has followed this saga over the years. To the extent they are available "freely," it is only because Steve McIntyre and other challengers of CRU's work engaged in a decade long legal campaign to get this publicly-funded data (necessary to verify and/or replicate the CRU's published work) released. Here is the McIntyre post to which CRU was responding, though they bend over backwards not to actually mention him.
Ted Rado:
Some of those posting on this blog are much enamored of peer-reviewd journals and denigrate any input from other sources. This blog, among others , serves a very useful purpose in giving us a place to exchange views on various subjects. Many of the comments are illuminating.
The whole landscape for scientific and technical information exchange has changed. No longer can someone with an impressive academic or government background make a statement and have it blindly accepted. I am sure this tightens up the thinking of those preparing a paper for publication.
What has gotten us into trouble in many areas (housing, energy, climate, etc.) is believing the written word unquestioningly. Had someone crtically examined many USG programs, much waste and misdirection would have been avoided. The current debate over climate change and renewable energy is a fine example.
Much of what is on the blogs is faulty, but at least there is the opportunity to question and discuss rather than accept mindlessly. It would be idiotic to go back to the old days and blindly follow every pronouncement in the journals or the USG.
May 29, 2012, 1:10 pmDavid:
Peer review is valuable. However, it isn't the acme of science. The acme of science is replication of experimental results. Anything which does not stand or fall based on replication of experimental results is not science in any rigorous sense of the word.
May 29, 2012, 1:19 pmme:
It's the usual defect of having an organization in which the theoretical counterweight is composed of the same people it's supposed to police. Think boards of directors, politicians, lawyers, teachers. What you want to set up if you care for quality are organizations with no links that have diametrically opposed goals with a system for competition (think independent testing organizations etc)
May 29, 2012, 4:55 pmAndrew_M_Garland:
Science and "science".
In 1964, Arno Penzias and Robert Wilson pointed a large Bell Labs radio receiver at the sky to calibrate it at zero signal. They found an annoying hiss that would not go away.
They began a long project of trying to eliminate that hiss as ground interference, atmospheric radiation, bad wiring, old circuits, you name it. Only after trying to rule out every possible earthly cause did they dare announce to the radio astronomy community that they were receiving a faint signal from space, uniform in all directions.
Their discovery led to confirmation and measurement of the Big Bang beginnings of the universe. The hiss is the remaining heat radiation throughout the universe as it has cooled from the time of the Big Bang 15 billion years ago.
That is real science. You criticize your own results in all possible ways before concluding that you have found something significant. Then, everyone else gets their chance to review what you have done.
Compare that to climate "science", where Michael Mann collected data from "proxies" such as old trees in Siberia, selecting them as he wished. Then he adjusted that data in complicated and unrevealed ways, until he produced a graph that would be interesting (if true), and immediately announced that he had represented the earth's temperature history for 400 years. There is no self criticism, no estimate of error, no search for other explanation, and no presentation of all his data and methods to the world. Further, he participated in suppressing publication of criticism in peer reviewed journals as indicated from discovered emails.
That is not science. It is trying to persuade the public by pretending to be science.
Peer Review Is Not What You Think
Shannon Love says this nicely [edited]
=== ===
"Peer Review" says nothing about conclusions. It is the fate of most scientific papers to be proven completely wrong.
Peer review protects a journal’s reputation. The journal hires experts to check for basic errors in math or methodology, along with grammar and spelling. It offloads responsibility for publishing bad papers onto anonymous scientists. It is a form of blame-passing that everyone would like to use. It does not confirm or refute experimental or theoretical conclusions.
Some people will say that a scientific result is true because it appears in a peer reviewed journal. That is the weakest defense possible. It means only that some editor and his reviewers found it to meet their minimum quality standards for publishing. It meets no standards if the editors and peer reviewers are corrupt.
May 29, 2012, 10:57 pm=== ===