Some Thoughts on Peer Review
Some thoughts on the obsession with peer review as the gold standard guarantee of climate science goodness, from Climate Skeptic:
One of the weird aspects of climate science is the over-emphasis on peer
review as the ne plus ultra guarantor of believable results. This is absurd.
At best, peer review is a screen for whether a study is worthy of occupying
limited publication space, not for whether it is correct. Peer review, again at
best, focuses on whether a study has some minimum level of rigor and coherence
and whether it offers up findings that are new or somehow advance the ball on an
important topic.In "big
boy sciences" like physics, study findings are not considered vetted simply
because they are peer-reviewed. They are vetted only after numerous other
scientists have been able to replicate the results, or have at least failed to
tear the original results down. Often, this vetting process is undertaken by
people who may even be openly hostile to the original study group. For some
reason, climate scientists cry foul when this occurs in their profession, but
mathematicians and physicists accept it, because they know that findings need to
be able to survive the scrutiny of enemies, not just of friends. To this end,
an important part of peer review is to make sure the publication of the study
includes all the detail on methodology and data that others might need to
replicate the results (which is something climate reviewers are particularly bad at).In fact, there are good arguments to be made that strong peer review may even
be counter-productive to scientific advancement. The reason is that peer
review, by the nature of human beings and the incentives they tend to have, is
often inherently conservative. Studies that produce results the community
expects often receive only cursory scrutiny doled out by insiders chummy with
the authors. Studies that show wildly unexpected results sometimes have trouble
getting published at all.
As I read this, it strikes me that one way to describe
climate is that it acts like a social science, like sociology or gender studies,
rather than like a physical science. I will ahve to think about this -- it
would be an interesting hypothesis to expand on in more depth. Some quick
parallels of why I think it is more like a social science:
- Bad statistical methodology (a hallmark, unfortunately, of much of social
science) - Emphasis on peer review over replication
- Reliance on computer models rather than observation
- Belief there is a "right" answer for society with subsequent bias to study
results towards that answer (example,
and another
example)
ErikTheRed:
My snarky answer to the peer-review-obsessed is that when it comes to AGW (and some other fields), "peer-reviewed" really means "politically correct" and that the blatant censorship that goes on is something you'd expect from fascists like those in Bush Administration. For bonus points, construct an argument based on the validity of your feelings.
It's not intellectually or logically rigorous, but it's enjoyable to turn their defective methodologies against them and watch their heads explode. Another fun thing to do is let them ramble on for 10 or 15 minutes, then respond by rolling your eyes, saying "fuck you," and walking away. If they're going to act like brain-washed zombies, you might as well enjoy yourself.
August 6, 2008, 3:26 pmSB7:
That reminds of one comment left in this Marginal Revolution post explaining why some fields are more PC than others:
"Careers for professors in some fields is driven by surprising your peers, in others by agreeing with your peers."
I think we can put climate science in the latter category along with English, Sociology, Psychology, etc.
August 6, 2008, 3:35 pmRoy Lofquist:
The problems with peer review also appear in some of the "harder" sciences. The Halton Arp story is illustrative. He is a world famous astronomer, an associate of Fred Hoyle, winner of numerous awards. His "Catalog of Peculiar Galaxies" is in widespread use in the field. He is also a heretic. His theories, supported by observational evidence, contradict the standard cosmological model (the Big Bang). They are well known and widely discussed. However, no peer reviewed journal in the field will publish his work. He has, however, been published in peer reviewed physics journals - just none that deal with astronomy or cosmology.
August 6, 2008, 4:05 pmAnon:
Kind of. First, though, the computer modeling is a red herring. It's not even a well-defined term. What the heck is a "computer model"? Is it a model that you run on a computer? Well, then F=ma must be a computer model, because it's used a gajillion times a day on computers all over the world. What about Navier-Stokes? Is that a "computer model"? Computers are used simply when there are too many numbers to use a calculator. That's usually because you are trying to apply the fundamental equations, like F=ma, over a non-trivial spatio-temporal domain.
For those of you who took freshman physics, remember how problems always assumed things like perfect spheres, points, etc.? That's because the calculations for many real world problems require a computer. But the fundamental equations are the same.
So, there is plenty of observation in climate science. There are sensors everywhere...observing. What is lacking is experimentation. For many sciences, such as many branches of physics, you can tweak things. You can hold some variables constant, and vary others to your hearts content. For other kinds of science, you cannot, for various reasons. For example, you can't resolve the nature vs. nurture debate once and for all by raising children under different conditions. It would be unethical.
Similarly, you can't remove 5% of the CO2 in the atmosphere to see how the climate changes, or add 5%. We can't increase the output of the sun by 100 W/m^2, to see the effect, etc.
August 6, 2008, 11:17 pmlinearthinker:
Anon:
Objection to computer modeling as expressed by climate skeptics is not a red herring. The rest of your post is bullshit.
----------
Regarding another pitfall to peer review, see
Climate Sensitivity and Editorial Policies, Lubos Motl, 5-12-06.
Lumo discusses the filtering of peer commentary by editorial agenda.
August 6, 2008, 11:52 pmJody:
I don't see where anon's point that you can't (or at least it's very difficult to) conduct real experiments is BS - I think that's an important point as to why climate science cannot ever be "big-boy" science.
August 7, 2008, 4:52 amdr kill:
I think everyone involved in real science knows the difference between 'peer-reviewed' and 'refereed'. And stop referring to Sociology and the like as science of any kind.
August 7, 2008, 5:38 amAnon:
Well linearthinker, do you care to actually make an argument of any kind, or do you prefer debating by assertion?
I'm not saying that computer modeling is a good thing, I'm saying that the fact that climate scientists use computers to do their calculations does not make it more like a social science, because plenty of hard sciences use lots of computers. Ever heard of CFD? Ever modeled the heat transfer in a plate? That's a typical introductory exercise in a scientific computing course. I am currently working with a cardiac bioengineer on his heart model. (I'm a CS.) He calls it a "model", but he is not in the social sciences nor is he in climate science. His model will run on a computer, because there is simply no way that you can could use a calculator to simulate a real heart.
In the heat-conduction-in-a-plate that is typically given as an exercise in courses, there is a "model" of the real world. This model is the heat equation. The question now, is how accurate is this model? Well, one way to validate the model is to try it out. So, you run the model numerically. Then you take a real plate and do the same thing to the real plate that you did to the model in the computer. If the results match, then it's evidence that the model is valid. To further validate the heat equation, you vary the temperature, vary the shape, vary the material. Once it is validated in enough different ways, it becomes generally accepted.
Note that the heat transfer equation really is a model. In reality, heat transfers by molecules bouncing around. But the heat equation does not go into such detail. It is a model of the real world.
Now, let's see if we can apply the same thing to the climate. Well, we can construct a model. We can do the simulations. Immediately there is one problem. To do the simulations at a fine granularity requires an enormous amount of computational power. But let's say we do them anyway.
Okay, so the simulation is done. Is the model valid? Well, you validate by comparing to the real world. So, let's just find a convenient Earth, set the initial conditions, let the climate go for 100 years, and see what happens. Hmm...can't do that. You can easily do that with a metal plate, when validating the heat equation. Can't do that when validating your climate model.
So what do scientists do? Well, they try to find situations in the past, then run the model using historical information. If the model predicts the same thing as what really happened, then that suggests the model is valid. (Note that we don't say "correct." Modeling is not about whether or not something is "true", but rather whether or not something is predictive, or "valid".) Problem is that they only have ONE experiment, because there is only one Earth. Further problem is that a lot of the data must be inferred. Validation thus becomes very problematic and challenging.
Further complicating the situation is the fudge factors. When modeling heat transfer using the heat equation, we are dealing with molecules, at the fundamental level. Compared to the climate, the system is simple. When modeling the climate, however, there are many complex factors and non-linearities. For example, when the heat increases, molecules don't change, they only jump around faster (within the valid temperature range of the heat equation). But when the climate changes, plants grow differently, for example. It is very difficult to accurately model plant growth for the entire Earth. To do requires finding some fundamental equations, then discretizing the entire Earth to a fine enough granularity. All very challenging. So fudge factors are used.
Now, IF you can conduct zillions of experiments, you can pretty accurately figure out the fudge factors. You can get pretty accurate models. For example, there are pretty good models of chemical reactions, even if they are not ab initio. That's because they can run a zillion experiments to validate the model. Problem is that climate scientists can't do that.
Note that a similar situation applies to the social sciences. We can model how a person behaves, but we can't very easily validate the model. We cannot construct real economies to test hypotheses, for example. Similarly, we cannot construct man-made real climates in order to test the theories.
Note that no where have I defended climate science. I don't really have any opinion on that, because I do not follow it and have not read the literature.
August 7, 2008, 7:26 amAnon:
And not that I believe in "argument by authority", but for the record, I have Ph.D. in CS at a respectable university. Overall it is not top 20 in CS, but it is strong in grid computing and scientific computing. I am a PI or co-PI on 5 NSF or DOE grants, mostly dealing with what the NSF calls cyberinfrastructure. My DOE grant is in the SciDAC2 (Scientific Discovery through Advanced Computing) program. I have been on five different NSF review panels (all CS related, some were interdisciplinary with CS), reviewing proposals. I have never reviewed one on climate science, but have reviewed a couple on mesoscale weather that involve interdisciplinary research with CS. I am currently a faculty member in CS at a Ph.D.-granting university.
I don't know much about climate science specifically. I have skimmed some of the papers, and thought to myself: "Boy, there sure is a lot of fudging." But on the other hand, I understand the challenges they face.
August 7, 2008, 7:38 amAnon:
Lastly, I do agree that peer-review doesn't mean that much, at least in my areas. I do lots of peer-review, and have lots of peer-reviewing done to me. Though I can't really speak for climate science, in my area, there are crappy NSF proposals that get funded, and plenty of crappy, peer-reviewed papers that get published.
August 7, 2008, 7:41 amPaul McLellan:
When I was doing a PhD in computer science, someone (I forget who now) pointed out that you should never trust sciences that had to put the word science in them. "Social science", "Political Science", "Computer Science". Real science is called Physics, Biology etc not "Physical science". I think "climate science" can be added to the list.
August 7, 2008, 8:04 amAnon:
Okay, one last post. As further evidence that computer modeling as practiced in climate science is not evidence that it is like a social science, here is a link to recent NSF awards in the PetaApps program. You'll see some climate science awards, but I didn't see any in the social sciences.
Also, here is a link to the DOE SciDAC (Scientific Discovery through Advanced Computing) program. You'll see climate along with other "hard" sciences, but no social sciences.
Both of these funding programs have "computer modeling" as the basis for most, if not all, of the awards.
August 7, 2008, 8:05 amLeonard Huff III:
Put 10 geologists (sciensits) in a room, all of them looking AT 3-D sesimic , well controls logs, geology maps of wells and asking each one were TO spot a location for the drilling of A well that cost $15,000,0000 to drill & complete and you will get 10 DIFFERENT places TO STAKE LOCATION!
I have FIRED ALOT of GEOLOGISTS due to EXPENSIVE DRY HOLES!!!!!!
CAN'T FIRED AL GLORE ( WARMING WIZARD!)
GO FIGURE!!!!
August 7, 2008, 8:35 amAnother guy named Dan:
Anon - It's not the use of computer models per se, but how they are used that is the problem.
If a particle physicist builds a model that states that a particle with certain properties will behave under certain conditions in a manner inconsistent with observation, he either changes the model or discards it all together.
In the "soft sciences" along with climate research, there is a tendency to "re-interpret" observations to bring them into the range that the model would predict. Thus the historical average temperature series for the US gets changed on an anual basis etc.
August 7, 2008, 9:00 amsohbet odaları:
thanks a lot
August 7, 2008, 10:00 amDavid Zetland:
First, I agree that a lot of the "science" in social sciences is really opinion. (I'm an economist.)
Second, peer-review requires that the author makes a reasonable case for their conclusions. It doesn't mean that they are "right."
Third, a reasonable case in math will probably stand forever; in physics, it can get knocked down by replication; in climate modeling (or other modeling), it will be subject to the calibration errors (data and parameters) -- some of which are subjective/biased; in humanities, it's all just opinion.
That said, publication is useful as a means of extending the debate (and getting paid) and providing something for others to work with. Publication is NOT the end of the debate.
August 7, 2008, 10:36 amlinearthinker:
Anon: First, though, the computer modeling is a red herring.
LT: Objection to computer modeling as expressed by climate skeptics is not a red herring.
Anon: Well, then F=ma must be a computer model, because it's used a gajillion times a day on computers all over the world. What about Navier-Stokes? Is that a "computer model"? Computers are used simply when there are too many numbers to use a calculator. That's usually because you are trying to apply the fundamental equations, like F=ma, over a non-trivial spatio-temporal domain. etc. etc.
[You left out Reynolds Number, Pitot, Galileo, Torricelli, Mariotte, Pascal, Bernoulli, Euler, et al. You must have been in a hurry. I understand. You're a busy fellow, as you've pointed out.]
LT: The rest of your post is bullshit.
Which it is. You never come back around to telling us why you think the computer modeling is a red herring. All you do is tell us how important computer modelling is, has been, and will probably always be, which is a nonissue. No responsible skeptic that I've read in the last four or five years has ever come close to saying what you're implying [they say], i.e., that the models themselves are the problem. What they are challenging is both the non-critical acceptance of the models by AGW proponents both in the sciences and in the political/entertainment sphere, and the stonewalling, obfuscation and name calling by the "climate scientists" when their methodology and conclusions are criticized.
Read carefully what I first said. I have, and I'll acknowledge that it could have been fleshed out just a bit so as to it read:
Objection to the uncritical acceptance of the AGW proponents' computer modeling as expressed by climate skeptics is not a red herring.
Appears to me neither of our statements were adequately clear.
Another guy named Dan seems to have captured the essence where I failed, at least in your case.
Anon - It's not the use of computer models per se, but how they are used that is the problem.
Thank you, Dan.
And thank you, anon, for sharing with us your impressive c.v. I'll rest easier tonight knowing science is in such well credentialed hands.
LT
August 8, 2008, 1:11 amlinearthinker:
This comment is for Leonard Huff III.
I was in the hallway where two senior engineers were being introduced to a new geologist on the staff. After the cordialities were exchanged, the more blunt of the two senior guys said, "I only hire one armed geologists."
He left it hang for a while, then he continued, "Every time I ask a geologist for an answer, he tells me one thing, and before I can reply, he says, but on the other hand."
August 8, 2008, 1:28 amlinearthinker:
Borrowing a thought from Gerard Vanderleun:
"It is impossible to speak in such a way that you cannot be misunderstood." -- Karl Popper
LT
August 8, 2008, 1:45 amDavid Zetland:
linearthinker -- the one-armed economist came first, via Truman :)
August 8, 2008, 8:09 amJohn Moore:
There are enough problems with climate sensitivity models to render the conclusions being loudly trumpeted to be very suspect.
This does not mean computer models are not real science. In fact, many phenomena in, for example, biology, chemistry, weather and yes climate are simply too complex to be comprehended by the reductionist approach that works so well for physics: reducing the problem to relatively simple equations or systems of equations. In the future, more "understanding" in science will be the result of computer models, not human understandable equations. As an example of this sort of model, consider that an important factor in determining your credit worthiness (in the credit card market) is the output of a neural network. Nobody has any idea how it makes its recommendations - it just does. The network understands credit risk in a way people do not.
Climate models are weak because, well, they are weather models, and weather models are weak. Basically, they cannot model every "billiard ball" in the game - there are too many molecules around. So they rely on equations that are generalizations. But the equations are coupled in complex, nonlinear fashions. On top of that, many of the coefficients are guestimates - which in the climate world are pretty hard to calibrate.
As one who uses weather models for forecasting, I find it fascinating that the highest resolution models can be very, very wrong just a few hours into the future. For example, the question of whether an isolated thunderstorm will form is often impossible to answer.
These same models (or ones like them) are used to evaluate climate sensitivity to CO2(or other factors). But many important phenomena in weather/climate take a lot longer than a couple of hours to operate - certainly longer than the competent forecasting time of the models. For example, the coupling of the Madden-Julian oscillation to western US precipitation is one that takes many days. Oops... we'll just have to parameterize that one. The sad thing is that almost everything in the problem is like that.
Also, I consider evolution to be an awfully good hypothesis, but experimentation, in the way some people mean it, has not been possible. Science is not about experimentation, it is about hypothesis formation and refutation. Sometimes an "experiment" provides evidence about the hypothesis. Other times the hypothesis may predict what will be determined about the past. This is certainly true in evolutionary biology (less so in the last decade or two). If your theory says that certain kinds of things happened in the past, and later evidence is found for that, you have the equivalent of experimental evidence.
August 8, 2008, 5:33 pm