"It's peer review, not God review" My wife's comment about peer review seems particularly apt for the current tempest regarding the resignation of Wolfgang Wagner from his post as editor in chief of the journal Remote Sensing. It regards a paper I mentioned in July, and related to the one that prompted Barry Bickmore to suggest "Just Put the Model Down, Roy".
I won't be taking the usual line of consideration here (surprise!). Rather, let's go back to talking about peer review. As my wife said, it is not God review -- reviewers and editors are human, and therefore make mistakes. At the end of peer review, therefore, you don't have gospel, you have something that has a fairly good chance of being worth your time to read. To rephrase Wagner, papers that pass peer review should at least not contain fundamental errors of method or false claims. And it now seems likely to him that this paper (Spencer and Braswell) may well not pass that standard.
Richard Feynman's comment about fooling yourself is commonly quoted:
We've learned from experience that the truth will come out.
Other experimenters will repeat your experiment and find out whether you
were wrong or right. Nature's phenomena will agree or they'll disagree
with your theory. And, although you may gain some temporary fame and
excitement, you will not gain a good reputation as a scientist if you
haven't tried to be very careful in this kind of work. And it's this
type of integrity, this kind of care not to fool yourself, that is
missing to a large extent in much of the research in cargo cult science.
"Cargo Cult Science", adapted from a commencement address given at Caltech (1974)
This underlies some parts, I think, of the failure in Spencer and Braswell's work, and the subsequent failure in the review and editorial process. Namely, it is assumed by reviewers and editors that authors have already done some work doubting themselves and checking to see how it is they might have fooled themselves -- and to take action against such possibilities. Further, the review process is based on the presumption that your purpose in publishing is to advance our understanding of science.
Wagner mentions one of those non-scientific aspects to the paper -- the public exaggeration of the paper's conclusions by the author himself. What the university press releases and certain media outlets do in exaggerating is pretty much outside the author's control. But Spencer's own exaggerations indicate (my thought) that his purpose was to make those grand claims, rather than to make an incremental improvement to our knowledge of how the universe works. As Wagner mentions, no single paper looking at a single data source, comparing with a single model, is capable of refuting all science on global warming.
A different non-scientific point, endemic to any of Spencer's work, is that he doesn't fundamentally view himself as engaging in science and an effort to understand the universe; he says "I view my job a little like a legislator, supported by the taxpayer, to
protect the interests of the taxpayer and to minimize the role of government." Note that -- his job. There are scientists who pursue political goals, whether to minimize government, or prevent what they think are bad political decisions -- but it's on their own time, as private citizens -- not as their job. The particularly active ones I know have always had a clear distinction between their job and their activities as a private citizen.
With Spencer, though, he views is job as being to achieve a political end. As such, science takes a back seat. This paper of his, then, is less a matter of trying to advance our understanding of the world, and more (as he views it as his job) a matter of trying to achieve a political result. Peer review was not developed for such things, and does not manage them well. When science is not the primary purpose of a paper, that presumption reviewers and editors make that the author has done his due effort at avoiding fooling himself is false.
Some comments have been made hither and thither about how the criticisms, like Bickmore's, of Spencer's work are not in the peer-reviewed literature. They then conclude that this means that Spencer is right and Bickmore (and others) are wrong, or the criticisms can, at least be ignored. If the authors have done their due efforts to not fool themselves, this may well be true. In such efforts the fundamental errors Wagner mentions would not be present.
Instead, as was the case for McLean et al., some errors were so elementary that they could be analyzed and discussed at blog-level. Granted I may often leave some of the middle-school kids behind, but Bickmore's analysis, for instance, should have been manageable by anybody with an undergraduate major in a math/science/engineering field.
There's a flip side to that. You seldom can publish things at that level in scientific journals. They're 'too obvious', or 'uninteresting', and the like. Consequently, if the errors in a paper are elementary enough, there will be no peer-reviewed literature to cite regarding the error. Authors are supposed to have done that checking themselves, and if they fail to, the reviewers are supposed to mention them. If both fail, well, now we do have the possibility of writing blog comments on the failings of a paper.
For comments and discussion here, I will encourage you to focus on the peer review process itself. I give links below for the science in the paper, and others for the story itself if you're more interested in those things.
Some links on the science:
http://www.realclimate.org/index.php/archives/2011/07/misdiagnosis-of-surface-temperature-feedback/
http://profmandia.wordpress.com/2011/08/03/spencer-braswell-2011-proof-that-global-warming-is-exaggerated-or-just-bad-science/
http://bbickmore.wordpress.com/2011/07/26/just-put-the-model-down-roy/
Some additional links on the story:
http://initforthegold.blogspot.com/2011/09/editor-apologizes-for-spencer-paper.html
http://www.skepticalscience.com/news.php?n=982
http://scienceblogs.com/deltoid/2011/09/editor_of_remote_sensing_agree.php?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+scienceblogs%2Fdeltoid+%28Deltoid%29
http://www.forbes.com/sites/petergleick/2011/09/02/paper-disputing-basic-science-of-climate-change-is-fundamentally-flawed-editor-resigns-apologizes/
http://retractionwatch.wordpress.com/2011/09/02/editor-of-remote-sensing-resigns-over-controversial-climate-paper-co-author-stands-by-it/
http://scienceblogs.com/stoat/2011/09/holy_editor_resignation_batman.php
http://rabett.blogspot.com/2011/09/honor-and-respect.html
http://blog.chron.com/climateabyss/2011/09/spencer-braswell-and-the-review-process/
http://bigcitylib.blogspot.com/2011/09/editor-steps-down-for-publishing.html
http://scienceblogs.com/gregladen/2011/09/cloudgate_denialism_gets_dirty.php
Spencer's blog and his comments on the news
03 September 2011
Subscribe to:
Post Comments (Atom)
Posts
6 comments:
Regarding Spencer's viewing himself as a legislator: Aren't legislators typically elected, rather then hired? (Let alone hired for a totally different position.)
(Saint) Peter Review
-- by Horatio Algeranon
If Peer Review
Were Peter Review
Would the rejected
Join Satan too?
(if you get this twice, my apologies - just delete one or the other)
Along the lines of what you've said, I think papers of irredeemably low quality just end up ignored. You read it, you realise the authors made some basic errors, you sigh, and you leave it aside. One might give it to a beginning grad student as an exercise in critical reading. Years later, you'll see that the paper has gone uncited, and was generally ignored. But what do you do when certain elements in society and the media seize upon the paper? In this case, a formal reply or comment seems to be called for. But it isn't necessarily easy to get a comment published, even if the comment has merit, and even if the journal will take a comment, it may have to be short. So it's hard to see what the best path is.
Are there useful analogues one can study from other fields, such as stratospheric ozone chemistry or evolutionary biology?
Anders:
Spencer did elaborate that he didn't think he _was_ a legislator, just that he takes his mission to be something _like_ a legislator. I think your observation still holds. I want my scientists to be doing science on the job, not politics. (Since Ray is working on federal grants, and I pay taxes, he's one of 'my' scientists.)
carrot:
As you say, mostly they're ignored. I mentioned that too in my earlier peer review note.
The usual comment/reply mechanism is, like the review process itself, aimed at papers which are written on the aim of advancing our scientific knowledge. And they've done the responsible effort to avoid fooling themselves.
When an author considers his primary purpose to be political, rather than scientific, those presumptions fail.
The usual comment must have some real substance of its own. That's because the original paper should be substantive itself. One can have difficult issues in science, where it is hard to say what is the right way to go. The original paper may not be right, but after making serious effort at not fooling themselves, and the reviewers looking to ensure that this is the case, it enters the scientific discussion. Part of that discussion can be a reply, picking up one of those points that is hard to decide and making the case that it is important enough to cause reconsideration of the paper. I'm finishing formatting of a paper in which the back and forth between original author and commenter is useful in laying out a point in favor of my (brilliant of course) idea.
When, instead, the level of error includes things that undergraduates in the area would get downgraded for, now what? There's no real standing mechanism for what and how to deal with that. Historically, the desire to publish was a desire to share one's discoveries about the natural world -- not a matter of furthering your political agenda. So mechanisms are lacking. It looks like science blogs are developing as one of the means of response. But probably another decade (to make up a number) before it gets to be as normal and worked out a route as the review process itself (something which evolved in response to the expanding population of scientists).
For analogues, one which occurs to me is S. Fred Singer regarding the Ozone Hole. His comments on that were basically ignored, except in some parts of the media. But those parts of media did not have the power they do today. And there was a much simpler scientific smoking gun to be found. So much simpler that, as I recall it, Singer eventually did allow as how there really was an ozone hole and it really did work as was said much earlier by scientists. It only took a decade or so after the time the science was nailed. This is the same S. Fred Singer who has spent the last 20 or so years making similar claims against climate change.
Maybe you can track down something useful on that example?
Bob, Eli thinks you are dead wrong that university press releases are pretty much outside the author's control. In practice the author brings the "hot new" paper to the press office and works on it with them.
If the release gets picked up the author will be called by some of the pickers uppers for a quote or more, so even that is not always outside the author's control
S. Fred was getting letters published in Science on the ozone issue well into the 1990s. That is pretty tough territory
Post a Comment