07 March 2010

What should be reproducible?

I'll make the assumption that astrophysics is science.  That shouldn't be terribly surprising, given both that folks tend to talk about it as being science, often as being a particularly beautiful branch of science, and that I've studied it myself.  But starting from that assumption suggests that either astrophysics is not science after all, or that many of the complaints about climatology are ill-founded.  At least in the sense that if they should be taken seriously (speakers rejecting the idea that climatology is science) then astrophysics, and quite a lot of other sciences, should be rejected as well.  Now I do know some folks who do reject astrophysics and some other sciences, so some people probably do mean that.

But let's think a bit more about what reproducibility means.  I took it up lightly earlier, regarding pretty much just climate data set reproducibility.  Please excuse the narrow focus as just attention to my own area.  I'll broaden scope some here.

On one extreme end of notions of reproducibility is the idea that anybody, anywhere, should be able to reproduce your results -- or else you aren't doing science.  I don't think that's actually been held to be a requirement for science at any time in history, so such folks are arguing for a change in how science is done.  Maybe they're right; let's think about it.  As with many ponderings, this goes for a while ....

In astrophysics, one of the things you do is collect data.  The equipment can be fairly minor -- amateur astronomers contribute to variable star observing, which is then used for professional research.  But often the equipment is major -- the Hubble Space Telescope, major ground-based observatories, exotic observatories for taking neutrino observations, and so on.  Such instruments are far outside the ability of individuals to pay for.  Since many people want to use these few instruments, few actually get to do so.  Certainly I would not be able to get time on the Hubble to reproduce observations that I was interested.  And it'd be fun to try to reproduce the observation program that lead to the conclusion that the universe is expanding at an accelerating rate.

Do we, then, conclude that conclusion is not science because you and I (bright and interested people) cannot reproduce the observing program ourselves?  Seriously, there's no way they're going to let us have the hours (thousands of hours?) of Hubble time to explore distant supernovae.  Same as they're not going to let us take over the Large Hadron Collider at CERN, or the accelerators at Fermilab or Argonne (which even have the advantage of being close to where I grew up) to look at whether I think the experiments really support the existence of quarks.  Nor will we get to charter an ocean drilling ship (closer to my fields) to reproduce paleoclimate results.  Not even to drill our own core on Greenland, where some interesting science is to be had, and where the two major cores in hand (GISP and GRIP) show some problematic differences.  (That's all well-known to people in the field, and they have some good ideas as to what they'd do next, if they got to go out again and drill such long cores.)  All areas that would have to be called 'not-science' if the standard were that anybody, anywhere had to have access to the equipment to reproduce the experiments.

Some will hold that hard line, in which case little science has ever been done.  That looks strikingly contrary to what we see in the world around us.  So let's consider stepping down the requirement some.  That, even if the equipment to collect the data doesn't have to be available to everybody, the data themselves do have to be available to everybody. 

But what does 'available' mean?  Astrophysics collects at least petabytes (1000 terabytes, 1,000,000 gigabytes) of data.  Climate, thanks to satellites, does as well.  Particle physics experiments throw away vast amounts of data in order to trim their storage down to that order.   And so on through, no doubt, many fields these days.  The data exist, but where will you put it?  And who is supposed to copy it for you?  It'd take eons to download 1 petabyte to my desk at home, and I'd have to by 1000 disks of 1 terabyte each (even at $100/terabyte, that's $100,000 -- plus making sure I had enough power to my house to spin them up) to store them.  Or to have the data source copy everything for me (are they working for free?  $100 per disk drive, plus someone's time to verify that the data were copied correctly and so forth ...).  Either way, even though the data are available -- for a price -- that price might still be too high for us as individuals to pay, or for the source we're trying to get the data from to be able to provide it.  Again, all large data set science would have to be ruled 'not science'. 

Suppose, though, that we're concerned only with some topic that the data volume is modest -- a matter of gigabytes.  I have downloaded that much to home myself.  Sometimes I can be patient.  Does that mean the data are really 'available'?  For me, since I already understood the data format, yes.  But data formats are many and varied.  People familiar with GRIB aren't fond of HDF, and vice versa.  While I've worked out how to use GRIB, I don't have that familiarity with HDF.  Astrophysics has its own formats, and particle physics has (probably) still others.  Each field has good, and some not so good, reasons for using the data formats they do.  Certainly every format has its fans.  But it means that just having a copy of the data file on your desk does not mean the data are truly 'available' to you.  You also need to be able to decode the data in to some usable form.  So here's another opportunity to declare an area 'not science' -- the data formats aren't easy for the non-professional to use.  Again, this would affect most science that's been done since computers came in to use.

Maybe you have a friendly data provider and they also give you a program to decode the data.  Are you home free?  If you've read this far, you know how this goes.  Of course not.  The decoder is a program and runs only on certain computers, or is commercial and you have to buy it, or you can get the program itself, but you have to buy a commercial product to run it, or ....  And, of course, the commercial software only runs on certain computers.  (I've encountered all these at work as well.)  I understand, from my reading in Science and Nature, that this problem -- of commercial software being used, and required, in doing scientific studies is particularly acute in biology.  And it raises, again, concerns about reproducibility.  Do we conclude that biology is not science?  I don't think so, but there are certainly concerns about how the science will progress when relying on black boxes provided by companies that may not be in business 5 years from now.

Some data have restrictions.  Hank Roberts in a comment mentioned this regarding the Arctic Ocean.  It's much more general than just Arctic Ocean concerns.  Biotechnology firms want to protect their assets -- which may include genes, how to conduct experiments, ....  Nuclear physics not only has commercial concerns, but national safety concerns.  If you want to set up nuclear experiments in your basement, you'll probably have some visitors from national or international security agencies. 

Even things as 'minor' as observations of the sea surface temperature or air temperature have their commercial concerns.  Ships do collect temperature (sea and air) observations.  Many share that data with their national weather services.  Cargo ships tend not to be concerned about other cargo companies knowing where their ship is (as far as I know).  But that's not at all true of fishing vessels.  There's substantial commercial advantage to knowing where the fish are -- and knowing where your competitors' ships are can be a clue to that.  Consequently it can be very difficult to get fishing vessel captains to reveal their location information -- and a temperature without a location is no good for science or weather prediction.  On top of that, in the 1980s and 1990s, many national weather services (Canada, New Zealand, European Union nations, ...) were partially privatized.  One of the aspects of privatization was, surprise, charging -- some people -- for data that had formerly been free to them.

Moving along to programs ... even theoretical work in astrophysics, as for physics, many areas of biology, climatology, and so on, uses computers to test theoretical ideas.  More straightforwardly, modelling requires computers.  All prior comments about programs to decode data apply to us being able to run theoretical and modelling programs.  On top of that, all comments about major facilities applies to the major programs.  If you want to run one of the major galaxy interaction programs, or the evolution of the early universe, etc., you need not only the source code (and libraries, and ...), but a big enough computer.  One standing truth is that no matter how big and fast today's computers are, scientists and engineers (I first heard this from one of my Electrical Engineering professors) will want to solve a problem slightly bigger than what those computers can handle.  So, although the computer on my desk at home is extraordinarily powerful compared to what I had 20 years ago, it is still dwarfed by the supercomputers used today for the major astrophysical programs.  Ditto, of course, climate programs, fluid dynamic programs for aircraft design, and on through science and engineering.  As for the major observing instruments, time on the world's biggest computers is limited.  If you have to be able to run the program yourself, in exactly the configuration of the research paper before something is 'science', then nothing more involved than can be run on any average home computer can be considered 'science'.  For climate modeling, that would mean something like the EdGCM is the limit of what could be considered scientific modeling -- a climate model with about 800 km resolution in the horizontal, and only 9 layers in the vertical.  Modern climate modeling is more like 100 km in the horizontal and 64 layers in the vertical -- for just the atmosphere.  Then it adds active ocean of similar resolution, and so on.

It seems a strange idea to cripple science down to what home amateurs can reproduce themselves.  Certainly it's against the history of science.  In our tour of where some of the obstacles are with that idea, I also didn't see much where there was an obvious reason that we should reject as science astrophysics that uses larger data sets than the few Gb I might download, and models small enough for me to use.  Nor that we should throw out much of biology and nuclear physics where there are restrictions on the data and on the conduct of experiments.  But maybe some readers see why science should be limited in these ways.

Rambling longer usual.  I'm prompted in this by wondering what pedant-general might mean in response to the original article there, which links to my earlier note about data set reproducibility.  I've asked the question there as to what the commenter means.  I've seen other people mean some of the things I considered extreme above, so that alone is worth a bit of thought.

I do still think that the widest possible degree of reproducibility is a good and important goal.  I've also worked with people who were working to make data distribution faster and easier (automatic reformatting, letting you extract only the subset you want, providing data analysis tools online, ...) so as to get around things which used to be major obstacles.  For programs, I do encourage, and try to practice, writing them so as to be easily transported.  And so on.  While the expanding scale of science has made for new problems, new tools usually show up to cope with the expanded scale.

Plus, of course, I like the idea of being able to reproduce important, and interesting to me, experiments or analyses from the history science.  And I'd like everybody else to be able to as well.  More fun to do my own versions of things, but I do like history, so some historic reproductions are nice.

In the mean time, though, it seems for now only climatology is being attacked over its 'nonreproducibility'; but the fundamental argument applies throughout science and engineering.  If climatology is to be trashed for this, then all the rest of science and engineering follows in time.

21 comments:

yea-mon said...

That's an excellent piece Robert. However, there's a large number of denialists for whom even reproducibility doesn't matter, as it's all about 'following the Scientific Method' - as if that ever had a rigid definition in real life.

Apropos reproducibility itself, Nick Barnes has been working with like minded souls on the Clear Climate Code project. I guess if some denialists actually can be bothered to use their code there's a minute chance it'll change their views...

jg said...

Your comment to pendant-general on reproducing a supernova event is delightful. I joke at astronomy events when someone says "look a meteor!" I usually look too late, and then declare their observation as unconfirmed.

Regarding reproducibility, you have to admit the denialists are reproducing their argument on this with great fidelity.

Not long ago I helped a friend answer a correspondence to the same effect: someone working for the Department of Energy argued that the holes in climate science data and lack of audit trails would provoke an immediate "stop work" order at the DOE. I didn't have a good answer for this point at the time, but will refer him to here.

Your suggestion that to throw out climatology will take astrophysics with it goes over well with lay audiences who need to hear this. Last week I made a similar comment to parents at an astronomy program when I summarized orbital connections to climate.

One of the parents commented that she wished global warming was taught more as I had done in my 10 minute summary, referring to the barrage of global warming education as mostly highlighting the negative. The parent wasn't asking for a positive message; but rather a more scientific one that shows how the experiment we're inflicting on the student's inheritance is also a scientifically fascinating area of study.

Thanks,
jg

Horatio Algeranon said...

A large part of the problem is that many who know little to nothing about science in general and climate science in particular assume that they are in a position to weigh in on key conclusions of climate science -- ie, human influence on the climate.

Most people would never even consider doing this in the case of astrophysics. They have come to recognize astrophysics as something that is done by a small number of mathematical wizards who spend their lives holed up on mountaintops staring through giant telescopes up at the heavens.

Though everyone can enjoy a starry night and a few amateur astronomers even make contributions to the science, by and large, most people shy away from weighing in on the central issues like the big bang and (rightly) leave it up to the experts.

The expanding universe and black holes are just not something people have direct experience with in their everyday lives and can ever hope to really come to grips with.

Climate science, on the other hand, is viewed quite differently by ordinary people. Every Tom Dick and Henrieta seems to have a theory about what is happening to the climate and why. Horatio even wrote a ditty about that: "I have a Theory and it is Mine"

The reason may be that climate scientists deal with something that everyone does have familiarity with: temperatures.

The attitude of the ordinary (largely science-illiterate) person seems to be, "Hey, how hard can climate science be? I know something about measuring temperatures. I have a thermometer in the fridge and outside the window and i even take Johnny's temperature when he's sick. And when I look at that graph of global temperature over time, it sure looks to me like temperatures have been flat or even dropping since 1998. No hard math required to see that!".

The demand for "reproducibility" seems to be based on the assumption that "This is easy to understand, so if I have the data, I can reproduce what the scientists have done -- and even tell if they got it wrong".

With something like astrophysics, the assumption of the ordinary person is just the opposite: "I could never reproduce the predictions of Einstein's theory in a million years and could never even hope to understand what to do with the astrophysics data so why would I even want it? "

Unfortunately, in trying to bring the science of climate down to the level that the ordinary person can understand, climate scientists may actually have contributed to the problem of everyone thinking they are a climate expert.

How's that for irony?

Add to that the folks who are using the whole "reproducibility" thing to harass scientists (eg, with a flood of FOI requests) and it would seem that you really can not win as a climate scientist these days.

jg said...

Horatio, (I enjoyed your blog, by the way)
I could add to your comment on temperature: I have a friend who's very adept in engineering. One of his comments has been how difficult it has been for his testing instruments to be reliable within a degree, so naturally when climatologist cite a 0.5-0.8 degree rise, he is skeptical as to their precision. I have no answer for him.

I would disagree slightly with your suggestion that because temperature is a common experience, it's the connection by which people dispute climatology (I'm simplifying). I posit that if there were a large social/political entity that benefitted from a flat earth, or no expansion of the universe, there would be a cottage industry supporting these views.

I don't think bringing the science down to the lay person is fueling the opposition. I say this in part because of the efforts I've made to give the public a dose of climatology in my astronomy presentations. Granted, I have a cherry-picked audience, parents who bring their students to science events, but the little dose of climate goes over well. One person remarked last week how she wished climate was taught as the fascinating subject it is before the anthropogenic CO2 is rolled out. In other words, correcting the propoganda is a buzz kill.

And on the subject of reproducibility, I read in Nature 4 Feb of Eocene ocean circulation models taking 1-3 years to run. I find that hard to believe. What do you do in those 1-3 years? If my desktop computer stalls for 10 seconds, I reboot.



thanks,
jg

deaner said...

On one extreme end of notions of reproducibility is the idea that anybody, anywhere, should be able to reproduce your results -- or else you aren't doing science.

Nice try, but that's a straw man. I don't personally want to reproduce any particular result - but I want to know that somewhere someone could try. No doubt, it would be someone better equipped, with a better base of knowledge, and greater understanding. Preferably, it would be someone who didn't a priori agree with the result under examination. In some fields, it may be that there is only one person who group who could possibly replicate the results - "big science" like the LHC requires the resources of several countries; in the case of the SCSC it seems to require even more than that - obviously no one is going to create a Higgs in their basement. That does not change the fact that when results from the LHC are announced, other researchers will be pouring over data to see if they agree with the analysis, or if there is an alternate explanation. If facilities are available, other teams will try similar experiments - either on the LHC or on future facilities. If the result cannot be replicated, there will tremendous effort put into determining just how the experimental set-ups are different - and what that implies for the validity of both results.

Compare that to the response to a request for data from a leading climate researcher: "why should I share it with you, when all you are going to do is try and find something wrong with it?" Well, yes, actually - that's rather the point, isn't it? One of these groups is doing something that no one in the world can currently hope to duplicate since they are using a 'one-of' piece of equipment; the other is running statistical manipulation that could be run at most university computing centres and many commercial sites. One is "doing science" despite the impossibility of reproduction; the other is doing something else.

Horatio Algeranon said...

JG said "I posit that if there were a large social/political entity that benefitted from a flat earth, or no expansion of the universe, there would be a cottage industry supporting these views."

You may be right about that. There is no doubt that a well orchestrated (and well funded) disinformation campaign has contributed greatly to the popular "skepticism" and outright denial about global warming (at least in the US)

Nonetheless, Horatio has seen more than his share of comments on the web that embody the (largely uninformed) "Armchair Expert" attitude referred to above.

Horatio was educated as a physicist and an important part of that training was to acquire a healthy respect for the knowledge and expertise of those who have studied a subject deeply.

That is not to say that the word of the experts should simply be accepted, but that one should not dismiss what they say without having a very sound reason for doing so (ie, based on a very good knowledge and understanding of the subject at hand) In other words, true skepticism is knowledge- and expertise-based.

Unfortunately, most of what passes for climate "skepticism" these days seems to be much loser to contrarianism than to true skepticism, as Horatio indicates in
The Global Warming "Skeptic"


With regard to your statement that "I don't think bringing the science down to the lay person is fueling the opposition,", horatio would not really disagree, with one important caveat:

IHHO, climate scientists have not been clear enough about the fact that at least some of the key analysis upon which they base their main conclusions about the human/climate correlation is simply NOT readily accessible to the "Armchair Expert" -- where accessible has nothing to do with data and code accessibility, but instead with scientific knowledge and expertise.

Also, when it comes to educating the public, climate scientists may have erred in not making it clear enough that the conclusion about human influence on climate is the result of FAR more than just the graph of the global mean temperature (or "record years" for warmth), which, unfortunately, seems to have become the poster child for global warming.

In short, climate scientists may have made things appear too simple -- ie, as something that any fool can weigh in on (and, seemingly, does)

Finally, with regard to the statement of your engineering friend

One of his comments has been how difficult it has been for his testing instruments to be reliable within a degree, so naturally when climatologist cite a 0.5-0.8 degree rise, he is skeptical as to their precision. I have no answer for him.


The critical element that your friend is missing is that, as described very clearly by Tamino in The Power of Large Numbers, "the average of a large number of estimates is more precise than any single estimate."

So even though any individual temperature measurement may only be measured to the nearest degree, the average of a large number of such measurements will be precise to within a fraction of a degree.

Penguindreams said...

jg:
I'll take up your friend's concern in a post tomorrow. In the mean time, remind him "square root of N". That should be enough for him to provide a "D'oh!" response. If not ... my longer article might do.

deaner:
No straw man. I have indeed seen the claim. The most extreme version being someone in a blog comment who wanted every version of every program ever used by anybody involved in the analysis, every data set, including every intermediate data set, every email ever sent by anybody involved, and all hand-written notes. I didn't save the link, so will offer Andy Schlafly instead, and his correspondance with Richard Lenski. Schlafly definitely qualifies as an 'anybody, anywhere', and was demanding experimental samples that he was grossly incapable of working with and would have destroyed through his lack of knowledge.

If you've spent a few years on the net, it should be no surprise to you that there is no statement so extreme that you can't find somebody who believes it.

The 20th century surface warming is something arrived at independantly by CRU, NASA-GISS, and NOAA-NCDC. If you wish to claim conspiracy, present evidence of conspiracy. Not the mere slander. Each of those groups has every reason to not want to support the other. You don't get grants by saying 'yep, the other group is right'. Each processes the data in different ways, which is part of the business of not being in it to prove the other group right. Yet when all is done, they each find that indeed the globe, as observed by the surface record, is warming.

Now if you want something independant of those groups, well, you could also go to Open Mind and see yet another approach (different methods yet again) and download both codes and data to run them for yourself and get, yet again, the same answer. (See also followup posts, wherein folks with no love of tamino, the rankexploits crowd, got the same answers as he.)

Or you could have a look at Roy Spencer, who has a history of making errors that overstate cooling and underestimate warming, and see that he, too, finds that his casual method ('not optimized') is in extremely good agreement with the CRU results.

The results not only can be, but have been, reproduced repeatedly, by different methods, and by different people.

Since you're willing to allege fraud on the part of climate scientists, I find it puzzling that you would trust them to give you the right data anyhow. If I though somebody was cooking the books, the last thing I want is the data that they've had a chance to cook. I'd go straight to the original sources and see what the data were originally, and work with that.

Irrespective of concerns of fraud, which I don't have, I prefer to work with the most original data possible. In any processing system, there are decisions to be made. Decisions that are best for the other group's purposes may not be right for my situation. So I work with more nearly raw data and make my own mistakes, er, decisions.

jg said...

Another reproducibility observation: I read in a News and Views in 11 Feb Nature that creating transuranic elements can take about a week just to get one at atom.

Deaner,
I see your point about wanting results to be replicated by non-sympathic reviewers, and if the sample quote in your last paragraph was a generalization for brevity, I'd like to ask that you take it one level deeper and share the full example so I can better understand the context.
This example is contrary to my take on the science based on reading Nature. Often, a new and significant climate study is prefaced with a News and Views by an independent reviewer who describes the research, the context, and major conflicts and caveats. I would consider this one example of the type of evaluation you're saying is missing.

I suspect our perspectives differ based on what we read.

thanks,
jg

deaner said...

If you've spent a few years on the net, it should be no surprise to you that there is no statement so extreme that you can't find somebody who believes it.

Yup - I would suggest that "replicable by anybody anywhere" is not a standard held out by reasonable skeptics; that doesn't mean it hasn't been said.

Since you're willing to allege fraud on the part of climate scientists...
I don't suspect deliberate fraud - but I do think there has been overwhelming group-think. In a highly uncertain realm, it is always easy to find reasons to discount results at variance to your own, and to overlook faults in results that are in accord with your work. Over time, that develops into a very clear "us versus them" world-view, which was more than evident in the CRU e-mails released last November

...and if the sample quote in your last paragraph was a generalization for brevity...
I will dig it out, although it may take a day or so - it came from the (purloined or lberated, take your pick) CRU e-mails.

Penguindreams said...

Deaner:
It's a good idea to note who you're answering. I was puzzled by the last part of your last comment, as it didn't sound like me you were answering. And it wasn't. As far as that one goes, however, if you're using the illegally obtained letters, your job is a little harder -- you also have to show that the thousands or tens of thousands of emails that were _not_ released by the person acting illegally and selecting a subset of the letters don't contradict the point you'd like to make.

Your original comment was that I had set up a straw man. That is, at least now, obviously not true. As to what 'reasonable skeptics' think, well, that's something of a question itself. Andy Schlafly says he's a reasonable skeptic. You disagree. But you don't tell me how I can identify the reasonable skeptics -- as you use the term -- from the unreasonable.

As you read my original, there is a progression of things which I suggest pose barriers to the ideal of everybody getting all data from everywhere. You agree that that sort of unlimited thing is not reasonable -- right? But how about identifying where between kilobytes and exabytes the reasonable skeptic agrees that data set scale (currently) says that it is not reasonable to expect the all data to all people ideal. And, ideally, on through the list -- how does your reasonable skeptic deal with proprietary information, information with restricted distribution rights, etc.?

Do be a reasonable skeptic here. One part of which is to answer what's written. You allege fraud, and now clarify that it's not intentional fraud you're alleging. But you don't notice or respond regarding the number of different groups I mention, and two I provide you links for, that show indeed there's a warming trend. Unless you're claiming that Roy Spencer is also part of the fraud to produce false warming trends?

In reality, determining that the surface observations produce a warming trend is dead simple. Not a 'highly uncertain realm'. Spencer used a quite trivial method, but one he trusted better than CRU (et al.) and still got what he considered a surprisingly high correlation to the CRU results. Rather than the refutation he expected.

The complexities and uncertainties aren't for deciding whether there's a warming trend over the last century. Or at least that's my reading of the science. Perhaps your 'reasonable skeptic' rejects that? Preferring a grand conspiracy (witting or no) between hundreds to thousands of scientists in dozens of countries?

EliRabett said...

A large part of the problem is that most people learn a cartoon version of science in junior high school and never get to experience the actual messiness of the real thing.

Science is like construction, a total shambles till the end of the project at which time you clean and polish it up like crazy and it is just beautiful. (Oh yeah, sometimes someone leaves an petrified sandwich behind the wall)

Horatio Algeranon said...

deaner said: "I don't personally want to reproduce any particular result - but I want to know that somewhere someone could try. No doubt, it would be someone better equipped, with a better base of knowledge, and greater understanding. Preferably, it would be someone who didn't a priori agree with the result under examination."

"Someone"?

Like another climate scientist perhaps?

As it stands, lots of individuals who are not climate scientists (or even scientists) -- and do not even claim to be -- have been making requests for data and code (and emails), sometimes through FOI requests.

If the goal is actually what you stated: that "somewhere someone could try [to reproduce any particular result]", then there is really no valid reason for scientists to provide data and code (to say nothing of emails!) to anyone besides other climate scientists (ie, those with the demonstrated knowledge and expertise to actually make legitimate use of the shared materials).

In other words, under your own standard, there would be no legitimate call for a scientist to provide anything (code, data, etc) to individuals like Steve McIntyre or Ross McKitrick, who are not climate scientists.

That seems to be what you implied above (with your "somewhere someone could try [to reproduce any particular result]" statement)

Is that really what you meant?

Note: The idea that all climate scientists are somehow going to just accept ('agree with") new scientific results that have not yet been independently validated is simply not credible.

New results are normally the focus of further tests precisely because they have not yet been accepted.

Horatio Algeranon said...

Eli says A large part of the problem is that most people learn a cartoon version of science in junior high school'

Yeh, like Roadrunner, where Wile E Coyote manages to repeatedly fall off 1000 foot cliffs and blow himself up yet somehow (miraculously) survive.

If only the rest of life on earth were as resilient to climate change.

deaner said...

In other words, under your own standard, there would be no legitimate call for a scientist to provide anything (code, data, etc) to individuals like Steve McIntyre or Ross McKitrick, who are not climate scientists.

Hmmm...

The climate reconstructions are a particular type of science - they are as much statistical manipulation as observation and measurement; and I would argue that McIntyre and McKitrick are at least as capable of statistical manipulation as most of the climate scientists who have been using the data.

deaner said...

...and if the sample quote in your last paragraph was a generalization for brevity...

My apologies - it didn't come from the CRU e-mails, but from an e-mail from Phil Jones to Warwick Hughes, after Hughes asked for data, and runs: Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.

I have not been able to find an original on-line, although there are many discussions (typically at "denier" websites - they may not be considered evidence here, I suppose) that cite the phrase. In a cursory examination I can't find it referenced on Hughes' website - but that is not conclusive. Search engines return over 20,000 hits for the phrase; I have linked one mainstream media citation here.

Horatio Algeranon said...

deaner said "McIntyre and McKitrick are ...capable of statistical manipulation"

No disagreement there. :)

But that actually does not address the point you yourself were making:

"somewhere someone could try" [to reproduce any particular result]

If that is truly your standard, the question you really need to answer is "Why can't that "someone" be a climate scientist?"

Or, if you happen to believe that it should not be a climate scientist, why not?

Understanding the way the world (in this case climate) works involves far more than "statistical manipulation". The reality is that the latter is neither sufficient nor even necessary for gaining such an understanding. In fact, sometimes statistics are used for just the opposite reason (indicated by what Mark Twain liked to say about statistics

Of course, the primary "statistical issue" for climate science (at least according to McIntyre & McKitrick) is the use of PCA (principle Component analysis) by Mann et al in their original "Hockey stick" paper.

But the simple truth of the matter is, the whole "PCA debate" was really a distraction because the "hockey stick" temperature reconstruction does not depend on PCA -- or any other statistical manipulation, for that matter.

As shown here you get the same "hockey stick" shape to the temperature reconstruction if you don't even use PCA.

Or, as Eli Rabett has indicated, "It [the hockey stick] writes itself"

PCA is merely one way (among many) of representing data. That's all. Despite the implications that it was somehow the "One True Way", there is nothing at all special (or preferred) about it.

Furthermore, as shown here, the hockey stick does not depend on any specific temperature proxy (tree rings, for example). It results if you look at borehole data, at lake sediment data, at corrals and lots of other proxies.

The latter fact appears to have been completely lost on some of those who are not climate scientists.

deaner said...

Horatio - you are quoting sowmwhat selectively. I agree that it is necessary that someone, somewhere could try to reproduce a result (or demonstrate that it cannot be reproduced). I continued to say: Preferably, it would be someone who didn't a priori agree with the result under examination...

So far, that hasn't happened.

Penguindreams said...

deaner:
With your latest comment, realize that you're telling us that Roy Spencer -- whom I cited and linked to earlier -- is someone you consider a priori committed to finding warming. If there were anybody with both professional credibility and a track record of not being disposed to believe the surface record (period) or the CRU, he'd be your man. For you to say it hasn't been done, after the reference and link to Spencer is given, says more about your a priori commitments than about the field.

re deaner's earlier comment, regarding your source being denier web sites. Others can speak for themselves. For me, if by 'denier' you mean sites that say CO2 is not a greenhouse gas and that sort of thing, no, that would not be a persuasive source. That CO2 is a greenhouse gas is a solid 150 year old result. If people are willing to claim the opposite of what's true on that, I have to figure this isn't the only thing they'll do it for.


I'll be off-net until late Monday, so further comments will have to wait for moderation.

Horatio Algeranon said...

deaner says" Horatio - you are quoting somewhat selectively."


Perhaps you missed that line in my first post. I quoted you in full, including the part "Preferably, it would be someone who didn't a priori agree with the result under examination..."

perhaps you also missed the end of that same post where I commented that "The idea that all climate scientists are somehow going to just accept ('agree with") new scientific results that have not yet been independently validated is simply not credible.

New results are normally the focus of further tests precisely because they have not yet been accepted."

deaner said...

...realize that you're telling us that Roy Spencer -- whom I cited and linked to earlier -- is someone you consider a priori committed to finding warming.

No - Roy Spencer has not been given access to the data or the codes used by CRU or GISS. The fact that he has conducted his own examinations does not relieve those institutions of the obligation to make their methods open to inspection.

deaner said...

...re deaner's earlier comment, regarding your source being denier web sites...

Sorry - it seems that I wasn't clear. My immediate source(s) for the Jones quote was WattsUpWithThat and ClimateAudit. I had mistakenly thought that I had seen a link to a source document, but that was not the case. In searching for the source document, I found ~23,000 citations through a search engine - I posted the link to one treemeat publication - in >23K citations, I am sure there are others. I have no way of knowing whether any of those citations are based on a viewing of the original material, or are all based on citations of citations.