25 August 2008

Testing Ideas 2

Finally, in comments on my cherry-picking article, I was invited to take a look at http://rankexploits.com/musings/2008/ipcc-central-tendency-of-2ccentury-still-rejected The commentator didn't really say what I should be learning from the link or what relevance he thought the page had to cherry-picking. I discussed some basics about making good tests in part 1 on testing ideas. Please check out both prior posts as I'll assume you know their contents. It's a little work, I know, but in order to have intelligent disagreement, or agreement, we have to know what the other is saying.

The title of that article is "IPCC Central Tendency of 2C/century: Still rejected". Certainly catchy. But let's look at the substance: Does the IPCC say there's a central tendency (and if so, what is it that is supposed to have that central tendency) of 2C / century? An immediate red flag for me is that the site does not provide a reference to where the IPCC said any such thing. Being better about doing my homework, I went to the IPCC WG1 report itself and looked in the summary for policy makers and the chapter (10) which discussed the global climate projections. I invite you all to do so as well. In my look for 'central tendency', I found no such term in either section. In my reading, which has not been exhaustive, of the two sections, I still found nothing that could be construed as a 'central tendency'. The projections were projections, not forecasts, and were made for a number of different scenarios (assumptions about greenhouse gas emissions and other things). Some included ensemble means, but none of those were 2.0 C. One was 1.8, but calling that 2 is back to the problem of deciding whether I'm tall and letting me round to the nearest foot or meter. Not even out of the title and already there are problems.

In the lead paragraph, the author writes "... compared to the IPCC AR4’s projected central tendency of 2C/century for the first few decades of this century." Again, I don't find the IPCC saying that, and again, the author doesn't say where the claim comes from. The nearest match I find is in the Summary for Policy Makers, p. 12, where it says "For the next two decades, a warming of about 0.2 C per decade is projected for a range of SRES emission scenarios." The author mutated a term of precision, two decades, in to a vagary, the first few. Then the fuzzy term 'about 0.2 C per decade' got cast as a hard term of 2C/century. If nothing else, in reading this site, you're not reading a reliable reporter. Once I've reached that point, I generally stop reading a source. There was no need to misrepresent the original. Nothing was saved or simplified.

In terms of part 1, remember, I mentioned that you had to be careful to make a good test, including that it had to test the thing at hand. In that case, you had to be looking at my height and weight in order to decide whether I was 'tall and thin'. In this case, you have to compare (in some 'good' way) observations that bear on the thing actually predicted by the IPCC. When you read chapter 10, you'll discover several things. One is that the variable being projected is the global average surface air temperature. Another is that the models have interannual variability (they'd better -- nature does, as I mentioned in the cherry-picking and detecting climate change articles). And you'll see that 30 years is the normal period for averaging temperatures to make climate conclusions; but even so, the projections were being (gingerly, you'll notice if you know how scientists write) made about 20 year averages. One final thing buried in those 100 pages: The projections assume that there are no volcanoes and the sun's input remains fixed at the observed average value. We haven't had any major volcanic eruptions since Pinatubo, but the sun has been quiet since 2001 (below the average output).

So several things to go in to making a good test:
  • We have to compare 20 year average vs. 20 year average (the variable being suggested as meaningful in the report, and 30 would be better).
  • We have to look only at global mean surface air temperature.
  • We have to adjust for the fact that the sun has been giving less energy than assumed in the projections.
  • In making the test, we have to allow for the fact that the 0.2 C/decade itself has error bars (due to interannual variability and between-model variability)
The author does none of these things, and presents no reason why they are not necessary. Instead, she computes trend lines for January 2001 - present, not 20 year averages. She includes satellite observations of temperatures through the lower-mid troposphere (the UAH and RSS), rather than using only surface air temperatures. She ignores that the sun has been quiet. And she makes no allowance for the error bars on the IPCC projection.

Even if absolutely everything that was done in the statistics were right, which I'm not in a position to say much about, the test is not a good test so the result is at best meaningless. Chances are good that it's misleading (tests that aren't good are usually misleading).

The satellite temperatures show a cherry pick themselves. But first, why they shouldn't be used in this context in the first place. The thing is, they're not observing the variable that is being predicted. While it is connected, which might lead us to thinking that it's ok to use them, it still may not be. Back to the question of whether I'm tall. I think that the data you want is a measurement from the floor to the top of my head. It's true that leg length is related to height, at least in the sense that tall people generally have longer legs. So maybe you'd accept that instead. And say you took Michael Phelps' leg length too. If you looked at the two, you'd conclude that I'm taller than Phelps. (His legs are short for his height, mine are long for mine.) You'd be wrong, however; the data don't address well enough the question you're asking.

The cherry pick is that only 2 of the 4 satellite temperatures were taken, and it happens that the two are the two which show the least warming (you'd have to know about this, which is easy enough to find if you look but isn't universal knowledge). The author gives no reason for this selection. Further, even ignoring that, one of the two also is not a global data set. The RSS goes only 70 S to 82.5 N. There are good reasons for this, and, conversely, to not be confident about the UAH figures, but they get technical. It suffices here that only 2 of 4 data sources were picked, one of them isn't even global, and, worse, neither measures the variable of concern.

Even without knowing anything about statistical methods, we can see that the given site does not support its headline (and have some question about the headline as well). All that is needed is to check what the source (IPCC in this case) actually said, versus what was being tested. They're different things, so the test doesn't tell us about IPCC's projections.

We can't make the converse conclusion from this, that the IPCC projections are correct. We didn't test that idea, so no conclusion about it is possible here. We only checked whether the site was making a good test. It wasn't.

A non-digression to something related. While I've taken probability and statistics courses, they weren't very deep (I felt). I was thinking about taking more, and asked a coworker -- a statistician -- about doing so. We talked some about what I'd studied (and remembered, this was a good 15 years after I'd had the classes). She concluded that there was no real point for me to take the courses. More important, she said, was to understand the system I was working with. The appropriate statistical methods would suggest themselves, or at least I'd be able to hand a well-constructed question to a statistician. But without understanding the system, there was no point in applying statistics.

Project: Pull down the 5 data sources given at that site and compare the last 20 years (7/2008 to 8/1988) average global temperature against the previous 20 years, 8/1968 to 7/1988. Is the more recent average greater or not? Can't do that for the satellites as they don't go back that far. But try 8/1979 to 7/1994 versus 8/1995 to 7/2008. Only 15 years versus 14 years, so not very informative about climate, but it's all the data we have. Won't be a good test, but the best (perhaps misleading) that can be done given those data. (Try changing the periods too.)

johnathansawyer: as you invited me to look at the source, did the above affect your opinion about it? Why or why not? As usual, substantive reasons (either way).

[Update 28 August 2008] See my next two notes what is climate - 2? and evaluating climate trend for more on what appropriate averaging periods look like, and what happens if you compare the average of the last 7.6 years to the previous 20 years' average, respectively.

[Update 29 August 2008] rankexploits made a lengthy commentary on things related to this post, though misrepresenting even my first paragraph. Comments on that post are disabled (I get an error message in response to my comment attempt), my response is #5317 in http://rankexploits.com/musings/2008/ipcc-central-tendency-of-2ccentury-still-rejected

Folks who are getting heated about my 'advocacy', or defense of IPCC, or whatever. Take a minute. Read what exactly I said. Saying that a particular test was not good is far from saying that there can be no such test (I give an example of one that would be better myself!). Nor, as I said directly, does it mean that the projection is good. Let's see what happens when a good test (which mine isn't, just better) is made. Until one is presented, a poor test is still not useful.

13 comments:

Anonymous said...

I'll say this for Lucia's analyses: they do point to a way that 'climate science / global warming / models' can indeed be falsified. Despite numerous claims of unfalsifiability.

I wonder if I can persuade you to write a little on this topic? You would be in good company (http://initforthegold.blogspot.com/2008/05/falsifiability-question.html).

Robert Grumbine said...

I'll be talking about this sort of thing from time to time. For now, I'll mention that all of the people I've seen making the claim that the models are 'unfalsifiable' are people who don't believe that the models are useful in the first place.

Michael Tobis' note (Initforthegold) mentions a different good point: That anthropogenic climate change is a result, not a theory itself. I encourage people here to take a look at it. I don't agree with Michael about everything, but he raises good points here to think about.

See also my most recent (as of 25 Aug 2008) comment in the cherry-picking thread, to a poster showing great confidence of what the science will have to say.

Anonymous said...

I also find it odd that Lucia has not published a similar analysis on older projections for climate change, such as IPCC reports 1-3, or Hansens 1981 paper.

She did do a 'eyeball and slide' test on previous IPCC reports and the results looked quite good to me.

Robert Grumbine said...

Maybe you can expand on what an 'eyeball and slide' test is? I'm wondering, too, if it looks something like what I did in tomorrow's note. (aside: I generally write a batch of these at a convenient time, and then schedule them to appear. Tomorrow's, I mostly wrote last Thursday evening.)

Anonymous said...

Actually its Slide and Eyeball - It was Lucia's term.

Basically draw the actual temperature and the projected temperature on the same chart and use the eyeball to see if it looks good.

http://rankexploits.com/musings/2008/introducing-my-version-of-slide-and-eyeball/

Anonymous said...

I think you may want to review some of Lucia's older posts where many of your issues were previously addressed. She makes a strong case for the methodology she has used and is quite frank taht future data may very well change the conclusions re: falsification (I don't like that term either, but a guess it is one tht statistician's like). Here is a good starting point: http://rankexploits.com/musings/2008/what-are-the-ipcc-projections-and-how-not-to-cherry-pick/

Robert Grumbine said...

I appreciate the link, but it only addresses why she took 2001 as a start, which I don't even comment about here, and not a single one of my bulleted points. Even though she links to the AR4, she still does not give a source of them saying 2C/century. 0.2C/decade for 2 (my p 12 cite) or 3 decades does not mean 2C/century. The former is limiting its scope of prediction relative to the latter.

See also today's note on what is climate for a consideration of how much data you need to be talking about climate vs. weather.

Anonymous said...

Lucia has an extensive reply to your posting in the comments of the article you are critiquing. Please read and respond. My question is why are you making a big ado about the difference between 2 deg/century and 0.2 deg/decade.

Robert Grumbine said...

Johnathan:
Two points about 0.2/decade for 2 decades vs. 2 C/century. First, it's a matter of honest reporting. If the source says one or the other, you should quote the one they say. Second, how much money would you owe me if it were $1/months for 2 months rather than $12/year? The latter leads you to a different,and erroneous, conclusion. Different side is that the original said 'about 0.2/decade'. Anything from 0.15 to 0.25 can be 'about' 0.2.

I've made a more relevant test in my new evaluating a climate trend post. It leads to rather a different conclusion than your source.

As to the 'response' ... not much there. She continues to not provide accurate links to what she's talking about, and I've already seen she's not an accurate reporter. She attributes something to Gavin at realclimate, with no link. Gavin is very prolific there, realclimate is enormous, and his comments are often embedded in the response tree rather than main articles.

She opens with a false report as well (the link you've already provided). The blogger, “penguindreams” is not the first to defend the IPCC projections against falsification by the odd claim that the IPCC projections do not have a central tendency and/or aren’t projections or whatnot. Except I didn't 'defend' IPCC against falsification (see again my evaluating climate trend post as well -- it could have come up otherwise). Rather that a specific attempt at it was not good enough. Nor did I say that there's no central tendency. Her poor reporting is the problem there -- there are many measures of central tendency and she didn't say which one she was using, nor did she cite where she'd gotten her figures. Something I've made the distinction between is prediction vs. projection. They're different things in my fields, but that'll be a different post.

The solar cycle business is part of the issue of understanding what you're looking at. If you're making a projection (vs. prediction), and it's of a 20+ year average, things that cycle on 11 year and shorter scales average out. (Try it yourself, take sin(2*pi*year/11) for year = 1..200, pi = 3.1416, and average it from 1 to 20, 1 to 30, 100, etc. also do 1-7, 2-8, and so on -- what we'd be looking at if the sun had continued its normal cycle.) No need, then, to include the solar cycle. Conversely, if some groups have included the normal solar cycle, they'd still be off -- the sun has not cycled normally since 2001. It's been stuck in low. Compare this with what she says.

She says she uses the satellite readings near the earth's surface. Again, knowing what you're looking at matters. If by 'near' she means a layer including the surface and up to about 10 km, centered at 4 km (13,000 feet) elevation, then I suppose 'near' is fine. As global mean elevation is 0.8 km, though, and I'm at about 0.05 km, I don't buy 'near'. See the 2000 National Academies Press report on reconciling observations of global temperature change. At that time, there were discrepancies. The discrepancy ironed out to errors in the processing of the satellite data; the other two were ok. (Check the version numbers on the UAH data set and their differing trends.)


See also, if you haven't already, my note what is climate. It should give you a feel for why 7 years is not a good averaging time for climate, much less to compute climate trends.

Anonymous said...

Re the solar cycle, it has actually declined from its peak to rock bottom over the 2001-2008 period. While the peak wasn't a big one, if Camp&Tung's findings from http://www.amath.washington.edu/~cdcamp/Pub/Camp_Tung_GRL_2007b.pdf are relied upon, the fall in temperature over the period due to the sun cycle must be in the order of 0.1 degrees. And since the ENSO index was predominantly in a positive phase in the first part of the period and negative in the last, this would also give a cooling trend. The clear conclusion is that Lucia's analysis cannot falsify the IPCC projected trend of 2 degrees/century, the period is simply too short to remove short time natural variations.

Anonymous said...

This is a pretty poor response.

1) You obviously haven't read any of Lucia's older posts. One of the main motivations for testing the 2C central tendency since 2001 is because bloggers (including Gavin) have frequently stated that the climate variability (with solar influences) is sufficient to explain the current flat or negative trends. From a statistics standpoint, Lucia is refuting this.

2) You state your knowledge of statistics isn't great and then criticise her statistical tests.

3) She did test GISS, HadCRUT and NCDC, although you claim she didn't.

4) Over a 20 year period the test probably does pass. Fair enough. But the current flat trend is what we're interested in. And she does account for the error bars in the IPCC report. Please do your homework.

Anonymous said...

Penguindreams,

Lucia has posted a blog post in response:

http://rankexploits.com/musings/2008/jonathan-asked-me-to-respond-to-penguindreams-heres-the-response/

It addresses pretty much all the issues you raise.

Robert Grumbine said...

Folks were busy while my connection was down.

Erik:
Thanks for the link on the solar effects.

Those who find Lucia's analysis to be 'thorough' or 'detailed' would do well to read this example and its cited papers.


Anon:

Minus rhetorical games, and further inaccurate reporting, alas, it does not. (c.f., she says I provide no link to her article, yet it's right there in the first sentence: Finally, in comments on my cherry-picking article, I was invited to take a look at http://rankexploits.com/musings/2008/ipcc-central-tende...)

Most importantly, it never bothers with the question of what is climate and how to properly examine it.

Joel:

1 No, I have not read every post she's made. I read the post I was invited to read and discussed that. The post itself contained few links to her other posts, none to the people she was criticising, and none seemed important to understanding her point about trying to falsify IPCC. Now, if I'd been invited to read 20 of her posts and respond to the whole body, that'd be a different matter.

2 My criticism isn't about the statistical methods, a point which I'll make a separate full post regarding (statistical vs. physical understanding). It is a GIGO (garbage in, garbage out) criticism. You can run a straight line through anything. Same as you can run a line through temperature anomalies w.r.t. year, you could run it through temperature anomalies w.r.t. my weight. The latter will also show a negative trend (as I've put on weight since 2001 it's gotten cooler). I don't need to be a professional statistician nor climatologist to know that the latter tells us nothing about the climate system. For only slightly more involved reasons, she's doing the same sort of thing.

3 Do provide the quote. What I see being said in my note is She includes satellite observations of temperatures through the lower-mid troposphere (the UAH and RSS), rather than using only surface air temperatures.

Critical reading skills help. 'includes' says that there's more than satellites being involved, and 'rather than using only surface air temperatures' says that surface air temperatures are included.

It's hard to employ them when what you're reading disagrees with you or what you want to be the case. For all that I've been criticised, however, about not reading rankexploits properly, nobody has actually shown me that she said something other than what I quoted. So that's a good sign on my reading, at least.

4 Looking at weather is a good thing to do. Nothing wrong with being interested in the last 7 years. But if you're going to make claims about climate, you have to look at climate. 7 years is short to make statements about climate, a point you can find illustrated in quite a few places, including my latest what is climate note. Yesterday before my connection went down, I'd found 3 others, looking at it in different ways from me -- tamino, Stoat, and Realclimate. All coming to very similar conclusions to each other and me, in spite of using very different methods from each other and (even more so) me. But take your own trip looking.

D. Johnson:
(Not posted; unsupported personal criticism)