Testing Ideas 1

I was invited (challenged, whatever) to take a look at a site that proposed to have disproved the 'IPCC prediction', over in comments to my cherry picking note. Here's part 1 of the look I promised in my comment reply over there. If you haven't already, please do read the cherry picking note. (Not only for my tiny little ego boost from having more page views, but because I'll be assuming here that you understand what all I mean by the term and examples of it for climate.)

Testing ideas is one of the central processes for science. Coming up with ideas is awfully easy. Supporting them takes work. Strengthening them so that they stand up to all good tests is extremely hard. But, they do have to be good tests. Same as it's harder to come up with supported ideas than just an idea, it's harder to come up with a good test than just a 'test'.

'Good test' does not mean 'comes up with the result I like'. Maybe it does, maybe it doesn't. You're usually much better off to not have specified before hand how you want the test to come out. A good test is one that is aware of the system it is studying, knowledgeable about the idea that it is testing, and has been devised carefully enough to confirm or deny the idea while also giving an idea of how firmly it is supporting or denying. Before launching in to the climate case, let's look at something very much simpler.

You might have the conjecture (even more tentative than a hypothesis), after having read my mention that I'm a distance runner, that I'm tall and thin. How would we test that? How can we make it a good test? You could just ask me. But that isn't a good test. You have no idea what I think is tall, nor do you have any idea what I think is thin. So if I say 'yes', you still don't really know anything. Poor test. You could try asking me my height and weight. But then you've made a poor test because you didn't establish what constituted tall or thin. If you're biased about the conclusion, it is far too easy to say after the fact that whatever figures I gave you do, or don't, constitute 'tall and thin'.

So you have to be precise about declaring what you're testing, what constitutes a pass and what constitutes a fail. You might arrive at something like 'taller than 80% of men your age means tall and lighter than 80% of men your height is thin'. Someone else might want those figures to be 90%, but, since you've specified how you arrived at your labels (and, of course, you will be sharing your data), they can examine for themselves whether they agree with your conclusion. You're still not out of the woods, however, because you don't know how I'm measuring my height and weight. Perhaps I'm wearing thick socks, shoes, and standing on my toes. Maybe I'm weighing myself fully clothed and carrying a backpack. You need to specify the conditions of the measurement as well. Further, you'll have to tell me how to report it. I could be 5'7" (170 cm) and round to the nearest foot/meter to tell you that I'm about 6' (2 meters) tall. That sort of ambiguity can completely ruin your test.

In any case, even for something as simple as deciding whether someone is 'tall and thin', you see that there can be quite a lot involved. One last thing, which winds up often being important in looking at people's conclusions. After going through the work of making a good test, you can only draw your conclusion about the thing you were testing. Suppose you decided that I was indeed tall and thin. That was your test, so that conclusion is reasonably good. What you cannot do, however, is conclude that because I am tall and thin, I'm a good distance runner. You never tested that, and haven't presented evidence that all tall and thin people are good distance runners. (They're not, nor is it true that all short and not-thin people are poor distance runners. Now you have to go back to the drawing board and decide what 'good distance runner' means and how to measure it.)

If something that simple involves so much work, something as complex as climate probably takes quite a bit more care. So, irrespective of much else, one thing to look for in reports about what is or isn't the case about climate is whether the author showed as much care in making their tests as I suggested for something as trivial as deciding whether someone was 'tall and thin'. In part 2, finally, I'll get to the examination I was invited to make.