Verifying forecasts 2

As I said last week, verifying predictions is difficult, and was prompted in to looking again at the matter by someone doing it wrong.  Of course the standard of 'wrongness' involved is mine.  Forecast verification is something of an art as well as mathematics and science.  But some points I think I'll get little argument from Allan Murphy* and his intellectual colleagues and descendants for are:

• You have to be clear what you're forecasting
• what variable
• at what time (or time span)
• for what place or area
• You have to be clear how the forecast is going to be evaluated
• You should evaluate all forecasts
• Forecast must be public
• Forecasts must be verifiable

That last might seem a little strange.  I hope not.  Suppose I said next July 20th at 3:34 PM at Washington National Airport the official temperature would be hot.  Very specific about what I'm forecasting and what it will be evaluated against.  But what is 'hot'?  To me, anything over 80 F (27 C).  As such, it's a near certainty that my forecast will be correct.  It's also awfully easy for me, on July 21st, to say, regardless of the temperature, that it was 'hot'.  This is one reason that we prefer numbers in science.  You can, and we do, work with qualitative predictions.  But it takes more work, as you have to find some way of making 'hot' objective, so that we can all agree that such a forecast was correct or not.

In general, if not as universal, we add a couple more items, at least desirable if not mandatory:

Verifying forecasts 1

I already discussed my earlier sea ice estimates and how they came out, but a few things have happened since then to occasion a two part look at forecast verification.  As usual, it's prompted by seeing someone do it wrong.

One of the errors, which I have to remedy on my own part, is that you should verify (compare to reality) all your forecasts.  I think that the end of May ice estimates are the most interesting and important, rather than later in the year.  Partly this is because of how I think the sea ice pack behaves.  Partly it is because the practical uses of sea ice information I know of require that kind of lead time.  It takes a long time to get a tanker up to Barrow from Seattle, for instance.

Xingren and I did submit a later estimate, for the August Sea ice outlook.  That estimated 4.60 million km^2 for the September average sea ice cover.  An excellent approximation to the NSIDC's reported minimum (4.60) but not as good compared to the observed average extent of 4.90.  Actually a touch worse than our May (30th, even though not reported by SEARCH until June) estimate of 5.13 from the model.  Both estimates were well within 1 standard deviation of the natural variability (errors of +0.23 and -0.30 for May and August's predictions, respectively, versus about 0.5 for the natural variability).  So, on the whole, pretty reasonable.  Just that we'd have expected better from the later estimate.   But ... there's more to that story ...

Thanks Teachers!

Quoting one of my sisters' pages:


Tonight, a teacher somewhere in your community is preparing lessons to teach your children while you are watching television. In the minute it takes you to read this, teachers all over the world are sacrificing their own time and, more often than not, investing their own money for your child's literacy, prosperity, and future. Re-post if you are teacher, love a teacher, or appreciate a teacher!!!

As for most things, the most media coverage is of the bad performers of a profession. But I grew up seeing my grandmother (another teacher) doing exactly as described, and see my sisters doing so when I visit them.  And some of my teachers, my kids' teachers, and so on were/are obviously doing likewise. 

It's American Education week, so I'll invite folks to contribute their stories of favorite teachers this week.

Vererans Day

Thank you to all veterans!

Happy Veterans day to all, my son included.

Sorry about the late word.

Sea Ice Predictions vs Reality

Ok, I didn't jump on the end of the ice season.  But, the good thing about doing science is that being right or wrong, or learning from your mistakes (or learning from your right answers, even if that's harder to do) is not a matter of a 'news cycle' or what is currently 'hot' in the blogosphere.

The observed ice extent for September 2010, monthly average, from the National Snow and Ice Data Center was 4.90 million km^2.  One thing about making your predictions and deciding how well you did is that you also have do be specific about what you're going to compare against.  You'll find somewhat different figures if you look at other places. 

If you were dishonest, or just not careful, you might select whichever observation was closest to your  prediction.  The problem with that is that it then becomes easy to claim an accurate prediction -- with little regard for the quality of the prediction itself.  Just select the most favorable observation, or process the data yourself in your own way.  (By changing how you do your land masking, you can change your ice areas or extents by upwards of 1 million square km.  ... he said with no tinge of annoying experience.)

It turns out that my May predictions did pretty well. 

Young scientists

In Knight anoles, you got to see part of the reason I made one of my goals for this blog to be inclusive of middle school students.  They can be quite interesting to listen to about science, and can learn quite a lot of it themselves.  Biased as I am in being a father and uncle, I still believe that kids other than mine can match, or at least approach :-) mine.

So I'll mention that if you're a teacher, parent, or a student yourself, and your kid/you write up a science essay, you're welcome to submit it here for consideration.  I'll look for the essay to teach me something about the science, and to show the love of learning about your topic that Kristen showed for hers.  As you might guess from my usual topics being climate and ice, but this note of Kristen's being lizards, you're not limited to my professional areas. 

There will be details to work out, maybe later we'd want to establish it independently of this blog.  But think of it as something in the vein of Journal of Young Scientists.  A chance to share your love of your topic with others.  You can send to me at bobg at radix dot net.  We'll play things by ear.  I've created the tag 'young scientists' and retro-applied it to Kristen's (first! :-) note.

Knight anoles and science writing

What Are Knight Anoles?

By: Kristen Martinet
December 15, 2008
Liberty Middle School
Science/ Period 2

Abstract
Knight anoles are very interesting lizards. They are the largest anoles in the world and have very distinct features such as their speckled backs and striped sides. These reptiles are an invasive species in Florida and originate from Cuba. People like to keep knight anoles as pets, but then release them into the wild without knowing the consequences for the lizard. This makes them more abundant in urban areas. They eat insects and other lizards in the wild and in captivity. When fighting off a predator, the lizard bluffs to scare it away. While fighting with other males, the anole bobs its head up and down and extends the dewlap to look tough.  In the summer, knight anoles breed to create at least eight new baby knight anoles in five-seven weeks.     Knight anoles (anolis equestris) are a very interesting species of lizard that are also called the Cuban anole. This reptile is part of the order squamata, the sub-order iguanidae, and the family polychroidae. The knight anole is part of the genus anolis, which has about 250 species (Crowther, 1999). A researcher from Centralpets.com stated that the common name “knight” is derived from the Latin species name “equestris” which is derived from “equester,” a Latin word for knight. The other common name, Cuban anole, is probably used because its first home is in Cuba.

Election Day

US readers:
Vote today, Tuesday November 2nd.
If you're not registered, get registered.
If you do neither, but are eligible, you don't get to complain about the results.