For precipitation forecasts, people (science fairs included) often think about either 'probability of detection' -- i.e., what fraction of the time that there's rain did the weather forecast call for rain, and 'false alarm rate' -- what fraction of the time did you get no rain even though the forecast called for rain. Both are potentially meaningful, and both have serious problems if used alone.
The meaningful side is that someone who really doesn't like rain, or needs it not to rain (say farmers after applying certain fertilizers/insecticides/...), wants a very high probability of detection (PoD). Conversely, if you take some expensive actions given a forecast of rain (rearrange your schedule, and you don't like doing that, apply some insecticides that need rain, ...) and it doesn't materialize, you want a very low false alarm rate (FAR).
But it is very easy to cheat either one of those measures of skill. My PoD score will be perfect if I say, every day, that it will rain. It's guaranteed on the days that it rains, that this will have been my forecast. But I'll have called for rain a lot of time that it didn't happen. Conversely, I can get a perfect FAR score easily -- forecast that it will never rain. If it ever does rain, then I'll be wrong, but the FAR doesn't care about that error. PoD and FAR cover each other in this respect -- each is sensitive to the cheating you might do against one of them.
There are actually 4 conditions that can occur:
- we say that there will be rain, and there is
- we say there will be rain, but there isn't
- we say no rain, but there is
- we say no rain, and there isn't
There are many more scores for this kind of situation. But this is enough complexity for us to start looking at using black and white vision to analyze sea ice extent, that is ice / no ice analysis with the satellite data.
For my first trial, I took the SSMI on F-15, the same 19 GHz, horizontal polarization, channel that ESMR had, for August 1, 2011. Then I lopped off all land points (no reason to give credit to an algorithm for noticing there's no sea ice on land), all northern hemisphere points (it's the Antarctic that's of most concern for the ESMR period), and all southern hemisphere points north of 48 S (an arbitrary round number -- the point being that we know before starting that there isn't and wasn't ice that far north from Antarctica). That trimmed things down to about 200,000 observations.
To start with, I just made a scatter plot of all the observed brightness temperatures against the concentration:
Having reassured myself that even with only 1 channel, and even with the ice temperature effect being ignored (for now), we get a plausible scatter plot, next is to think of some algorithm (method) for going from the observed brightness temperature to an ice / no ice decision. What I'll do is take the algorithm "If the temperature is above this number, ice is present in the field of view". And define "ice is present" to mean that the ice analysis has greater than 15% ice concentration. The 'this number', the critical temperature in my algorithm, I'll simply vary all the way from 80 K (colder than the coldest in the diagram) to 273 K (melting point of ice) -- and see what the scores come out to be.
If I take an extremely cold brightness temperature, I can get perfect Probability of Detection. But the False Alarm Rate it horrible, as is the % correct. The interesting zone, for our algorithm evaluation is between about 120 and 160 K. Probability of Detection is getting worse all the time through that range, but False Alarm Rate is improving (getting smaller). The % correct rises rapidly to its peak, at a temperature around 140 K and then declines slowly. Let's magnify the upper parts of the curves for PoD and % correct:
We're far from done, of course. This is just one day, we've paid no attention to the possibility of using surface temperature estimates (climatology, weather analysis, other satellites, ...) to improve our estimate, the algorithm is extremely simple, and so on. Also, this is a check for each individual observation. For obtaining estimates of Antarctic sea ice extents, we want full grids of ice / no ice decisions. Some of the observations are in the same cell as others, so perhaps the few we're getting wrong would be corrected by others that were in the same cell. And ... add your own in the comments!
Still, the first exploration here suggests that the 1 channel only method can give a probability of detection better than 95%, and % correct around 97%. We'll have to think some about whether false alarms are worse than failure to detect, and see how this holds up as we look at more days and methods.