Successive approximations

The other title for this would be 'the relativity of wrong', if I were to steal the title that Isaac Asimov used for an essay on a related topic.  It's a common issue in science, and successive approximations is both a common and a powerful tool.  In the comment section for 800,000 years of CO2 we had a local introduction to the issue.  But I'll start with the example that Asimov used, and that I have a number of times myself.  (It's entirely possible that I used it because he did -- I've read most of his science and science fiction writing.)

Let's start with the shape of the earth.  At some distant time in the past (probably more distant than you're thinking) we start with the thought that the earth is flat.  A different version of thinking about this is that the radius of a spheroidal earth would be infinite.  Clearly there are local variations, as any runner, hiker, or biker can tell you.  But basically a flat surface.  Get out to the Great Plains, some of the major deserts, or, especially, the coasts, and it's obvious that the earth is flat.  Just look!



Of course it isn't actually flat.  And this was known by several hundred years BC.  Aristotle (died 322 BC) dismisses the shape of the earth being a sphere as common knowledge in his Meteorologica.  He offers a couple of illustrations of how we know this, but it's not a challenging matter.  Something like how today we might mention that the Beatles were an influential rock band and then point to how many records they sold, or number of number 1 songs.  The earth was known to be round no less than several hundred years earlier than that in India (though I can't lay my hands on the reference right now).  The approximate size of the spherical earth was figure out by Eratosthenes in the 200s BC.  How accurately he did so depends on which of several versions of 'stadia' (unit of distance) you choose.  But certainly far closer to correct than to take an infinite radius (flat earth).  That's one part of the relativity of wrong.  The very ancient people who took the earth to be basically flat were as wrong as it is to wrong to say that infinity equals 6400 (roughly the value, converted to km, of the most favorable estimate from Eratosthenes). 

Two things here.  One is, the measurement method that Eratosthenes used is one that middle and jr. high students can carry out today.  I'll be happy to help schools do this (you do need a pair of students at different latitudes, but same longitude).  It's an experiment well worth doing.  The second is, notice that we've started the successive approximations -- first estimate was infinity.  Second estimate is 6400 km.  That's a huge change.  But almost all that change was due to the fact that the second estimate was based on observation, versus saying 'it's obvious'.

If you continue your observations, and make them increasingly accurately by using methods that Eratosthenes didn't have available, you get towards the earth being a perfect sphere with a radius of 6371.2 km (give or take -- history intervened).

 But the earth is not a perfect sphere either.  Now, the error in flat earth vs. spherical earth is infinite (infinity - 6400 is infinity).  The error here is distinctly smaller.  Concern about it goes back to about Newton's Principia, and an argument that ensued between Newton and Cassini.  Both very sharp scientists, both very knowledgeable.  Newton argued that the earth should be bigger around the equator than from the center to the pole, because as the earth rotated, the equator should be slung out farther from the center.  Cassini argued essentially the converse.  (See Chandresekhar's Ellipsoidal Figures of Equilibrium for a more detailed discussion of the arguments and the history.)  It wasn't until observations made in 1738 by Maupertuis and Clairaut that the argument was resolved.  As mathematics, neither Cassini nor Newton had made an error.  Newton, however, was more correct as to what physical processes to be doing the math on.

The equator being slung out more means that the earth is not a perfect sphere, it is an oblate (bigger around the middle) spheroid.  We have 2 different radii now, the equatorial (6378 km) and a polar radius (6357 km).  The two differ by 21 km.  So, while it's incorrect to say that the earth is a perfect sphere, the difference between that and the oblate spheroid is no greater than 14 km (the 6371 km best sphere radius versus the polar radius) and generally smaller, depending on where you try to get the radius.

Once you finish building your mathematical model for an oblate spheroid, and get very specific about exactly what you mean by "the earth's surface", you can then find that the geoid (which is what is used -- the surface that water would all wind up at if there were no currents -- the 'bottom of the hill') deviates by upwards of 100 meters from the best oblate spheroid (which are now specified to about 1 meter, not the 1 km that I rounded to up there).  So, while oblate spheroid isn't exactly correct, it's within about 0.1 km out of the 6357 km polar radius, or 6378 km equatorial radius.  And once you get to paying attention to ocean currents, you need to make corrections of upwards of 1 meter for variations in the currents.  Plus, no doubt, there are other processes. 

As we learn more, our range of error decreases.  Each shape is an approximation to the real shape, so our successive approximations get increasingly better.  But, particularly once we discard a flat earth as candidate, we also need the prior estimate of the shape to make our improved estimate.  It's something of a bootstrap process as well.

Application 1 -- doing science and reading blog posts
The case that just played out in the comments on 800,000 years of temperature and CO2 is a simpler illustration.  I found the best fit line between temperatures 1000 years earlier and CO2 concentrations.  Afterwards, I computed the standard deviation of the difference between the CO2 you would expect, given the temperature, and the CO2 that was observed.  That was 11 ppm.  A reader did the step you would do before what I did and found 26 ppm.  What he did was take all CO2 observations, and the principle that we know nothing about what's going on -- a good starting point -- and ask what the standard deviation was. 

When we start from honest ignorance, we see 26 ppm standard deviation.  After we've looked farther at the data, we see that CO2 correlates well with temperature.  So much so that if we use our knowledge of temperature, we reduce the standard deviation -- of the difference between what we predict from that relation and what is observed -- to 11 ppm.  That's a measure of how much better our knowledge is now -- our initial uncertainty was 26 ppm, and we've cut that to 11 ppm by understanding something about the correlation between CO2 and temperature.  To do better (which we will) we'll have to bring in additional knowledge -- that both temperature and CO2 are affected by orbital variations.  The residual standard deviation then (residual meaning difference between our informed expectation and observation) will be something smaller.  How much smaller will tell us how much better our understanding has gotten.

One last bit here: From our initial honest ignorance, the modern 365 ppm (it's now about 387 ppm, but 365 was what was used in the data set we were analyzing in that previous post) was extremely surprising.  The average of all CO2 observations was 224 ppm, and had a 26 ppm standard deviation.  The modern figure was somewhat over 5 standard deviations from the norm, and was the only one more than about 2 away.  We would already suspect it was an outlier just from our initial honestly ignorant look at the CO2 data that something was extremely different about that last figure.  When we learn a bit more, that temperature and CO2 are correlated and temperature leads by about 1000 years, our surprise at the modern value increases enormously.  The modern figure is now 9 standard deviations away from that very good correlation between temperature and CO2.  The true modern figure, the 387 ppm, is 11 standard deviations away from the relationship that had held for the previous 799,000 years!  As we learn more about the system, the modern values will get even more astonishing.

Application 2 -- selecting sources and engaging in discussions
Honest ignorance is a very good thing.  And ignorance, in its own right, is nothing to be particularly concerned about.  We're all ignorant of enormous amounts.  If you think you're an exception, do as I once did: walk in to a library and ask yourself how many shelves there are where you've read every  book (if literature or the like), or know the full contents of the shelf (if it's, say, history of the Roman Empire shelf, or math, etc.).  There were a few shelves where I had the full content.  But it was very few.  So I'm not concerned about the fact of my honest ignorance of very many of those shelves, nor concerned if it should turn out that somebody else's honest ignorance includes a shelf that I do know.  Plenty of room for good discussion with someone honestly ignorant, and I'm sometimes the one asking those ignorant questions.  That's how I become less ignorant.

This is something of an issue with the relativity of wrong, and carrying on discussions in blog world or even the 3d world.  If someone says that they haven't looked at much data, but think the current CO2 is not really very much higher than it has been, you can have a discussion.  They're wrong, but it's an honest ignorance sort of wrong.  They know that they don't know much, and further information will be a plus.  Provide that information if you can; if you can't, bring in a friend who can.  The scale of error here can be very large (even as large as the earth being flat), but the person is not very invested in that conclusion.

Something of an intermediate situation is something I did here myself.  Namely, the 9 (or 11) standard deviations I computed for how different present CO2 is from what we'd expect, given the temperatures, has an (at least one) error.  The CO2 observations, and temperature observations, aren't all independent from each other, but my computation assumed that they were.  In truth, if it was hot and high CO2 for one thousand year period, then the next thousand years probably was too -- the figures aren't independent.  That means the correct standard deviation is different (larger) than I computed.  You have three situations here: 1) the authors knows what they're doing and trying to misrepresent reality 2) the authors know what they're doing and are trying to keep the illustration simple 3) the author doesn't know what he's doing and is just accepting an agreeable conclusion.  Maybe some more.  I'm one of those category 2 people (at least in these posts).

Category 2 folks will be wrong, but you'll have little difficulty explaining their errors to them and getting them to accept the correction.  Category 3 folks may or may not be wrong in their conclusion, but either way they can be very difficult to have a discussion with.  They think they know a lot, but are actually just committed to the answer that they like.  The reasoning is merely window dressing for the answer they like.  Category 1 folks, those who are intentionally misrepresenting the situation, are obviously (I hope it's obvious to you!) impossible to have a discussion with.  They might debate you, but debate is not discussion.  The scale of wrongness here is, say, to be treating the earth as a sphere rather than the more correct oblate spheroid.  It's wrong, but it's only wrong to a modest degree and it's one that (at least for categories 2 and honest 3s) can be fairly readily corrected.

Then out on the extreme, you have the people who say things like "Temperature leads CO2 in the ice age record therefore the current CO2 rise has nothing to do with humans."  They might be in the previous class, maybe categories 1 or 3.  But they're enormously extreme -- the magnitude of the error involved is no longer something where they can take moderate steps, or learn just a few things, to get to something reasonably close to what our scientific understanding is.  Here, it's more the person having decided that the earth is a cube, or an enormously flattened sphere (say 2000 km difference between polar and equatorial radii), or ..., well, quite a number of bizarre shapes that have been suggested.  They're so wrong that your first order of business can't even be to talk about the real observations of the shape of the earth.  If you engage this at all, I'll suggest that your first steps should be towards understanding how the person got to such a wildly incorrect view, and why they are so attached to it.