I have no reason to doubt that the writers at The Economist, and Jane Galt, are truly educated people, if only because, as George Bernard Shaw noted, the mark of intellect is a passion for statistics.
The Economist’s new blog demonstrates this passion in the ongoing debate of the Lancet Study. The posts continue with the issues raised in (surprise surprise) Assymetrical Information (amongst other places) such as the inconsistency of the Lancet numbers with the Iraqi Body Count but this time we also get an accurate, if diverting and rather patronising lesson on statistical errors. It is also telling. The purpose of the statistics lesson is to draw attention to a potentially fatal flaw in the Lancet Study but The Economist used some sleight of hand here.
This [systematic] is the sort of error that critics are alleging that Burnham et al. made. This is the sort of error that plagues most scientific studies that turn out to be wrong. Thus, stating (over and over) that “their statistical methodology was sound” is irrelevant. What critics saying is not “their calculations were faulty” but “there was something wrong with the sample”. It is perfectly possible that the statistical methodology was sound and that there is something wrong with the sample. Computer scientists have an acronym for it: GIGO, or Garbage In, Garbage Out.
Did you see it? The suggestion is that Burnham is wrong because of the statistical error.
But what does “Burnham is wrong” mean and does it automatically follow from the existence of the sampling error?
One interpretation of “Burnham is wrong” is that the oft quoted 650,000 figure is wrong. To anyone with a basic statistics background though that is evident from the results of the study. 650,000 is only the mean of a distribution that lies with 95% probability between 400,000 and 900,000. In this sense, I agree that Burnham is probably wrong but only because I am looking at the distribution as a whole rather than focusing on one point estimate.
A less charitable interpretation of “Burnham is wrong”, and the one I suspect The Economist is trying to promote, is that the whole study is only more slightly believeable than the results of Saddam’s last election victory. But does the existence and size of the sampling error mean we can throw out ithe conclusion of Burnham in its entirety?
Well of course it depends on what you think the conclusion is. My statistics lecturer used to stress that the underlying message of any analysis had little to do with the numbers. That is especially true in this case because the true message has to be that since the invasion, things have got a lot worse for ordinary Iraqi’s when the promise was that they’d get better. If you think Burnham is wrong in the second sense, then this is the conclusion you disagree with. Now, would the existence of a sampling error reverse this conclusion? I very much doubt it because that would require the actual distribution to contain negative deaths (i.e. more individuals are living now than under Saddam) as well as the number zero – which would suggest the invasion has had a neutral effect on deaths: no more and no less than under Saddam.
Now not even the IBC’s measures include zero so things have demonstrably got worse, so in that sense, Burnham is not wrong. Critics of the study then are either trying to deflect attention away from the failure of the Iraqi policy and/or, even more depressingly, neglecting the need to change that policy.
Perhaps that policy doesn’t need to change because, as The Economist post points out, time may prove that half a million or a million deaths was a price worth paying by Iraqis to rid themselves of Saddam. But then let’s have this debate. Let’s talk about the benefits of the post-Saddam Iraq in terms that go beyond the hyperbole of George Bush. Let’s talk about what Iraqis expect and want in the future and what price they are willing to pay. Let’s talk about the exact nature of their trade-off. Thus far all I hear is the benefits to the West of a democracy in the Middle East and the cost to US/UK taxpayers. What the Iraqis want to know is, how does their substantive freedom from tranny translate into a real capability to enjoy that freedom?
There is nothing wrong in being passionate about statistics. Indeed Keynes’ passion for the subject was second only to fornication. He thus not only proved Shaw’s assertion but also demonstrated a healthy preference ordering. He also knew exactly what he meant when he said “I’d rather be vaguely right rather than precisely wrong”. Burnham, even with a moderate sampling error, is indeed vaguely right and to argue that the situation in Iraq is a lot better is, I’m afraid, precisely wrong.