Skip to content

The Perils of Not Knowing that You Don’t Know

Suppose you needed to get from New York to Washington for a personal emergency. An airline told you that the projected time of departure for your one-hour flight was 2pm this afternoon. Of course, there is some uncertainty about this. The unstated promise of the airline is that this uncertainty will be kept within familiar bounds, with delays up to an hour not unusual, and delays of several hours possible with unexpected weather or mechanical failure. But suppose the airline secretly knew that due to a large set of unreliable links in the chain like a flight attendants’ union that might strike, a large risk of major engine failures throughout its fleet, and problems with the mechanics’ union that might prevent repairs, that the possible delays were not in HOURS, but in DAYS.

You would be angry at the airline for not telling you this, and once you knew how much they didn’t know, you would quickly drop the airline and take the train instead.

The moral of the story is that knowing how much uncertainty there is about a projection – that is, knowing how much the projector DOESN’T KNOW – is often more important than the projection. An “estimated departure time based on the best available information” is meaningless to an airline customer if the uncertainty is about DAYS rather than HOURS.

Not knowing that you don’t know is at the root of many recent disasters, starting with the crisis itself. Holders of opaque derivatives apparently didn’t know how leveraged and exposed they were to shocks like falling housing prices, nor did they know how uncertain the housing prices were. Outside of economics, not knowing what we didn’t know is one of many causes for bad outcomes in Iraq and Afghanistan.

Not knowing that you don’t know was our concern about the World Bank’s global poverty projections. In his response on this blog and on the New York Times blog Economix, Ravallion misses our point. It is the degree of uncertainty of his poverty projection that is unacceptable. He said: “Faced with all these perceived “impossibilities,” Easterly and Freschi would apparently prefer to wait and see rather than take action when it is needed, based on the information available at the time.” Ravallion called our stance “analytic paralysis in the face of uncertainty.” He portrays us as unwilling to live with ANY uncertainty, which is ridiculous. Economics is filled with uncertainty.

But when the uncertainty is so large as it is with the poverty projections (for all the MAJOR reasons we pointed out, which Ravallion does not address*), then the implication is not “paralysis” but choosing actions that take into account the uncertainty. For example, you DON’T want to have a centralized bureaucracy like the World Bank allocate global poverty relief based on such wild uncertainty. It would be better to support local coping mechanisms (public or private) that flexibly respond at the local level using local knowledge about crisis effects like poverty, hunger, and dropping out of school.

An article in the Wall Street Journal illustrated well how uncertain theoretical economic predictions can be used in the very un-theoretical realm of influencing how scarce aid resources are directed, and to whom:

A joint development committee of the World Bank and the International Monetary Fund estimates the lingering financial crisis could drive an additional 90 million people around the world into “extreme poverty.” To combat the staggering statistic, the World Bank is aggressively lobbying for its first capital increase in two decades.

Not only would it be the wrong response to channel poverty relief through the centralized bureaucracy, but confusion is here piled on uncertainty. The capital increase is NOT for lending to the poor countries, but to the middle-income ones, as was made clear in a number of other news stories which explained that the money would be shared between the IBRD, the branch of the Bank which lends to middle-income countries on near-commercial terms, and the IFC, which lends to companies.

This only strengthens our original argument that the poverty projection was a political exercise. We respect Ravallion’s academic work, but this exercise seems to be in a different category. Not knowing what you don’t know is indeed dangerous.

*The closest Ravallion comes is citing his paper that tests the assumption of distribution neutrality of growth ON AVERAGE, which still misses our point about variance around the average. His claim that what people thought would happen before the crisis is a good benchmark for what would have happened without the crisis simply begs the question of the accuracy of country growth forecasts–not responding to our concern that they are radically unreliable (with or without crises). Since his point is invalid, it remains true that his projection of the effect of the crisis on growth is based on no meaningful evidence.

  • email
  • Facebook
  • Twitter
  • Digg
  • Google Bookmarks
This entry was posted in Arguments, Logic and Use of Evidence, Research and Statistics for Good and Evil. Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.


  1. Matt Berkley wrote:

    A common element in both the creation of the financial crisis and in common errors about large-scale statistics presented as being on poverty may be this:

    To cite income statistics as indicating prosperity or poverty is in part to mistake income with profit.

    To cite consumption expenditure statistics as indicating prosperity or poverty is to mistake consumption (in the sense in which non-economists would use that word) with spending.

    A notable example of this kind of error occurred with the widespread belief that income showed profit to individuals under different policies or conditions

    (”growth is good for the poor”). For such arguments, no premisses about the relative cost of living under different policies or conditions were employed.

    To the extent that parasitic activities were counted as productive in conclusions about trends in prosperity, a similar error might be thought to have contributed to the financial crisis.

    Posted October 8, 2009 at 7:30 am | Permalink
  2. The key point that Easterly and Freschi miss is that, for the purpose at hand, we are only trying to estimate the aggregate poverty impact, such as the 50 million figure for the number of extra people who will find themselves living below $1.25 a day this year due to the crisis. (We have to use a forecasting method since sufficient household survey data will not be available for a couple of years, by which time the crisis will hopefully be over.) As we have explained very clearly I think, we are not trying to estimate the impact in specific countries, for which the relevant Bank country teams and others use much more country-specific information, as required.

    Our estimation method relies on aggregation across countries to increase precision. (The source paper is available at the VOX site: This works pretty well. Indeed, when we use this method on past economic contractions–pretending that we do not know the poverty impact and then checking how well the method performs–we find the predictions of the aggregate impacts are very good (as is shown in the VOX article). I am not sure how they compare to airline schedules but they are certainly acceptable by the standards of economic forecasting.

    This whole critique is greatly exaggerated.

    Posted October 8, 2009 at 11:03 am | Permalink
  3. Bill Easterly wrote:


    I continue to be surprised that you don’t respond directly to our concerns about how margins of error on each of the links in the chain to get a poverty projection are very large.

    You just keep mentioning this evidence on how you came out ON AVERAGE with one of the more minor links in the whole chain, which even for that one doesn’t address the variance around the average.

    We’ve given evidence on huge margins of error on very important links.

    To be more precise, one that I know well from my own research is that the confidence interval for the transitory (i.e. mostly unforecastable) component of annual growth rates is on the order of [-10,+10] percentage points. To that known unknown, you add the completely unknown unknown of how the crisis affected growth.

    You don’t think this is a wee bit of a problem?

    Best, Bill

    Posted October 8, 2009 at 11:31 am | Permalink
  4. jose wrote:

    Yes, poverty forecast in a world crisis has a lot of uncertainty, but there is little doubt that poverty increases. Is it dangerous to have some educated guess about the trajectory? I do not see it.

    Christina Romer and her team projected unemployment with and without stimulus. Her projections were too low, which presumably indicates large uncertainty. Was it a useless (or even dangerus) exercise? On the contrary, in my view, it helped guiding the policy discussion.

    Posted October 8, 2009 at 12:39 pm | Permalink
  5. jose wrote:

    Your only prescriptive point is the following: “It would be better to support local coping mechanisms (public or private) that flexibly respond at the local level using local knowledge about crisis effects like poverty, hunger, and dropping out of school.”

    But how do you know which locality to support? Unless you mean supporting all localities of the world, in which case you are saying no supporting anyone.

    Posted October 8, 2009 at 12:44 pm | Permalink
  6. Anna wrote:

    jose, I disagree that making policy decisions based on projections that have a huge margin of error cannot be harmful. Among other things it allows policy makers to come up with whatever projections necessary to support their agenda.

    As far as Christina Romer’s projections go – now they are of course simply saying that matters would have been even worse without the stimulus.

    This means they believe they know how many jobs the stimulus created, but for some reason they were waaaay off in projecting the unemployment rate sans stimulus. There is no way of proving this and it is simply the most convenient answer a politician can give. Without any implications from these incorrect forecasts – if you can always explain your mistakes away by saying the world would have been even worse off without you – how can they be of any use besides pushing some predetermined agenda? The results of these types of interventions need to be measurable, or they should not be taken. Otherwise important resources might be diverted from their most productive use on a massive scale.

    Politicians and bureaucrats shouldn’t have the power to meddle with something they don’t fully understand.

    First, do no harm.

    Posted October 8, 2009 at 4:06 pm | Permalink
  7. Even in a world of certainty about the future (which doesn’t exist), significant variance obviously can exist within a population. So two categories of variance have to be addressed separately here:

    1. Variance attributable to uncertainty about the future (variance Easterly is emphasizing, if I understand correctly).

    2. Variance in the correlation of aggregate outcome measures with direct measures of well-being (capabilities, functiongs … literature to which Ravallion has been such an important contributor).

    Since the concern in this discussion is with one end of the income distribution, it seems to me that the second source of variance is at least as important as the first.

    Getting to the point: The unwelcome (to almost all) global recession was accompanied by a very welcome (to most) bursting of the 2008 commodity price bubble.

    So the real question from my standpoint (unanswered by the exchange above, unles I missed it, and imperfectly addressed to my mind by the WB analysis in question) is to what extent decreases in commodity prices, particularly for high-income share items like staple foods and cooking oil, more than offset losses via diminished remittances and economic opportunities. Your not going to get that by looking at averages, even if they are forecast with certainty.

    Another possibility is that it isn’t the levels of commodity prices that matter most, but their own intrinsic variance (ref. ). That may be true, but the implied policies are very different than they would be from those based on arguments that focus on levels… and in this case I really can’t see how they would include more money for Bank lending–whether to low- or to middle-income countries.

    Posted October 8, 2009 at 5:42 pm | Permalink
  8. Matt Berkley wrote:

    Martin Ravallion wrote:

    “we are only trying to estimate the aggregate poverty impact, such as the 50 million figure for the number of extra people who will find themselves living below $1.25 a day …”.

    How do statistics on what people spent tell us about the adequacy of what they got in return?

    Posted October 8, 2009 at 6:28 pm | Permalink
  9. jose wrote:


    There are two discussions here, one about forecasts (and margins of error); another about which is the best policy (given some estimates and uncertainty).

    With respect to the first one, I think we just have to keep trying measuring things better. The argument of Easterly seems to be: “estimates are imprecise, so let´s just quit”. I did not read anything from him (in this discussion) indicating how we can get better estimates.

    On the other hand, we do have the past, and in all crisis poverty increased, so there is well founded evidence that allows us to forecast (very imprecisely) the effects of this one.

    With respect to the best policy, given those imprecise estimates, we can only use a mix of theory with analysis, with what worked in the past. In the case of Romer et al, there was evidently a consensus (based as I say in a mix of theory, analysis, and history) that doing something was better than nothing.

    At the international level, I do not see why trying to solve some problem would not be better than just accepting it. Yes, it can fail, but at least we would have learnt something. After all, “trial and error” is the basis of most progress.

    In sum, yes, I would like to see more accountability of the international organizations, but also I would like to see them with better statistical agencias (to measure better) and with more power (to be able to apply best practices in public policy science).

    Regards to all.

    Posted October 8, 2009 at 6:39 pm | Permalink
  10. jose wrote:


    One more thing: not doing anything is one particular policy. You might be doing much more harm by that.

    Moreover, the “government” (politicians, bureaucrats) is always doing something. In fact, without them, modern economies (by that I mean rich) would not exist.

    The big question here is what government do we need to establish the conditions for a prosper and fair world. My answer at this moment (although far from settled) is better measurement, better social science, and more power for international organizations.

    Posted October 8, 2009 at 6:51 pm | Permalink
  11. zulusafari wrote:


    A-freakin-MEN (as in AMEN). I guessing the economist that sepecialize in this wrealm have some kind of explanation for this.

    Personally, I am totally against using any $ figure to identify a person as ‘poor’ or not. I guess economist have to use some kind of cold fact to ‘project’ poverty, but the majority of the world who live at or below the poverty ‘line’ ($1.25/day) are by and large doing okay in their community. The average person in my community has around $40/month, just a hair under that poverty line. The community here is fine and not ‘poor’ by and large. By and large they are healthy and living productive lives.

    Sure they could use ‘development dollars’ but no way is this community in need of aid (handouts). Despite the fact that I do still see WFP & USAid handouts (just a few days ago). Haven’t seen any of it for sale in the market yet, but I’m sure it’s there. Just haven’t seen it yet.

    Posted October 9, 2009 at 1:19 am | Permalink
  12. geckonomist wrote:

    economic data?

    In my poor country, almost all economic activity flies below the radar of the World bank and their poverty experts. It’s called, the informal sector. I takes place in areas they would never bother to go.

    I meet them here and there in the expat community but they don’t see the booming (informal) housing sector (nobody gets VAT receipts anyway =no data), the scarcity of building materials, the smuggling,

    they don’t notice that the (non-traditional) agricultural exports grew by 1000% in the last decade,

    they don’t see those few government officials that surprisingly own all the prime properties in town where they like to hang out so much,

    they don’t wonder how 1 of many mobile phone companies can make a real hard currency profit that corresponds to 15% of the official GDP (!), etc.

    They never doubt themselves and their data, because they’re the real experts.

    So they do know that the GDP grew by 5% last year, 1% less than expected because of power cuts, corruption and bad roads.


    I can safely conclude they have no idea and no poor man would miss them and their narratives.

    Posted October 9, 2009 at 3:57 am | Permalink
  13. Shakeb Afsah wrote:

    Ravallion confronts some research realism here–this blog actually has some feedback from readers who are supposedly part of Ravalion’s stat but seem to disagree with him. In any case, it would be good for Ravallion to respond to Easterly’s original point about the margin of error. I went throug

    Posted October 9, 2009 at 9:20 am | Permalink
  14. Anna wrote:


    Thanks for the response.

    I don’t believe Easterly wanted to say “…so let’s just quit”. I understood it as: let’s be honest about what we don’t know. Of course we should try to estimate unknowns, but we have to include the margin of error in our estimates so policymakers and others don’t make decisions thinking they have better information than they actually do. An average of 100 tells me very little if I don’t know the variance. Is it 0.05? Is it 50? If it is 0.05 I might take action A. If it is 50 I might take action B or decide to do nothing. Easterly’s point is, I think, that in an environment of such uncertainty better progress can be made on a micro-level where individuals get to make decisions about what is best for them.

    Also, I personally don’t view doing nothing as being a policy because any action by government (or other similar institutions) requires that a case be made for that action. Inaction is the default state so doing nothing does not *cause* harm.

    Trying to solve problems that you have a very limited understanding of can indeed be harmful if the actions cause incentives to get distorted, productive behavior to become undesirable, corrupt parties accumulate power by their relations to the institution etc. and there is no way for you to dissect cause and effect.

    “Trial and error” is impossible without some credible measure of the success of each trial.

    Posted October 9, 2009 at 10:03 am | Permalink
  15. The Randomizer wrote:

    I think there’s any easy way out of this. Obviously, people making quite ambitious predictions which rely upon counter-factuals really don’t want to publish the confidence intervals right next to their “best guess”, because quite likely those confidence intervals may come very close and even cross the line of equality. An alternative – and I think more honest – way to deal with and present the uncertainty inherent in such estimates is to simply publish a histogram plotting the predicted probability of observing any given outcome. This would address Easterly’s concern that the uncertainty is such estimates is not presented along with the “best guess”, but also prevent people focusing on the ultimately arbitrary confidence interval over the more interesting number that is the “best guess”.

    Posted October 9, 2009 at 11:41 am | Permalink