ELDRLIVE
$ eldr --course politics --day 03

Day 03 of 07

Base rates and the news

The news is a sample selected for being unusual. That single structural fact explains most of the gap between how the world feels and how it measures.

Course
Politics: checking a claim, including your own
Minutes
30

Claim

A base rate is how often something happens in general

Before you can judge whether an event is a trend, you need the denominator: how many of these are there, over what period, in a population of what size.

This is the number almost no report gives you, because it is not news. Fourteen incidents is a story. Fourteen incidents out of ten thousand cases, down from nineteen last year, is a different story that nobody will print. Neither version is false. One of them lets you form a view and the other one only lets you have a reaction.

A grid of ten thousand squares with fourteen highlighted as the reported cases, beside three questions to ask of any count.
The reported cases against the population they came from.

Evidence

Availability, and why vividness beats frequency

Amos Tversky and Daniel Kahneman described the mechanism in 1973 in Cognitive Psychology: people estimate how common something is by how easily examples come to mind. They called it the availability heuristic.

Coverage is exactly what makes examples come to mind easily. So a category of event that is rare and heavily reported will feel common, and a category that is frequent and unreported will feel rare, and the gap has nothing to do with intelligence or attention. It is the predictable output of a memory system meeting an editorial process, and knowing this does not switch it off. It only tells you which question to ask.

Claim

The three questions to put to any count

Out of how many. Compared with when. Who is counted.

The third one catches more than people expect. Definitions change: what counts as an assault, a homeless person, a strike, an unemployed adult, a hospital waiting list. A jump in a series is frequently a change in the counting rule or in reporting behavior rather than in the world. Official statistics almost always document these changes in a methodology note, and the note is almost never mentioned by anyone quoting the series.

Example

Ellie was certain that thefts from cars had exploded in her town, because her local social group had six posts about it in a month.

She found the police open data portal, which published recorded crime by category and ward going back years. Vehicle crime in her ward was up about 9% on the previous year and lower than it had been five years earlier. Six posts in a month was two more than the usual rate and they had all been shared into the same group. She still locks her car and stopped telling people the town was falling apart, which is a smaller and more accurate opinion arrived at in about twenty minutes.

Action

Find one official series for something you worry about

Pick one topic that bothers you: crime in your area, hospital waiting times, school performance, road deaths, the price of something.

Find the official data series for it, from a statistics office, police force, health authority or regulator. Look at ten years, not one. Write down the current value, the value five years ago, and the value ten years ago. Bookmark the page. The next time you see a claim about that topic, you will have the base rate to hand, which is the single most useful preparation available in this course.

Caution

Base rates do not cancel a real event

There is a way of using this day that is genuinely obnoxious: replying to someone whose house was burgled with a statistic about the regional trend.

Aggregate data answers questions about populations. It says nothing about whether the thing that happened to a specific person happened, or how bad it was. The two live at different levels and a man who uses the base rate to dismiss an individual account has misunderstood the tool. Use it on claims about how the world is going. Do not use it on someone telling you what happened to them, and notice that the two are different sentences even when they are about the same subject.

Aside

When the base rate is the wrong instrument

Rare events with very large consequences are exactly where frequency reasoning performs worst. A once in fifty years flood, a bank failure, a pandemic: the base rate is small, the expected cost is enormous, and a man who dismisses preparation because the annual probability is low is making an arithmetic error, not a sober judgment.

There is also a category where the historical rate is not informative at all, because the underlying system has changed. So the rule is narrower than it first looks: base rates are the right tool for judging whether a familiar thing is getting more common, and the wrong tool on their own for judging what to do about a rare catastrophe.

Find the official ten-year data series for one thing you worry about, and write the current, five-year and ten-year values.

Use a statistics office, police force, health authority or regulator, not a news article. Read the methodology note for any change in definitions. Bookmark the page.

25 min

Day 03 — reading your progress.