Confidence 7/10 is not a 70% chance

$ eldr --explain confidence

Every read carries a number out of ten, and the most common misreading of it is that 7/10 means a 70% chance of a good outcome. It does not mean that, it cannot mean that, and if it did the whole product would be a forecasting tool, which is the one thing it has promised never to be.

Here is what the number measures: how much of the position survives the counter-case. That is it. It is an internal consistency score on your own fourteen answers, not a probability attached to the world.

What moves the number

• Agreement between your stated lean and your revealed behavior. A man who says he is 70% toward leaving and has taken zero externally visible steps in nine months produces a lower confidence number than a man who says he is 55% toward leaving and has already had two conversations. Stated conviction plus avoidance is the loudest contradiction in the input set, and it suppresses confidence hard.
• Whether the decision is one decision. If the sentence you gave contains an "and", confidence drops, because the position is being computed over two things that can resolve in opposite directions. This is the single most common cause of a low number.
• Whether the window is executable. A position you cannot act on inside the window is advice, not an instrument. If your stated move speed is slower than the deadline allows, confidence falls to reflect that the read is describing something you cannot do.
• Whether the counter-case found a load-bearing assumption. When the strongest argument against the position names a fact you do not have and cannot get in time, the number comes down. That is the system marking its own homework.

Notice that none of these are facts about the world. They are all facts about the file. The number is an audit of the argument, and the honest translation of 7/10 is not "70% likely to work" but "this position holds up under three of the four ways I know how to attack it".

The claim

A confidence score that cannot be checked against outcomes is still useful, and a confidence score that is checked against outcomes would be a lie.

The first half: a consistency score tells you where to look. Low confidence is an instruction — split the decision, get the missing fact, shorten the window. It does real work without ever claiming to know the future.

The second half is the uncomfortable one. Suppose we did calibrate the number against logged outcomes, published a chart, and told you that reads at 7/10 came good 68% of the time. It would be the most persuasive thing on the site and it would be close to meaningless. The outcome of your decision is determined overwhelmingly by what you did after the read and by events nobody had access to. Attributing it to a number generated from fourteen answers would be taking credit for the weather. Tetlock spent twenty years demonstrating that confident long-range judgment by experts barely beats chance; we are not going to quietly reintroduce it because a percentage looks good in a screenshot.

How to catch the number lying

One test, and it is cheap. Read the position and the confidence, then cover them and read only the counter-case. Now ask: if the counter-case is correct, how much of the position is left?

If the answer is "almost none" and the confidence said 8/10, the number is wrong and you should discard it. That happens. The failure mode is an intake where a man has written a decisive-sounding decision sentence and then filled the rest of the answers with hedges; the position comes out clean and the counter-case comes out devastating, and the number lands between them looking respectable. When you see that, trust the counter-case. It is the falsifiable half.

The case against the whole number

A ten-point score on a decision is false precision with a smaller font. Nobody can distinguish a 6 from a 7, the distinction has no operational consequence, and putting an integer on a judgment makes it feel measured when it has only been quantified. You would be more honest shipping three bands — weak, moderate, strong — and refusing to go finer.

I think that objection is mostly right, and it is the strongest internal criticism of the instrument I know of. The defense is not that ten points are meaningful. It is that the number is an interface, and the alternative interfaces are worse: a paragraph of hedging gets skimmed, a traffic light gets read as permission, and men who have been sitting on a decision for a year need something that can be compared to itself across two reads a month apart. The integer does that. Six to eight is a real move in the file even if six versus seven is noise.

What I will concede in writing: do not treat one point of difference as information. Treat 3 and 8 as different worlds. Treat 6 and 7 as the same reading. If you find yourself re-running the intake to nudge a 6 into a 7, you are no longer using an instrument, you are shopping for permission — and that is a documented failure mode with its own debrief in this library.

What it is for

The number exists to tell you how hard to push and how soon to review. High confidence, short window, act. Low confidence, the read will usually return SPLIT and the correct move is to go get the one missing fact the counter-case named. Either way the loop closes on a date, which is the only outcome this product is actually trying to produce.