Matched-reasons forecasting
Controlled language-model evaluation
Does knowing someone’s reasoning make their next decision easier to predict? A controlled experiment comparing personal explanations with matched alternatives.
- Status
- Completed experiment
- Kind
- Data & geometry
- Stack
- Python · LLM evaluation · Bootstrap · Google Colab
BUILD NOTES / Completed experimentMatched-reasons forecasting
Project overview · not an interface capture
Controlled language-model evaluation
Python / LLM evaluation / BootstrapProject overview · not an interface capture
The problem
An explanation can contain useful information without being specific to the person who gave it. I wanted to separate the value of extra context from the value of personal reasons.
What was built
- A matched comparison of choices alone, a participant’s own reasons, and another participant’s matched reasons.
- Evaluation on 720 held-out decisions from 120 participants, with probability scoring and bootstrap uncertainty estimates.
- Checks on the saved forecasts and an audit of whether the observed differences support the original hypothesis.
What I learned
The experiment found no reliable personal-reasons advantage. A well-controlled negative result is useful: it narrows the claim the evidence can support.