Questions / Quant research, Research decisions
Choose the success metric first
A model has no real effect. You test 20 independent metrics at 5% each. What is the probability that at least one looks significant?
Answer it in the app It's free. You get hints, a second try if you miss, and a follow-up question.
Show the answer and worked solution
Answer: 0.6415 probability
- Each metric stays non-significant with probability 0.95.
- All 20 stay quiet with probability 0.95²⁰ ≈ 0.358.
- At least one false positive: 1 − 0.358 ≈ 0.642.
Interview shortcut. At least one of 20: 1 − 0.95²⁰ ≈ 64%.
Takeaway. Reporting the best of many metrics rewards noise.
In the app this has a follow-up where one thing about the setup changes. Try the follow-up.