Questions / Quant development, Numerics & testing
A numerically fragile variance formula
Values are all close to a very large common offset. Why can E[X²]−E[X]² be numerically fragile in floating point?
Answer it in the app It's free. You get hints, a second try if you miss, and a follow-up question.
Show the answer and worked solution
Answer: D. Subtracting nearly equal large terms loses significant digits.
- Write values as a large offset plus small deviations. Both E[X²] and E[X]² are then dominated by the square of that offset.
- Their rounded difference may be comparable to the rounding errors rather than the true variance. Centered or stable incremental calculations reduce that subtraction problem.
Interview shortcut. E[X²] − E[X]² cancels catastrophically near a big offset.
Takeaway. Equivalent formulas can have very different numerical stability.
Why the other choices fail
- The large mean makes the mathematical variance itself undefined. Variance can be small and well-defined around a large mean. The issue is floating-point evaluation, not the definition.
- The squares may overflow even for moderate offsets. Overflow needs enormous values; precision is lost long before.
- Squaring cancels the offset exactly. The cancellation is algebraic in exact arithmetic. Rounded estimates of the two large terms need not cancel accurately.
In the app this has a follow-up where one thing about the setup changes. Try the follow-up.