Questions / Quant research, Data & feature research
Test generalization to new entities
You want to predict outcomes for companies unseen during training. Many rows belong to each company. Which split directly tests that target?
Answer it in the app It's free. You get hints, a second try if you miss, and a follow-up question.
Show the answer and worked solution
Answer: B. Hold out complete companies, with any required time separation.
- The target population is companies absent from training.
- Assign every row of a held-out company to the test side, preventing entity overlap.
- Also respect chronology if the intended prediction has a future-time requirement.
Interview shortcut. Split by the unit you will predict for: whole companies.
Takeaway. Choose the split from the deployment population, not from convenience.
Why the other choices fail
- Split each company’s history in half and pool the later halves. A within-company time split evaluates future observations of known entities, not unseen companies.
- Randomize rows within each company, keeping class proportions. Both sets still contain the same companies, so the test does not directly measure performance on unseen entities.
- Split rows randomly and keep the class balance across folds. Both put the same company in train and test.
In the app this has a follow-up where one thing about the setup changes. Try the follow-up.