Experimentation interview questions
A/B testing is one of the most common topics in product data science interviews, and the questions tend to start after the test has run. A new feature showed an 8% lift in week one and nothing by week three. What happened, and what would you tell the team?
Interviewers ask how you’d size a test and choose a minimum detectable effect, why checking results every day inflates false positives, what a sample ratio mismatch tells you, and when users influencing each other breaks the test altogether. Bandits and variance reduction with CUPED come up for more senior roles.
Try one now
A new feature shows an 8% lift in the first week of an A/B test, but the gap shrinks to nothing by week three. What's going on?
Free, no sign-up. The interviewer asks follow-ups on your reasoning, then rates your answer.
What the 7 questions cover
- A/B testing6
- Power, sample size and minimum detectable effect1
- Sample ratio mismatch1
- Peeking and sequential testing1
- Variance reduction (CUPED)1
- Novelty and primacy effects1
- Interference and network effects1
- Multi-armed bandits1
- Reinforcement learning1
A question can cover more than one topic, so the counts add up to more than 7.
How the practice works
- The AI interviewer asks the question and follows up on your answer, the way an interviewer would.
- Stuck? Ask for a hint. Each one gives a little more away, so you can get unstuck without being handed the answer.
- Get a written debrief. Four ratings, each from Needs work to Interview-ready, tied to what you actually said, and what to study next.