Analytics and experiments

Experimentation interview questions

A/B testing is one of the most common topics in product data science interviews, and the questions tend to start after the test has run. A new feature showed an 8% lift in week one and nothing by week three. What happened, and what would you tell the team?

Interviewers ask how you’d size a test and choose a minimum detectable effect, why checking results every day inflates false positives, what a sample ratio mismatch tells you, and when users influencing each other breaks the test altogether. Bandits and variance reduction with CUPED come up for more senior roles.

Try one now

Standard

A new feature shows an 8% lift in the first week of an A/B test, but the gap shrinks to nothing by week three. What's going on?

Free, no sign-up. The interviewer asks follow-ups on your reasoning, then rates your answer.

What the 7 questions cover

  • A/B testing6
  • Power, sample size and minimum detectable effect1
  • Sample ratio mismatch1
  • Peeking and sequential testing1
  • Variance reduction (CUPED)1
  • Novelty and primacy effects1
  • Interference and network effects1
  • Multi-armed bandits1
  • Reinforcement learning1

A question can cover more than one topic, so the counts add up to more than 7.

How the practice works

  1. The AI interviewer asks the question and follows up on your answer, the way an interviewer would.
  2. Stuck? Ask for a hint. Each one gives a little more away, so you can get unstuck without being handed the answer.
  3. Get a written debrief. Four ratings, each from Needs work to Interview-ready, tied to what you actually said, and what to study next.

More analytics and experiments questions

See all categories