A/B Test Design Template
Design rigorous A/B tests with clear hypotheses, metrics, and success criteria.
Prompt
You are an experimentation expert helping me design a rigorous A/B test. I need a complete test plan that will yield valid, actionable results.
Test Context:
- Feature/change being tested: [Describe what you're testing]
- Current state (control): [What users see today]
- Proposed change (treatment): [What the new experience is]
- Why we think this will work: [Hypothesis rationale]
- Primary metric we want to move: [e.g., conversion rate, engagement]
Please create a comprehensive A/B test design:
1. Hypothesis Statement
Format: "If we [change], then [metric] will [direction] by [expected magnitude] because [reason]"
- Clear, testable hypothesis
- Specific expected outcome
- Rationale based on research/data
2. Test Design
- Control description (A)
- Treatment description (B)
- What exactly differs between variants
- Any interaction effects to consider
3. Metrics Framework
Primary Metric:
- Metric name and definition
- Current baseline value
- Minimum detectable effect (MDE)
- Why this metric matters
Secondary Metrics:
- Supporting metrics to track
- Expected direction for each
Guardrail Metrics:
- Metrics that should NOT decrease
- Thresholds for concern
4. Audience & Targeting
- Who should be in the test
- Exclusion criteria
- Segment considerations
- Traffic allocation recommendation
5. Sample Size & Duration
- Required sample size (with assumptions)
- Estimated test duration
- Statistical power (recommend 80%)
- Significance level (recommend 95%)
6. Success Criteria
- What constitutes a "win"
- What would make us NOT ship
- Decision framework for mixed results
7. Risks & Mitigations
- Technical risks
- User experience risks
- Novelty effect considerations
- Rollback plan
8. Analysis Plan
- When to analyze (not before sample size reached)
- Segments to examine
- How to handle inconclusive resultsHow to use
- 1Describe the change you want to test and the control state
- 2Identify the primary metric you want to impact
- 3Provide any baseline data you have (current conversion rates, etc.)
- 4Replace placeholders with your specific context
- 5Review the hypothesis - is it specific and testable?
- 6Validate sample size calculation with your data team
- 7Get alignment on success criteria before launching
Pro Tips
- • One hypothesis per test - don't test multiple things at once
- • Define primary metric upfront and don't change it
- • Include guardrail metrics to catch negative effects
- • Don't peek at results before reaching sample size
- • Plan for at least 1 full week to capture weekly patterns
- • Document everything for future reference
- • Statistically significant ≠ practically significant
