A/B Test Design Template

You are an experimentation expert helping me design a rigorous A/B test. I need a complete test plan that will yield valid, actionable results.

Test Context:
- Feature/change being tested: [Describe what you're testing]
- Current state (control): [What users see today]
- Proposed change (treatment): [What the new experience is]
- Why we think this will work: [Hypothesis rationale]
- Primary metric we want to move: [e.g., conversion rate, engagement]

Please create a comprehensive A/B test design:

1. Hypothesis Statement
   Format: "If we [change], then [metric] will [direction] by [expected magnitude] because [reason]"
   - Clear, testable hypothesis
   - Specific expected outcome
   - Rationale based on research/data

2. Test Design
   - Control description (A)
   - Treatment description (B)
   - What exactly differs between variants
   - Any interaction effects to consider

3. Metrics Framework
   Primary Metric:
   - Metric name and definition
   - Current baseline value
   - Minimum detectable effect (MDE)
   - Why this metric matters
   
   Secondary Metrics:
   - Supporting metrics to track
   - Expected direction for each
   
   Guardrail Metrics:
   - Metrics that should NOT decrease
   - Thresholds for concern

4. Audience & Targeting
   - Who should be in the test
   - Exclusion criteria
   - Segment considerations
   - Traffic allocation recommendation

5. Sample Size & Duration
   - Required sample size (with assumptions)
   - Estimated test duration
   - Statistical power (recommend 80%)
   - Significance level (recommend 95%)

6. Success Criteria
   - What constitutes a "win"
   - What would make us NOT ship
   - Decision framework for mixed results

7. Risks & Mitigations
   - Technical risks
   - User experience risks
   - Novelty effect considerations
   - Rollback plan

8. Analysis Plan
   - When to analyze (not before sample size reached)
   - Segments to examine
   - How to handle inconclusive results