A/B Testing Implementation

Implement a production-ready A/B testing and experimentation platform with feature flags, variant assignment, conversion tracking, and statistical analysis for the following requirements:

Application Overview

Platform Details

Application Type: [Web app / Mobile app / Both]
Framework: [React / Next.js / Vue / Angular / React Native / Flutter]
Backend: [Node.js / Python / Go / Ruby / Java]
Analytics Platform: [Segment / Mixpanel / Amplitude / PostHog / Custom]
User Base: [1k users / 10k users / 100k+ users]

Experimentation Goals

Primary Use Cases: [Product features / UI/UX / Pricing / Onboarding / Marketing]
Experiment Frequency: [1-2 per month / 5-10 per month / Continuous]
Team Size: [Solo / Small team / Large org]

Experiments to Implement

Experiment 1: [Name, e.g., New Pricing Page]

Experiment ID: [pricing-page-2024 / unique-id]
Hypothesis: [What you expect to improve]
Target Metric: [Conversion rate / Revenue / Engagement / Custom]
Variants:
- Control: [Current experience, 50% traffic]
- Variant A: [New design, 50% traffic]
- Variant B: [Optional third variant, 0% or split]
Traffic Allocation: [50/50 / 33/33/34 / Custom]
Target Audience: [All users / New users / Specific segment]
Duration: [2 weeks / 1 month / Until significant]
Sample Size Needed: [Calculate based on baseline and MDE]

Experiment 2: [Name, e.g., Checkout Flow Redesign]

Experiment ID: [ID]
Hypothesis: [Hypothesis]
Target Metric: [Metric]
Variants: [Define variants and traffic split]
Traffic Allocation: [Split]
Target Audience: [Audience]
Duration: [Duration]
Sample Size Needed: [Size]

Experiment 3: [Name]

Experiment ID: [ID]
Hypothesis: [Hypothesis]
Target Metric: [Metric]
Variants: [Variants]
Traffic Allocation: [Split]
Target Audience: [Audience]
Duration: [Duration]
Sample Size Needed: [Size]

[Define 3-10 experiments]

Feature Flags

Flag 1: [FlagName, e.g., new-dashboard]

Type: [Boolean / Multivariate / Percentage rollout]
Purpose: [Enable new dashboard for testing]
Default Value: [false / control / 0%]
Rollout Strategy: [Gradual / Instant / User-based]
Kill Switch: [Yes / No]

Flag 2: [FlagName, e.g., premium-features]

Type: [Type]
Purpose: [Purpose]
Default Value: [Default]
Rollout Strategy: [Strategy]
Kill Switch: [Yes / No]

[Define 5-15 feature flags]

Variant Assignment

Assignment Strategy

Method: [Deterministic hash / Random / Sticky session]
User Identifier: [User ID / Session ID / Device ID / Cookie]
Consistency: [Same user always gets same variant]
Hash Algorithm: [Custom hash / MD5 / SHA256 / MurmurHash]

Traffic Allocation

Control Group: [Percentage, e.g., 50%]
Variant A: [Percentage, e.g., 50%]
Variant B: [Percentage, e.g., 0% or split]
Holdout Group: [Optional, e.g., 10% see neither]

Targeting Rules

User Attributes:
- Country: [US, UK, etc. / All]
- Device: [Mobile / Desktop / All]
- Plan: [Free / Pro / Enterprise / All]
- Signup date: [Before/after date / All]
- Custom attributes: [List custom targeting]
Exclusions:
- Internal users: [Exclude / Include]
- Bots: [Exclude / Include]
- QA environments: [Exclude / Include]

Conversion Tracking

Primary Metrics

Metric 1: [Name, e.g., Purchase Conversion]

Event Name: [purchase_completed]
Metric Type: [Binary (conversion) / Continuous (revenue)]
Value: [1 for conversion / Dollar amount]
Tracking Location: [Checkout page / API endpoint]
Success Criteria: [> 5% improvement]

Metric 2: [Name, e.g., Signup Rate]

Event Name: [user_signed_up]
Metric Type: [Binary / Continuous]
Value: [Value]
Tracking Location: [Location]
Success Criteria: [Criteria]

[Define 2-5 primary metrics]

Secondary Metrics

Engagement: [Time on page, clicks, scrolls]
Retention: [7-day retention, 30-day retention]
Revenue: [Average order value, lifetime value]
Performance: [Page load time, error rate]

Guardrail Metrics

Error Rate: [Should not increase]
Page Load Time: [Should not degrade]
Bounce Rate: [Should not increase significantly]

Analytics Integration

Event Tracking

Assignment Event:
- Event name: [experiment_assigned]
- Properties: [experimentId, variant, userId, timestamp]
- Trigger: [On variant assignment]
Conversion Event:
- Event name: [experiment_conversion]
- Properties: [experimentId, variant, userId, metric, value, timestamp]
- Trigger: [On conversion action]
Exposure Event:
- Event name: [experiment_exposed]
- Properties: [experimentId, variant, userId, timestamp]
- Trigger: [When user sees the variant]

Analytics Platform

Platform: [Segment / Mixpanel / Amplitude / PostHog / Custom]
API Key: [Environment variable]
Batch Events: [Yes / No]
Retry Logic: [Yes / No]

Statistical Analysis

Sample Size Calculation

Baseline Conversion Rate: [Current rate, e.g., 5%]
Minimum Detectable Effect (MDE): [e.g., 10% relative improvement]
Statistical Power: [80% / 90% / 95%]
Significance Level: [0.05 / 0.01]
Calculated Sample Size: [Per variant]

Statistical Tests

Test Type: [Two-sample t-test / Chi-square / Bayesian]
Confidence Level: [95% / 99%]
Multiple Testing Correction: [Bonferroni / None]

Results Interpretation

P-Value Threshold: [< 0.05 for significance]
Confidence Intervals: [Calculate 95% CI]
Effect Size: [Report absolute and relative lift]
Statistical Power: [Ensure adequate power]

Experiment Management

Experiment Lifecycle

1. Planning

Hypothesis: [Clear hypothesis statement]
Success Criteria: [Define success metrics]
Sample Size: [Calculate required sample]
Duration: [Estimate experiment duration]

2. Setup

Create Experiment: [Configure in system]
Implement Variants: [Code changes for each variant]
Setup Tracking: [Implement event tracking]
QA Testing: [Test all variants]

3. Launch

Gradual Rollout: [Start with 10%, then 50%, then 100%]
Monitor Metrics: [Check for anomalies]
Validate Data: [Ensure events are firing]

4. Analysis

Wait for Significance: [Don't peek too early]
Calculate Results: [Run statistical tests]
Check Guardrails: [Ensure no negative impact]
Make Decision: [Ship winner / Iterate / Kill]

5. Conclusion

Document Results: [Record findings]
Ship Winner: [Deploy winning variant]
Archive Experiment: [Mark as completed]
Share Learnings: [Communicate to team]

Experiment Configuration

Experiment Settings

Status: [Draft / Running / Paused / Completed]
Start Date: [When to start]
End Date: [When to end or auto-stop]
Auto-Stop: [Stop when significant / Manual]
Ramp-Up: [Gradual traffic increase / Instant]

Variant Configuration

{
  "experimentId": "pricing-2024",
  "name": "New Pricing Page",
  "status": "running",
  "variants": [
    {"name": "control", "weight": 50},
    {"name": "variant-a", "weight": 50}
  ],
  "targeting": {
    "countries": ["US", "UK"],
    "devices": ["desktop", "mobile"],
    "userSegment": "all"
  },
  "metrics": {
    "primary": "purchase_conversion",
    "secondary": ["time_on_page", "bounce_rate"]
  }
}

Implementation Requirements

Client-Side (Frontend)

React Hook

useABTest Hook: [Get variant for experiment]
useFeatureFlag Hook: [Check if flag is enabled]
Variant Assignment: [Assign on mount, track assignment]
Exposure Tracking: [Track when variant is shown]
Conversion Tracking: [Track conversion events]

Variant Rendering

Conditional Rendering: [Show different UI based on variant]
Component Variants: [Separate components per variant]
CSS Variants: [Different styles per variant]

Server-Side (Backend)

Variant Assignment API

Endpoint: [GET /api/experiments/:experimentId/variant]
Input: [userId, experimentId, userAttributes]
Output: [variant, experimentId, assigned]
Caching: [Cache assignment for consistency]

Experiment Configuration API

Endpoint: [GET /api/experiments]
Output: [List of active experiments]
Filtering: [By status, date, etc.]

Analytics API

Endpoint: [POST /api/analytics/track]
Input: [event, properties, userId]
Processing: [Send to analytics platform]
Batching: [Batch events for performance]

Database Schema

Experiments Table

id: [Primary key]
experiment_id: [Unique string ID]
name: [Experiment name]
hypothesis: [Hypothesis text]
status: [draft, running, paused, completed]
variants: [JSON array of variants]
traffic_allocation: [JSON object with weights]
targeting_rules: [JSON object with rules]
primary_metric: [Metric name]
start_date: [Timestamp]
end_date: [Timestamp, nullable]
created_at: [Timestamp]
updated_at: [Timestamp]

Assignments Table

id: [Primary key]
user_id: [User identifier]
experiment_id: [Foreign key]
variant: [Assigned variant]
assigned_at: [Timestamp]
user_attributes: [JSON snapshot of user attributes]

Events Table

id: [Primary key]
user_id: [User identifier]
experiment_id: [Experiment ID]
variant: [Variant]
event_type: [assignment, exposure, conversion]
metric_name: [Metric name if conversion]
value: [Metric value if applicable]
timestamp: [Event timestamp]
properties: [JSON additional properties]

Dashboard & Reporting

Experiment Dashboard

Active Experiments: [List running experiments]
Experiment Details: [Hypothesis, variants, metrics]
Real-time Stats: [Sample size, conversions, p-value]
Variant Performance: [Conversion rate per variant]
Statistical Significance: [Is result significant?]
Confidence Intervals: [95% CI for each variant]
Recommendation: [Ship / Iterate / Kill]

Metrics to Display

Sample Size: [Per variant]
Conversion Rate: [Per variant]
Absolute Lift: [Difference from control]
Relative Lift: [Percentage improvement]
P-Value: [Statistical significance]
Confidence Interval: [95% CI]
Statistical Power: [Achieved power]

Code Generation Requirements

Generate a complete A/B testing platform including:

Variant Assignment System:
- Deterministic hash function for consistent assignment
- Traffic allocation logic (50/50, 33/33/34, custom)
- User bucketing algorithm
- Targeting rules engine
- Assignment caching for consistency
Frontend Integration:
- React hooks (useABTest, useFeatureFlag)
- Vue composables (if Vue)
- Variant assignment on mount
- Exposure tracking
- Conversion tracking helpers
- TypeScript types for experiments
Backend API:
- Variant assignment endpoint
- Experiment configuration endpoint
- Analytics tracking endpoint
- Experiment CRUD operations
- Results calculation endpoint
Database Layer:
- Experiments table schema
- Assignments table schema
- Events table schema
- Migrations
- Indexes for performance
Analytics Integration:
- Event tracking functions
- Analytics platform SDK integration
- Batch event processing
- Retry logic for failed events
Statistical Analysis:
- Sample size calculator
- Statistical significance calculator
- Confidence interval calculator
- Bayesian analysis (optional)
- Multiple testing correction
Experiment Dashboard:
- List active experiments
- Experiment detail view
- Real-time metrics
- Statistical results
- Variant comparison charts
- Export results
Feature Flag System:
- Boolean flags
- Percentage rollout
- User-based targeting
- Kill switches
- Flag management UI
Admin Interface:
- Create/edit experiments
- Configure variants and traffic
- Set targeting rules
- Start/pause/stop experiments
- View results and make decisions
Testing & QA:
- Unit tests for assignment logic
- Integration tests for tracking
- Mock analytics for testing
- QA mode to force variants
- Debug logging
Documentation:
- Experiment setup guide
- Integration guide for developers
- Statistical analysis explanation
- Best practices
- Troubleshooting guide

Output production-ready A/B testing infrastructure following best practices with:

Deterministic variant assignment for consistency
Proper statistical analysis with significance testing
Comprehensive event tracking
Real-time experiment monitoring
User targeting and segmentation
Feature flag system for gradual rollouts
Kill switches for emergency rollback
Sample size calculations
Confidence intervals and p-values
Dashboard for experiment management
Clean separation of concerns
Type-safe TypeScript implementation
Scalable architecture for high traffic
Privacy-compliant data collection