My testing philosophy
A/B testing is not a button color exercise. It's a disciplined practice of reducing uncertainty before committing resources. Every test I run starts with a falsifiable hypothesis: 'We believe that [change] will result in [outcome] because [reason]. We'll know this is true when [metric] moves by [amount].'
Test design
Before launching any test I calculate the required sample size based on baseline conversion rate, minimum detectable effect (usually 5-10%), and desired statistical power (80-90%). Underpowered tests are noise - I'd rather wait for significance than ship based on a 60% confidence result.
What I test and what I don't
High-value tests: onboarding flows, pricing page copy, CTA placement, paywall timing, notification copy. Low-value tests: color changes with no behavioral hypothesis, cosmetic UI tweaks. I prioritize tests by expected revenue impact × test velocity × confidence we can ship the winner.
Reading results
I always segment results by user cohort (new vs. returning, mobile vs. desktop, geo). A winning result in aggregate can hide a losing result in a key segment. I also look at secondary metrics - a test that improves conversion but tanks NPS or LTV is not a win.
Real example
At SmartNews, I ran a series of onboarding A/B tests as part of the Ads business program. By restructuring the advertiser onboarding sequence and reducing the time-to-first-campaign from 4 weeks to 10 days, we increased advertiser activation rates significantly - a result that directly contributed to the +$2M/year revenue potential unlocked by the program.