Beyond A/B tests
Experimentation at scale is more than running tests - it's building a system where every team can make evidence-based decisions quickly. That means a shared infrastructure, a common language, and a culture where 'the test lost' is never a failure, it's information.
Framework components
1. Hypothesis backlog: a prioritized list of bets ranked by expected impact and testability. 2. Experiment calendar: visibility on what's running, what's shipping, and what's in analysis. 3. Results repository: every experiment documented with setup, results, learnings, and follow-up actions. 4. Guardrail metrics: metrics that must never move negatively, regardless of test outcome.
Governance
I run a weekly experiment review with PM, data analyst, and engineering leads. New experiments require a one-page spec (hypothesis, metrics, sample size, duration). Results are reviewed at 50% of expected runtime to catch anomalies early and at full runtime for final decisions. No 'peeking' at results mid-test without a pre-registered analysis plan.
Scaling across teams
At SmartNews, I applied PDLC (Product Development Lifecycle) and DACI frameworks across 5 product teams to standardize how experiments were proposed, approved, and analyzed. This reduced the average time from hypothesis to test launch from 3 weeks to 5 days.
Tools and stack
Feature flagging: LaunchDarkly or custom-built flags. Analysis: Mixpanel, Amplitude, or internal BI tools. Documentation: Confluence or Notion experiment log. Statistical methods: frequentist t-tests for most tests, Bayesian methods for sequential testing or early stopping.