Frontend System Design · Lesson 83 of 86 · 5 min
A/B Test Design and Rollout Strategies
This is how you show you can ship features safely and prove they work, not just build them.
When you're asked to design a new feature in an interview, the implementation is only half the answer. The other half is how you get it to users without breaking things. The senior-level answer involves a controlled process of validation and gradual release. You need to articulate a plan that minimizes risk and measures impact.
This is that plan.
How to Frame an Experiment
Before you write a line of code for a new feature variant, you must define what success looks like. Answering "how would you test this?" in an interview requires three parts: a hypothesis, your metrics, and the target audience.
1. Hypothesis: A formal hypothesis follows a simple structure: "We believe that [the change] will result in [the expected outcome] because [the reason]." For example: "We believe that changing the 'Buy Now' button color to green will increase checkout conversions because it creates a stronger visual call to action."
Stating a clear hypothesis shows you think scientifically, not just shipping features based on feelings.
2. Metrics: This is the most critical part. You need to show you understand the feature's business impact and potential side effects. Break your metrics into three categories.
- Primary Metric: The one number that proves or disproves your hypothesis. For the button color change, it's the click-through rate on that specific button.
- Secondary Metrics: Other positive outcomes you hope to see. This could be overall checkout completion rate or time to purchase. They provide a fuller picture of user behavior.
- Guardrail Metrics: The critical numbers you cannot afford to harm. These are your safety net. Common examples are page load time, JavaScript error rate, or user unsubscribe rates. A 10% lift in clicks isn't worth a 5% drop in overall revenue or a 50ms regression in TTI.
3. Segmentation: Who is this experiment for? All users? Only users in a specific region? Only users on mobile devices? Defining the target audience, or segment, ensures the results are relevant and not diluted by noise from unaffected user groups.
The Mechanics of Variant Assignment
The core of any experiment or rollout is a mechanism to show different UIs to different users. This is usually managed by a feature flagging service, but you need to understand the client-side logic that consumes those flags.
The process is straightforward: take a stable user identifier, combine it with an experiment identifier, and use a deterministic hash to bucket the user into a group. This ensures a user consistently sees the same experience.
// A simple, deterministic hashing function.
// In production, use a mature library like MurmurHash.
function getHashBucket(userId: string, experimentId: string): number {
let hash = 0;
const input = `${userId}:${experimentId}`;
for (let i = 0; i < input.length; i++) {
const char = input.charCodeAt(i);
hash = (hash << 5) - hash + char;
hash |= 0; // Convert to 32bit integer
}
return Math.abs(hash) % 100; // Bucket user into 0-99
}
interface ExperimentConfig {
name: string;
trafficAllocation: number; // % of users in the experiment (e.g., 50%)
variants: {
name: string; // e.g., 'control' or 'treatment'
weight: number; // % of allocated traffic for this variant
}[];
}
function getVariant(userId: string, config: ExperimentConfig): string {
const bucket = getHashBucket(userId, config.name);
// Is the user even in the experiment?
if (bucket >= config.trafficAllocation) {
return 'control'; // Default experience
}
// If they are, which variant do they get?
let cumulativeWeight = 0;
for (const variant of config.variants) {
cumulativeWeight += variant.weight;
if (bucket < (config.trafficAllocation * cumulativeWeight) / 100) {
return variant.name;
}
}
return 'control'; // Fallback
}
This logic decides which variant a user sees. The frontend code then just needs to render the correct component based on the returned string.
The trap: Describing a purely client-side implementation is fine, but you should acknowledge its main weakness: the "flicker" effect. If flags are fetched asynchronously, the user may see the default UI for a moment before it's replaced by the experiment variant. This is a key reason for server-side rendering of experiments.
Choosing a Rollout Strategy
Not every change is a 50/50 A/B test. The strategy you choose depends on the risk and goal of the change.
Phased Rollout: This is your default for shipping new features. You release the feature to a gradually increasing percentage of users: 1%, 10%, 25%, 50%, 100%. At each stage, you monitor your guardrail metrics. If error rates spike at 1%, you turn the flag off, impacting a minimal number of users. This is about risk mitigation.
A/B Test: Use this when you have a specific hypothesis you want to validate, like whether a green button converts better than a blue one. Traffic is typically split 50/50 (or evenly among multiple variants) for a fixed duration to achieve statistical significance. This is about learning.
Canary Release: This term is often used interchangeably with phased rollout, but it's more specific. A canary release typically routes a small percentage of live traffic to a completely separate, new deployment of your infrastructure. It's best for validating high-risk infrastructure changes (e.g., a new service version, a library upgrade) rather than product features.
Dark Launch: You release code to 100% of users, but the feature is invisible. This is for testing the performance and stability of a new backend system or data pipeline under full production load. For example, you could shadow a new search algorithm, sending it the same queries as the old one and logging its performance and results without ever showing them to the user.
When asked "How would you launch X?", your answer should demonstrate risk awareness. Start with: "It depends on the risk of the change."
- For a risky checkout flow rewrite: "I'd use a phased rollout, starting with 1% of users and closely monitoring error rates and conversion funnels before ramping up."
- For a low-risk homepage text change: "A 50/50 A/B test would be appropriate to measure the impact on engagement."
Statistical Traps That Invalidate Your Results
Running an experiment is easy. Running a valid experiment is hard. Mentioning these concepts shows you understand the difference.
The trap of "peeking": Don't stop an experiment as soon as it looks like it's winning. Short-term variance is normal, and results often regress to the mean over time. You must determine the required sample size and run duration before the experiment starts, and then stick to it. Ending early because the numbers look good is a classic way to fool yourself.
The novelty effect: Users often react to change itself, not the quality of the change. A new design might get more clicks simply because it's new. This effect wears off. Run experiments for at least a full week to capture different daily usage patterns and allow the novelty to fade.
A/A tests: How do you know your testing tool is working correctly? By running an A/A test. You deploy the exact same experience to two groups of users. If your primary metric shows any statistically significant difference between them, your randomization or logging is broken. It’s a sanity check for your entire experimentation platform.
Try it
Pick a feature on a website you use daily (e.g., YouTube's comment section, Amazon's product recommendations). Propose a change. Write down a one-sentence hypothesis, then list one primary, one secondary, and two guardrail metrics for your proposed change.
Next: We'll dig into the trade-offs between Client-Side vs Server-Side A/B Testing.
Lock it in — apply this lesson under interview conditions.