MBA Marketing Experiments Concept Card Template

Marketing Experiments Concept Card: [Campaign, test or course]

A concept card for the marketing experiments a second-year marketing analytics elective expects you to design, read and defend: A/B tests, holdout (conversion-lift) tests, geo experiments, sample size and power, incrementality and incremental return on ad spend (iROAS), sample ratio checks, peeking, variance reduction and the gap between platform-reported and incremental results. It is for MBA students in marketing analytics and digital marketing electives and for managers who sign off test plans and budget moves. It builds on the core card mba/mba-regression-for-managers-card (which ends by recommending a geo-randomised test) and on academic/stats-experiment-design and academic/stats-power-sample-size, and does not re-teach them. Source basis: Fisher (1935), Neyman (1923, translated 1990), Rubin (1974), Armitage, McPherson and Rowe (1969), Abadie, Diamond and Hainmueller (2010), Deng et al. (2013), Lewis and Rao (2015), Johnson, Lewis and Nubbemeyer (2017), Gordon et al. (2019), Johari et al. (2022) and Kohavi, Tang and Xu (2020), written up in original words with fictional Northfield Outdoor Co. data. Every number was computed in Python and is shown with its Excel formula. Replace every [bracketed] value and every row marked Example.

Document control

When to use it

One-sentence rule: run a marketing experiment when a budget or design decision depends on the causal (incremental) effect of an action, and you can control who or where gets the action.

The idea in plain language

Every customer has two potential outcomes: what they would do if they saw the campaign and what they would do if they did not. We only ever see one of the two (Rubin 1974 formalised this; Neyman 1923 used the same idea for field trials). Randomisation solves the problem for groups rather than individuals: if a coin decides who sees the campaign, the two groups are alike in every respect, observed or not, apart from the campaign itself, so the difference in their average outcomes estimates the average causal effect. Fisher (1935) made randomisation the foundation of experimental design for exactly this reason.

Marketing adds three twists to the textbook experiment:

1. Most conversions are not incremental. Loyal customers buy anyway, and ads are targeted at people who were already likely to buy. A platform that reports "conversions after an ad" counts all of them. The experiment counts only the extra ones: incrementality.
2. Effects are small and noisy. A campaign that moves a 0.9 percent purchase rate to 1.05 percent is very profitable at scale, but detecting it needs hundreds of thousands of users. Lewis and Rao (2015) showed how unfavourable this arithmetic is for advertising in general.
3. The unit of randomisation is a business choice. Users (cookies, logged-in IDs), stores, cities, regions or time periods. Coarser units are easier to run (no tracking needed, which matters as user-level tracking declines) but give fewer independent observations.

Three designs cover most marketing questions:

Formulas and notation

Effect on a conversion rate (treatment $T$, control $C$):

$\hat{\Delta} = \hat{p}_T - \hat{p}_C, \qquad SE(\hat{\Delta}) = \sqrt{\frac{\hat{p}_T(1-\hat{p}_T)}{n_T} + \frac{\hat{p}_C(1-\hat{p}_C)}{n_C}}$

Test and interval: $z = \hat{\Delta}/SE(\hat{\Delta})$; 95% CI $\hat{\Delta} \pm 1.96\, SE(\hat{\Delta})$; relative lift $\hat{\Delta}/\hat{p}_C$.

Incremental conversions and money (spend $S$, average order value $v$, gross margin $g$):

$\text{Inc} = \hat{\Delta}\, n_T, \qquad \text{iROAS} = \frac{\text{Inc} \cdot v}{S}, \qquad \text{Profit} = \text{Inc}\cdot v \cdot g - S$

The campaign pays back when $\text{iROAS} > 1/g$. If an incremental conversion is a new customer, replace $v \cdot g$ by the customer's lifetime value (mba/mba-customer-lifetime-value-card).

Intent to treat and exposure. If only a share $e$ of the treatment group was actually reached, the effect per reached user is approximately $\hat{\Delta}/e$ (treatment on the treated), assuming the unreached users were not affected.

Sample size per control user for a two-sided test at level $\alpha$ with power $1-\beta$, allocation $n_T = \kappa\, n_C$:

$n_C = \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \left[\dfrac{p_T(1-p_T)}{\kappa} + p_C(1-p_C)\right]}{(p_T - p_C)^2}$

Minimum detectable effect for a given design: $\text{MDE} \approx (z_{1-\alpha/2} + z_{1-\beta})\, SE$.

Sample ratio mismatch (SRM) check: $\chi^2 = \sum (O - E)^2/E$ over the arms, with $E$ from the planned split.

Variance reduction with a pre-period covariate (CUPED, Deng et al. 2013): with $\theta = \text{Cov}(Y, X)/\text{Var}(X)$,

$Y^{\text{adj}} = Y - \theta\,(X - \bar{X}), \qquad \text{Var}(Y^{\text{adj}}) = \text{Var}(Y)\,(1 - \rho^2)$

Geo difference-in-differences for pair $j$: $\text{DiD}_j = (T_{j,\text{post}} - T_{j,\text{pre}}) - (C_{j,\text{post}} - C_{j,\text{pre}})$.

Assumptions and limits

Limits to state in every readout: experiments measure the effect of this campaign, at this spend, for this audience, in this period. They say little about the next dollar (the marginal return): for that, combine several tests at different spend levels or calibrate a response curve (mba/mba-marketing-mix-modeling-card).

How managers use it in practice

• Incrementality programmes. Large advertisers run holdouts on their biggest channels on a rolling schedule and use the results to discount platform-reported conversions (an "incrementality factor").
• Ghost ads and public service ad controls. In a holdout, the control group sees nothing from the advertiser; designs such as ghost ads (Johnson, Lewis and Nubbemeyer 2017) log the users who would have been shown the ad, so the comparison can be restricted to comparable reached users.
• Geo experiments for channels without user-level tracking (TV, radio, out-of-home, catalogue). Matched markets or synthetic controls (Abadie, Diamond and Hainmueller 2010) create the counterfactual.
• Product and site experimentation at scale: online firms run thousands of controlled experiments a year; Kohavi, Tang and Xu (2020) describe the practices that keep them trustworthy (SRM checks, guardrail metrics, A/A tests).
• Checking observational methods. Gordon et al. (2019) compared large Facebook experiments with observational methods applied to the same campaigns and found the observational estimates often far off; that is why experiments are the reference point in a measurement programme.
• Platform tools. Major ad platforms offer built-in lift studies. As of October 2026 their names, eligibility and minimum budgets change often: check the platform's current documentation and ask how users are randomised and what "conversion" means before you rely on a result.

Worked example A (fully solved): does Northfield's paid social create sales?

Situation (fictional). Northfield Outdoor Co. spends USD 60,000 over four weeks on a spring paid-social campaign. The ad platform reports 2,400 purchases "after an ad", a reported ROAS of 4.4. The CMO asks whether to double the budget. The analytics team had run a holdout: 500,000 eligible users were randomised 80/20 before delivery; the holdout group was blocked from the campaign. Average order value USD 110, gross margin 45%. Purchases are counted for every user in each group, whether or not they saw an ad (intent to treat).

Spreadsheet layout: users in B2:B3, purchases in C2:C3, rates in D2:D3, the difference in E2, its standard error in F2 and incremental purchases in G2.

Step 1: check the split (SRM). Expected 400,000 and 100,000. $\chi^2 = 0.442$, p = 0.506 (=CHISQ.TEST(B2:B3,{400000;100000})). The split matches the plan; analysis can proceed. A p-value below 0.001 here would mean broken assignment and a stop.

Step 2: the lift. $\hat{p}_T$ = 1.0500% (=4198/399812), $\hat{p}_C$ = 0.9003% (=902/100188), $\hat{\Delta}$ = 0.1497 pp. $SE$ = 0.0339 pp (=SQRT(D2*(1-D2)/B2+D3*(1-D3)/B3)), $z$ = 4.413, two-sided p = 0.00001 (=2*(1-NORM.S.DIST(4.413,TRUE))). 95% CI 0.0832 to 0.2162 pp (=E2-NORM.S.INV(0.975)*F2). Relative lift 16.63% (CI 9.2% to 24.0%). The pooled-variance z (4.217) gives the same conclusion.

Step 3: incremental purchases. $\text{Inc} = 0.001497 \times 399{,}812 = 598.5$ (95% CI 332.7 to 864.2). Of the 4,198 purchases in the treatment group, about 3,600 would have happened anyway; only 14.3% are incremental.

Step 4: money.

The incrementality factor is $598.5/2{,}400 = 0.249$: one in four platform-credited purchases is incremental. Break-even needs $\text{iROAS} \geq 1/0.45 = 2.222$, or 1,212.1 incremental purchases. Even the upper end of the CI (1.584) falls short.

Step 5: exposure. Only 55% of the treatment group was actually reached. The lift per reached user is about $0.1497/0.55 = 0.272$ pp. This changes the interpretation (the ad works on people who see it) but not the economics: spend and incremental purchases are both already measured on the whole group.

Step 6: was the test big enough? With these sample sizes the minimum detectable effect at 80% power was about 0.094 pp, a 10.4% relative lift. The observed 16.6% lift was detectable; the power at the observed effect is 0.993.

Step 7: the recommendation. "The campaign is real but unprofitable on first-order margin: each USD 1 brings USD 1.10 of incremental revenue against USD 2.22 needed. Do not double the budget. Two options worth testing: (a) cut spend to the most responsive audiences and re-test, since the average return at a lower budget is likely higher; (b) if most incremental buyers are new customers, value them at lifetime value: the campaign breaks even if an incremental customer is worth at least USD 100.26 of gross profit."

Sample size, power and duration

The chart above shows the core economics of ad testing: halving the effect you want to detect roughly quadruples the sample. For a 0.90% baseline, 5% two-sided test and 80% power:

Excel for the 15% row: =(NORM.S.INV(0.975)+NORM.S.INV(0.8))^2*(0.01035*(1-0.01035)/4+0.009*(1-0.009))/(0.01035-0.009)^2 returns 49,439 control users; multiply by 5 for the total. Power software that uses Cohen's $h$ (arcsine) instead of this unpooled formula gives a slightly different n (academic/stats-power-sample-size). An equal 50/50 split needs only 165,047 users in total (82,524 per arm), because a small control group is the bottleneck. The 80/20 choice trades statistical efficiency for keeping more customers exposed to the campaign during the test.

Duration. Run whole weeks (purchase behaviour differs by weekday) and long enough to include the purchase delay: for a considered purchase such as outdoor gear, count conversions for at least the typical consideration window after the last exposure.

Peeking, many metrics and other ways to fool yourself

Peeking. Checking a fixed-sample test every few days and stopping at the first p < 0.05 inflates the false-positive rate. Armitage, McPherson and Rowe (1969) analysed repeated significance tests on accumulating data. A simulation for this card (400,000 A/A tests, equally spaced looks, no true effect) gives:

Remedies: fix the sample size and look once; use a group-sequential design with adjusted boundaries; or use methods built for continuous monitoring, such as the always-valid p-values and confidence intervals of Johari et al. (2022).

Many metrics and segments. Twenty metrics at 5% produce about one false win by chance. Name one primary metric and a few guardrails (unsubscribes, returns, page speed) in advance; treat segment findings as hypotheses for the next test. Uplift by segment is worth testing explicitly: Ascarza (2018) showed that customers at highest risk of churning are not necessarily the ones most responsive to a retention offer.

Novelty and learning effects. A new design can win in week one because it is new. Plot the daily lift; if it decays, extend the test.

Winner's curse. The variant that won among several is likely to have been lucky; its effect is overstated. Re-test or shrink the estimate before forecasting the roll-out.

Worked example B (half solved): a free-shipping threshold test

Northfield tests lowering the free-shipping threshold on its web shop. Visitors are randomised 50/50 for four weeks.

1. Order rates: control 4.20%, treatment [ ]%. Difference [ ] pp.
2. Standard error of the difference: =SQRT(0.042*(1-0.042)/30000+0.0455*(1-0.0455)/30000) = [ ] pp. $z$ = [ ]. Significant at 5%? [ ]
3. Gross profit per order: control $96 \times 0.45$ = USD 43.20; treatment $91 \times 0.45 - 3.50$ = USD [ ].
4. Gross profit per visitor: control $0.042 \times 43.20$ = USD 1.8144; treatment [ ]. Difference [ ].
5. Break-even order rate for the treatment: =0.042*43.20/37.45 = [ ]%. Did the treatment reach it? [ ]
6. The team wants to re-run the test on revenue per visitor with a pre-period covariate whose correlation with the metric is 0.5. By what percentage does the required sample fall? [ ]
7. Write the one-sentence recommendation. [ ]

Exercise C (unsolved): a regional TV test

Northfield runs extra regional TV for four weeks in one region of each of 10 matched pairs (the treated region chosen by coin flip), spending USD 150k in total. Weekly average sales, USD k (fictional):

1. Compute $\text{DiD}_j$ for each pair (USD k per week).
2. Mean DiD, its standard error and a paired t-test (=T.TEST(treated change range, control change range, 2, 1)).
3. 95% CI for the mean DiD (=T.INV.2T(0.05,9)).
4. Total incremental sales over four weeks across the 10 treated regions, its CI, and the iROAS.
5. At 45% gross margin, does the TV burst pay back? What does the CI say about the risk?
6. Name two threats specific to geo tests that could bias this result, and how you would check them.

Test plan (pre-registration)

Connections to other cards

• Regression core: mba/mba-regression-for-managers-card shows why the observational ad coefficient was untrustworthy; this card is the remedy it recommends.
• Marketing-mix core and MMM: mba/mba-marketing-mix-card (response curves, allocation) and mba/mba-marketing-mix-modeling-card (experiments calibrate the model's channel effects).
• Attribution: mba/mba-digital-attribution-card. Attribution divides credit; experiments measure the increment.
• Customer value: mba/mba-customer-lifetime-value-card turns incremental new customers into money.
• Course map: mba/mba-marketing-analytics-guide and mba/mba-decision-analytics-guide.
• Statistics library: academic/stats-experiment-design, academic/stats-power-sample-size, academic/stats-proportion-tests and academic/stats-hypothesis-testing for derivations.

Common mistakes and misconceptions

1. Reading platform-reported conversions as incremental. In example A the platform credited 2,400 purchases; the increment was about 600.
2. Comparing exposed users with unexposed users instead of randomised groups. Exposed users are selected by the delivery algorithm and are more likely to buy anyway.
3. Judging on revenue lift without profit. Example B wins on orders and loses on gross profit per visitor.
4. No power calculation. An under-powered test that finds "no significant effect" is often read as "no effect".
5. Peeking and stopping early at the first significant result (14.3% false positives with five looks).
6. Ignoring a sample ratio mismatch. A broken split invalidates every other number.
7. Choosing the metric after seeing results, or reporting the best of many segments.
8. Using a 50/50 split by habit or a 95/5 split to "protect revenue" without checking what it does to power.
9. Forgetting the conversion delay. Ending counting on the last campaign day misses late purchases.
10. Extrapolating the average return to the next dollar. A test at USD 60k says little about USD 120k.
11. Spill-over in geo tests: viewers in a control region who see treated-region TV or shop across the border.

Self-test questions

1. In one sentence, what does the control group in a holdout represent?
2. Why can the platform-reported ROAS be 4.4 when the iROAS is 1.1?
3. Write the standard error of a difference in two proportions.
4. What iROAS does a campaign need to break even at a 45% gross margin?
5. Only 55% of the treatment group saw an ad. What is the effect per reached user if the intent-to-treat lift is 0.15 pp?
6. Why does an 80/20 split need more users than a 50/50 split for the same power?
7. What is a sample ratio mismatch, and what should you do when you find one?
8. You check a test daily for ten days and stop at the first p < 0.05. Roughly what is your false-positive rate?
9. CUPED uses a pre-period covariate with correlation 0.7. By how much does the variance fall?
10. Give two reasons why a geo experiment might be preferred to a user-level test.
11. Why can a statistically significant A/B test result be a bad business decision?
12. What does the winner's curse imply for forecasting the roll-out of a winning variant?

My notes and next steps

Sources and further reading

• Fisher, R. A. (1935). The Design of Experiments. Edinburgh: Oliver and Boyd.
• Splawa-Neyman, J. (1990). On the application of probability theory to agricultural experiments. Essay on principles. Section 9 (translated from the 1923 Polish original). Statistical Science, 5(4), 465-472.
• Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688-701.
• Armitage, P., McPherson, C. K., and Rowe, B. C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society, Series A, 132(2), 235-244.
• Abadie, A., Diamond, A., and Hainmueller, J. (2010). Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493-505.
• Deng, A., Xu, Y., Kohavi, R., and Walker, T. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (WSDM), 123-132.
• Lewis, R. A., and Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. Quarterly Journal of Economics, 130(4), 1941-1973.
• Johnson, G. A., Lewis, R. A., and Nubbemeyer, E. I. (2017). Ghost ads: improving the economics of measuring online ad effectiveness. Journal of Marketing Research, 54(6), 867-884.
• Ascarza, E. (2018). Retention futility: targeting high-risk customers might be ineffective. Journal of Marketing Research, 55(1), 80-98.
• Gordon, B. R., Zettelmeyer, F., Bhargava, N., and Chapsky, D. (2019). A comparison of approaches to advertising measurement: evidence from big field experiments at Facebook. Marketing Science, 38(2), 193-225.
• Johari, R., Koomen, P., Pekelis, L., and Walsh, D. (2022). Always valid inference: continuous monitoring of A/B tests. Operations Research, 70(3), 1806-1821.
• Kohavi, R., Tang, D., and Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
• Textbooks (further reading, by name only): Gerber and Green, Field Experiments: Design, Analysis, and Interpretation; Angrist and Pischke, Mostly Harmless Econometrics; Imbens and Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences.
• Open resources: OpenStax, Introductory Statistics 2e (CC BY-NC-SA 4.0), chapter on hypothesis testing with two samples; Microsoft support pages for NORM.S.DIST, NORM.S.INV, CHISQ.TEST and T.TEST.

Answer key

Worked example B. (1) 4.55%; 0.350 pp. (2) SE 0.1670 pp; $z$ = 2.096 (p = 0.036); yes, significant, 95% CI 0.023 to 0.677 pp, relative lift 8.33%. (3) USD 37.45. (4) Treatment $0.0455 \times 37.45$ = USD 1.7040; difference minus USD 0.1104 per visitor. (5) 4.845%; no, 4.55% is below break-even. (6) Variance falls by $\rho^2$ = 25%, so the required sample falls by 25% (the standard error falls by 13.4%). (7) "The lower threshold raises orders by about 8%, but smaller baskets and shipping costs make each visitor less profitable; keep the current threshold or test a threshold between the two."

Exercise C (parts 1 to 5). (1) DiD by pair: 14, 3, 17, -3, 11, 6, 15, 2, 19, 9. (2) Mean 9.30, SD 7.196, SE 2.276, $t$ = 4.087 with 9 df, p = 0.0027. (3) 4.15 to 14.45 (critical value 2.2622). (4) $9.30 \times 10 \times 4$ = USD 372k (CI USD 166.1k to 577.9k); iROAS = 372/150 = 2.48 (CI 1.11 to 3.85). The ratio method (scale the control test-period sales by the pre-period ratio 1.0108) gives USD 371.1k, almost the same. (5) Gross profit $0.45 \times 372$ = USD 167.4k against USD 150k spend: a small gain of USD 17.4k, but the CI for iROAS spans the break-even 2.22 widely, so the result is consistent with a loss; extend or replicate before scaling. Part 6 is for your own work.

Self-test.

1. What the treated users would have done without the campaign: the counterfactual.
2. The platform credits every purchase that follows an ad (2,400), about three in four of which (all but about 600) would have happened anyway; iROAS counts only the increment.
3. $\sqrt{p_T(1-p_T)/n_T + p_C(1-p_C)/n_C}$.
4. $1/0.45 = 2.22$.
5. About $0.15/0.55 = 0.27$ pp, assuming unreached users were unaffected.
6. The small control group dominates the variance; precision is lost faster on the small side than it is gained on the large side.
7. Observed group sizes that differ from the planned split by more than chance; stop, find the assignment or logging bug, and do not interpret the results.
8. About 19% (simulation for this card: 19.2% with ten looks).
9. By $0.7^2 = 49\%$; the variance is 51% of the original.
10. No user-level tracking needed (works for TV, radio, catalogue and under privacy limits); captures offline sales and spill-overs within a region.
11. The lift may be too small to cover costs, or it may come with lower order values or higher costs, as in example B.
12. The observed effect is biased upwards; forecast with a shrunken estimate or confirm with a new test.

Revision history

Standard and licence note

Based on the experimental design and causal inference literature (Fisher 1935; Neyman 1923, translated 1990; Rubin 1974; Armitage, McPherson and Rowe 1969; Abadie, Diamond and Hainmueller 2010; Deng et al. 2013; Lewis and Rao 2015; Johnson, Lewis and Nubbemeyer 2017; Gordon et al. 2019; Johari et al. 2022; Kohavi, Tang and Xu 2020). The text, data and exercises are original and fictional; no textbook, published case or course material was copied or paraphrased. This template is not affiliated with or endorsed by any business school, publisher, author, ad platform or software vendor. Check the current documentation of your ad platform and software before you rely on a specific tool or output. Template licence: MIT.