Measuring Return On Investment In Online Adveritising

Measuring Return On Investment In Online Adveritising

Using 25 online field experiments, representing $2.8 million in ad spend, Randall A. Lewis (economic research scientist, Google) and Justin M. Rao (economic researcher, Microsoft) showed that you cannot measure the causal impact of choice variables on profit even given access to reliable signals.


In their own words:

"We find that even when ad delivery and consumer purchases can be measured at the individual level, linked across purchasing domains, and randomized to ensure exogenous exposure, forming reliable estimates on the returns to advertising is exceedingly difficult, even with millions of observations. As an advertiser, the data are stacked against you."

There's just too much noise in the system. To arrive at statistical significance, you'd need to run so many ads that marketplace conditions will have changed between start and finish. You'd need millions of person-weeks of trials to "reliably distinguish disparate hypotheses such as 'the ad had no effect' (-100% ROI from 'the ad was profitable for the firm' (ROI>0%)."