a few thoughts:
1) It seems that while direct customer advertising is plausibly so noisy as to defy evaluation at reasonable cost, it seems to me that most operations of most companies are not in this category. There are either much less noisy measures where impact can be measured directly (e.g. the time or quality of a manufacturing process) or where a logic model can connect an immediate measure to desired outcomes (e.g. what subject words in an email generate the most opens). I suspect the later is what Manzi was referring to when he talked about tens of thousands of experiments.
The former is the basis of the Toyota Way, an approach to firm-level learning an improvement that certainly has an impressive history of results. On a related note, I think Toyota has something to teach others about the application of impact evaluation results in different contexts.
2) It would be interesting to hear how this information is changing your thinking about the Manzi book specifically. Should we be revising our priors down about firm learning in the age of big data?