So, Uh, Are We Doing This Right?

Challenge Two:
Designing Meaningful Tests
If you are fortunate enough to have created a single, integrated database, your next challenge is to design meaningful tests. The old, single-channel, single-variable “split tests” may not be sufficient.

For example, if you are wondering if direct-mail copy A is better than direct-mail copy B, you could simply run an A vs. B split test to a random sample of the universe of names being mailed. Assuming you designed the test correctly and only changed one variable at a time (here it would be the copy), you could easily measure which copy “won” or did better according to appropriate measures like percent response, average gift, income per 1,000 letters mailed or long-term value. Thus, judging the results is straightforward.