Avoiding the Pitfalls
The author lists some of the common mistakes that can harm the validity of a test.
No Control
Failure to have a control against which to test can skew or invalidate your results. You must compare the performance of your test to a baseline that is running at the exact same time. Conversion rates fluctuate over time, so comparing this week to last will produce a variance that is not the subject of the test. Comparing the results of a test to the site's averages over time likewise skews the results for the natural variance that occurs during the test period.
Insufficient Data
Remember that the reliability of statistics depends on large numbers, so conducting a test over a short time with few users will produce results that have a high probability of error.
This is particularly a problem with analytics that can measure actions in a day, hour, or minute and attempt to adjust on the fly. Unless you get thousands of visitors per minute, it's too small a number, and recall that time is a factor that will bring variance (your site likely has different audiences at different times of day).
Also, consider the net result, not just a part of the equation. Click-through on an advertisement seems important, but ultimately, you want to make the sale. If you improve click-through and sales are flat, then you are doing nothing for the business (actually, you're wasting money to attract the wrong audience)
Variable Interactions
There is a tendency for use to measure what is can be measured - but just because something can be measured doesn't mean it's important.
And so, look at covariance to determine when factors are interrelated - and especially when multiple covariance exists, it may be the conflux of elements, rather than any single element, that has the greatest impact.
Seasonality
A test is short-lived, and the results will be impacted by the time period in which the test takes place. There is no such thing as an unchanging visitor population that behaves consistently over time.
Delayed Conversions
There is an assumption that the effects of a change are witnessed immediately - but it's been found that users may see an ad and not visit a site until later, or visit a site and not make a purchase until later. This is especially true when a product is high-involvement, and the user will take some time to make up their mind before purchasing.
With that in mind, the activity on a site should be monitored for a reasonable amount of time after a test to better determine the long-term effects. If possible, use tracking mechanisms (such as cookies) to determine when a test subject returns at a later date.
Search Engine Considerations
A significant portion of marketing effort is focused on getting visitors to come to a site, via promotion to search engines. Search engine optimization (SEO) has become a discipline unto itself, and it's a very tricky business.
One of the main problems is that it is over-emphasized. Drawing a high-visitor count requires tweaking a landing page. Unfortunately, the changes made to increase the likelihood of a person seeing the page in search engine rankings may decrease the likelihood of their doing what you want them to do once they get there.
There is likewise the reluctance to change a page once it has been tweaked so that it has a high search engine ranking. Even though your goal may be to increase revenue on improve the number of leads, you may encounter a reluctance to change a page in ways that might harm its search-engine ranking.
There is also the believe that changing a landing page often will result in its being banned from search engines for "cloaking" (a practice used by unethical advertisers who attempt to misrepresent their site by providing a shifting set of pages to a search engine). Search engines were a bit draconian when this practice was first discovered, but have better tuned their algorithms to help differentiate between a cloaked site and one that is merely evolving.
If you cannot overcome the SEO-based objections, consider testing on a separate page or site for which you will promote separately (a special URL in an advertisement or promotional e-mail). This has the added advantage of enabling you to test off-site, on a "temporary" site, to discover improvements that can be implemented in the "real' site at a later time.
Inaction
Finally, there is the problem of inaction - which his not so much a problem for a test as it is for a testing program. If you test, but do nothing with the results, there is no benefit in having tested at all.