If you have used Google's Web Site Optimizer you know what a useful tool it can be. If you have used it a number of times you may found some limitations with testing what you want, in the way you want to test it.
Well, I thought I found a flaw in how the Optimizer works, and Google has just confirmed that I am correct in my thinking. It's a little confusing to explain, so I will really try to make it clear what the problem is with a simple example:
Goal: To find which of two home page versions convert better.
Method: Create two variations of the home page and see what visitors that hit the home page fill out a form and then see a "thank you" page. Conversion tracking code is located on the "thank you" page and there are no direct links to this conversion page on the site.
Test: The test runs and after some period of time enough data is collected and we may or may not have a "winner" depending on how the conversions went.
Traffic sources: Organic traffic and some paid search traffic to the home page and other topical pages.
So, this sounds like just about every test you have run, right? Therein lies the flaw. Did you spot what the problem is?
I didn't either until I ran a test over the past few weeks, and the results for 9 variations were not only unclear, but also kept changing. This is also something I have seen before, but this time it got me thinking about the process and here is what I realized:
Because there is NO CONTROL over where the traffic lands or it's potential path to the conversion page, one cannot be sure that the conversion ( or lack of conversion ) had anything to do with the page being tested.
Visitors may enter the site at the home page, or they may enter someplace else. If they enter someplace else, then see the home page, they will be included in the test data. And if they do hit the home page, then visit another page which causes them to convert, it will be counted toward the home page version, even if it was really another page that resulted in them converting.
So, the answer to this is to set up your experiment so that you control what people are seeing and what options they have for converting. What I plan on doing in the future is setting up the test on a special page of the site that only gets PPC traffic so I know where it is coming from, and also a special conversion page as well. Visitors may really like our test page, and it may be behind a conversion on a later page, but there is no way to know that so the data really is not that useful and should be excluded.
To restate my solution, I will be setting up landing, contact, and conversion pages. If the conversion results from the test page, I will capture that info. If the visitor leaves the landing page without converting, but converts later, it will not be counted in the test. This will result is some test conversions being lost perhaps, but the data will not be inflated when it should not be. Those of you with one-page sites have been and will still be able to run perfectly accurate tests. :-)
