Q. How long does a test take?
A. A test will run until it reaches statistical confidence. In some cases it is never reached and the test is stopped early. A test should run for at least 7 days. Most tests run for 4-6 weeks before they conclude. There are often a series of tests over time. Although one test can be done, it is more common to run tests for 3 to 6 months. Many companies never stop testing each section of the conversion funnel (home page, category page, product, page, checkout process, landing pages, emails).
Q. How much traffic do you need to do a test?
A. There are a number of factors that determine what will cause a test to conclude. In general you will need at least 500 unique visitors to a particular page per month to test, the more the better. The maximum number of users under test is 10k per month for this plan.
Q. Do we need to suggest ideas to A/B test?
A. We will take your ideas plus our own ideas from years of testing experience.
Q. Do we need to do any creative or coding for each test?
A. In most cases, we can make the changes through the testing system and you will not have to do any creative or coding until there is a winning result. A piece of code will have to be added to the site for testing. Once a new design is proven to create a conversion lift, then you will have to make the changes to the site or page.
Q. How many tests are included?
A. One page is tested at one time. Multiple elements on the page will be tested. If there is enough traffic, multiple variations will be tested at once. The fee is a retainer charged every month no matter how many tests are run.
Q. Is there a guarantee of results?
A. No, it is impossible to guarantee results because there are too many variables. The test result is statistically accurate for that time period and traffic. For example, a test of a consumer product page during December will have a different conversion rate than in August. With that being said, the amount of lift usually changes over time, not the fact that it is still a positive lift in conversion rate.
Q. What is the deliverable?
A. A substantial four page report will be provided at the end of the test. The report has the hypothesis, screen shots of the variations, the data, and recommendations for the web designer to make to the site or landing page.
Q. After activation what are the next steps?
A. Once the product is activated, we will set up a meeting with you within 3-5 business days to start the project.
