What Makes a Good AB Test?
In a perfect world of infinite time and resources, one might be tempted to split test every change implemented into every listing. Unfortunately, this is not only impractical, it might also not be as helpful as it sounds.
Test One Variable at a Time
AB testing is not designed for testing multiple options at once. The more variables in play, the harder it becomes to tie a result to a specific change. That is why split testing is best for “either/or” situations. You might have many ideas to improve a listing, but it’s important to narrow them down to the ideas that have the best chance of doing the most good. If version A represents the listing as it initially appeared and version B has completely different titles, images, and A+ content, then you may have proven the success of one version over the other, but you’ve done very little to understand the root cause of results.
This does not mean that you should run a separate test for each word changed, but that keyword optimization should be a different split test than an updated version of a product video. Amazon recommends testing changes that are significant in magnitude, not quantity. In addition, only one AB test can be run on each product at any given time. If you’re hoping to test multiple factors, each will have to be run consecutively.
Give Tests Enough Time to Gather Statistically Significant Data
Your testing period should range between 4-10 weeks. The longer you’re able to run tests, the more statistically valid they will be. Short tests might sound convenient, but outside forces that occurred during that time threaten the validity of your results.
Jennifer Johnston explains, “Rogue sellers pricing down, buy box issues, lightning deals, and listing suppressions can derail a short test SO quickly. With a longer test you’re able to get additional data to normalize any impact from those potential factors.”
Other factors that impact short tests more than long ones include seasonality (short test during December vs. March could show different results), off-market advertising and promotions. If you are running any advertising on or off Amazon for products being tested, attempt to keep efforts as consistent as possible throughout the testing period to avoid impact.
It is also important to note that experiments cannot be changed once started. Once canceled, you will have to start a new test from scratch and results from each test will not be linked.