Pierre Croft
Director of Experience Design at Tribal Worldwide
I wouldn't say that testing big is always the best approach to arrive at the optimal solution. It can be an ideal method if there are obvious problems with the page, or if visitor numbers are low. But it's not a magic bullet. The main reason why this is so is because you lose precision in your understanding. If you do a big test which changes 10 things at once, the page might indeed perform better, but it might be due to only one or two factors. Without a more granular series of tests, you would be unable to isolate what these factors were. Sometimes testing big is better, sometimes it isn't. As with many things the answer is: it depends!
Daniel Lee
Digital Manager: Conversion at Virgin Mobile UAE
I agree with you Pierre, thanks for the comment. From my experience, as you've mentioned, testing only one change on a page enables the AB Tester to attribute any impact in variation performance to the changes made. I shared a posts on my top tips and 'testing small' made the list (http://www.ecomtricks.com/ab-test-tips-9-tips-to-improve-your-tests/). After I shared this post, I got the most questions from the theory of 'testing big', so I decided to cover the topic in more depth. As you said, different strokes for different folks, but these general tips work very well for most testers. Thanks again.
Chris McCormick
Director of UX & Optimisation at iProspect (Dentsu Aegis Network)
I feel the key here is to understand that by doing just radical(strategic) testing or iterative testing you will not be gaining the most benefit from your programme. With iterative changes you are constantly learning but are you necessarily growing?....constant big radical changes could mean your growing but are you ever really learning about what your doing? You need to find the balance between iterative, innovative and strategic/radical testing not just do one or the other. The bigger and more radical tests may take up to 6 weeks to implement due to resource and development time therefore test iteratively in the meantime to gain learnings which will potentially inform that bigger strategic test
Daniel Lee
Digital Manager: Conversion at Virgin Mobile UAE
Thanks for the comment Chris, I agree with you. As mentioned in my article (http://www.ecomtricks.com/ab-test-tips-9-tips-to-improve-your-tests/) testing big works in conjunction with testing small. I've found that testing big first, then taking small iterative steps afterward, is a method to consistently find the most optimised version of a page. Another thing you said which rings true - 'learning about what you're doing' - this has to be the most important factor with AB Testing, so only testing big won't help you achieve the goals set.