AB-testing new Shopify theme before full launch

The benefit of a lean, nimble, low-code approach to more rapid design and development on Shopify is that you can move faster and test things with less overall investment.

The potential downside is that you might not go as deep on the research, strategy, and design leading into a major change.

The best way to validate any change is to test.

I have been working on a major theme rebuild for a Shopify store that hadn’t had a major upgrade in 7+ years. It gets a lot of traffic so change is a little scary.

We decided to AB test the new theme using Intelligems—driving 50% of traffic to the old theme and 50% to the new theme.

After a couple of weeks, it has highlighted some interesting things about the new theme. On the positive, we have a 23% drop in bounce rates (with just half of the traffic on the new theme), with visitors spending minutes rather than seconds on the site!

On the downside, while the conversion rate is higher overall with the new theme, customers are buying smaller-ticket items and only buying larger-ticket items on the old theme. 

By split-testing the themes, we are able to see these nuances and tweak and continue to test accordingly.

In this case, we made the decision to pause the test while we address some of the missing features in the new theme. As you can see, the bounce rate jumps back up once the test has been ended. Fascinating.