Updated September 27, 2021
Do you know what’s the ideal price to set your products? Or the color of add to cart button converts best for your audience?
A/B testing gives you factual data that allows you to answer questions like this.
| This guide shows you how to test product prices, theme sections, apps, and any Shopify site feature. |
How to A/B test on Shopify
- Brainstorm ideas
- Choose 1 idea to test
- Run a split test for at least 2 weeks
- Analyze the results and statistical significance
- Keep it running or test the next idea on your list
That’s the summary, read on for the details.
In this guide
- A/B Testing Overview
- A/B Test ideas for Shopify
- Tools and Apps for A/B Testing on Shopify
- How to Run an A/B Test on Shopify
- Summary
A/B Testing overview
What is A/B Testing?
In ecommerce, A/B testing is a method of improving conversion rate by testing 2 variants of an element to see which performs better.
It’s also known as split testing. The idea is to test 2 versions of an element on your website to see which converts better for your particular audience / niche.
A/B testing is the vehicle that delivers data that you can use to make smarter business decisions.
What to expect from A/B testing
You gain knowledge with each test.
Over time A/B testing leads to improvements that contribute to a higher conversion rate, more engagement, and a better user experience.
Some tests will win, some will fail, some will have no significant outcome.
The winners will come, and when they do they can have a huge positive impact.
Example: One time I ran an A/B test on a product’s price to compare $23 vs $28. After 1 month the data showed that sales of the higher priced version were about 20% greater.
The new price became permanent and that 1 change led to an estimated additional $1,200/mo profit. Nice!
I’ve also had plenty of tests that yielded insignificant results, but it’s still knowledge gained… I think of them not as failed tests, but as “now you know” experiments.
If you have a newer site with low traffic and sales I would focus first on building quality traffic + implementing proven conversion rate optimization tactics.
Testing methodology
The 2 most popular methods of split testing are time based and traffic based.
Time based testing means swapping test variants at regular time intervals (like hourly or daily).
Traffic based (or 50/50) testing means swapping test variants for each visitor.
Which is better is a matter of opinion and the application to your scenario.
For Shopify A/B testing, I think time based testing of 24 hours is the best method.
Why? Modern shoppers often view you your site with multiple devices. You can’t cookie users cross-device, so a 50/50 split could show both test variants to the same person in the same day.
If you’re doing a 24 hour daily swap, you ensure that users will have a consistent experience throughout their sessions during that day, regardless of the device accessing your site.
With daily swap testing, you can also handle all the change of variants on the back end, so you don’t risk page flickering or performance lag (more on that later).
A/B Test Ideas for Shopify
How to Generate Ideas
I typically start by asking myself, “What do I think would improve conversions on this site?”.
Some ways to gather ideas:
- Go through the website and brainstorm ideas you think would improve the site (form a hypothesis)
- Start with the idea that you think will have the most impact or most significance
- Look at high traffic pages or poor performing pages for potential big gains
- Check Amazon for ideas – they spend a lot of time and money A/B testing. All their features might not work for your site / audience, but you can get ideas to try.
Example Test Ideas
Here are some ideas to get you started:
- Try a higher product price on one of your products (you might be surprised)
- Color A vs color B on calls to action (add to cart, checkout, etc)
- Product featured image: lifestyle image vs studio image
- Home page first impression at top of page: Product(s) vs Collections vs Collage of customers with product vs Brand messaging vs Video.
- Product layout: Simple layout (price, bullets, add to cart above the fold) vs complex layout (long description, badges, add to cart below the fold)
Tools and Apps for A/B Testing on Shopify
You can A/B test with free tools, it will just take more time and might not be as accurate.
I’ll share the free method + app methods.
A/B Testing the free way
I’ve done this method when I was in a scrappy spreadsheet phase – It works but requires data entry and daily attention.
- Set up a spreadsheet to record your results. Here’s a template.
- Set up your test variants (if you’re testing a theme feature, use the live theme for variant A and create a duplicate theme for variant B)
- Set an alarm to swap your variants every day at the exact same time (I recommend 12:01am store time)
- Record the results from your analytics into your spreadsheet each day (you can do this either daily or at the end of the test)
Shopify Analytics will give different results than Google Analytics. I prefer Shopify Analytics because the sales data is more accurate but either is fine as long as you’re consistent.
Automate A/B Testing with an App
The manual method is kind of a pain but you got plenty of app options to help automate.
There are 2 ways to swap variants: server side and client side.
Server side swapping changes the variant on the back end, before the page loads. This way is best for performance and using the daily swap method.
Client side swapping changes the variant on the front end while the page is loading. This way is best for the 50/50 traffic split method, but can cause page flickering or perceived lag in page load.
The way you can tell server vs client side is this: If the app is loading a script on your page to swap variants, it’s client side. If all the changes are happening on the back end, it’s server side.
What app to use?
I use the Shopify app Theme Scientist.
Why? I’ve been a Shopify developer since 2012 and have used a lot of apps. I never found one that worked well for me, so my team and I built the ideal A/B testing app for Shopify. That’s our app.
Theme Scientist works by swapping your themes or products at regular intervals (30 minutes – 24 hours) and tracks the stats of each version.
But it might not be the best for your situation (for example if you want to do 50/50 split traffic testing) so here are some respectable alternatives:
Google Optimize is a free client side testing tool and integrates with Google Analytics. It can be challenging to set up but here’s a guide to help if you’re not an Analytics pro.
If you’d like to get in contact with a Google Analytics expert that also does CRO and works with Google Optimize, contact us for an introduction.
Optimizely is an enterprise solution. I don’t think it’s a good fit for most Shopify sites but if your store is ballin and you want greater flexibility with your A/B testing, this could be good for you. Prices aren’t listed, so ya you probably need a fat budget.
Convert is a mid range+ level solution, starting around $800/mo but has advanced testing methodologies, a suite of tools, and I’ve heard good things about it.
Other Shopify apps might work for if you’re not into Theme Scientist or those other tools. I’ve used most of the Shopify apps at some point in my career but none of them worked how I wanted, so feel free to browse but I don’t have any to recommend at this point.
Theme Scientist is maintained by PreviewX and our team of Shopify experts, so if you use our app and have feature requests let us know and we’ll do our best to accommodate.
How to Run an A/B Test on Shopify
Ok, now you have the ideas and tools you want to use, it’s time to get a test going. 🤘
How to A/B test product price
If you’re doing the manual swap method, update your product price in the Shopify admin when it’s time to swap.
If you’re using an app, just set your test variants at the start and let it ride.
There’s another way to do it, which can be messy but worth mentioning: You can create a duplicate product and have the 2 products exist at the same time. This can be useful if you’re driving paid traffic directly to a product page and want dynamic testing in real time.
How to A/B test Shopify sections and site features
If doing it manually:
- Duplicate your live theme (variant A) and name your new theme indicating it’s variant B
- Make the changes you want to test on the B theme
- Swap the themes at regular intervals (hourly, daily, etc) by publishing the next theme in your Shopify admin
If using Theme Scientist:
- Duplicate your live theme (variant A) and name your new theme variant B
- Make the changes you want to test on the B theme
- Create the AB test in Theme Scientist and it will automatically swap at 12:01am and record the stats
If using an app for 50/50 client side testing it will depend on their instructions, but typically you’ll use CSS to identify both variants on the page, then insert the app’s script tag to swap the CSS classes to show / hide the variant to the visitor.
Stats: With an app you’ll get the analytics recording automatically. If you’re doing the manual method, use your preferred analytics software and record the stats daily or weekly.
How to A/B test Shopify apps
3rd party Shopify apps that need to interact with your store front will either inject their script into the Shopify header, add it as a theme snippet, or include as an asset.
To A/B test an app you’ll want to either prevent their assets from loading (ideal for performance) or hide them with CSS.
When I test an app I duplicate the theme and hide the assets for that app on the new (B) theme, then run the daily swap method to test.
If you’re using an app with the 50/50 traffic split test method, you can sometimes (depending on the app) identify the app via CSS class or ID, then hide it for one set of visitors.
Timeframe – How long should you test?
With the daily swap method I recommend running a test for 2 weeks minimum, regardless of traffic volume.
This gives each variation a full day of the week.
During your test period, be aware of your marketing promotions. For example if you send an email blast with a discount offer and you’re using the daily swap method, split the email between 2 days so the discount traffic on 1 day doesn’t skew 1 variation.
I try to run 1 test per month and aim for a statistical significance of 95% or higher.
Statistical Significance
Statistical significance is the probability that 1 variant will outperform the other long term.
The higher % of statistical significance you achieve during a test, the less factor random chance has on the results.
You can use this A/B testing calculator to measure statistical significance.
Aim for at least 95% statistical significance for confident results.
Pro tip: If you’re testing a single product, run extra paid traffic directly to that product page to get more data quicker and a better chance at achieving statistical significance.
Keeping records
If you’re using an app, you’ll likely be able to see past results, but either way I like to have an overview spreadsheet to track marketing experiments like this.
Here’s the template I use: Marketing experiments template. Copy that into a new sheet and adjust as necessary.
Summary
A/B testing is a key component of conversion rate optimization. By running experiments you get data on what the audience in your niche respond to best.
1 single A/B test can have a significant long term affect on your store success by raising conversion rate and getting more customers.
You can use Theme Scientist for easy A/B testing using time intervals (30 mins – 24 hours). If you want to do 50/50 split traffic testing or prefer another tool, contact us and we’ll introduce you to a partner that’s expert with Google Analytics and conversion rate optimization.
For more ideas and proven methods of increasing conversion rate, check out this Shopify conversion rate optimization guide.
Enjoy the journey and good luck on leveling up your biz! ✊