Split testing checkouts with Netlify

In eCommerce, every interaction between customer and brand can be quantified.

With experimentation tactics and analytics, businesses gain insight into where optimizations can be made to increase engagement and conversions.

This post focuses on checkout conversion experimentation, what you can A/B test at the checkout, and then show you how to use Netlify’s A/B testing suite for split testing your Commerce.js checkouts.

Defining Split testing

Also known as A/B testing, Split testing is a process of displaying two variants (control (A) and variant (B)) of a page at the same time to different segments of customers. The goal of this experimentation is to statistically quantify the impact the variant (B) has on conversions vs the conversions from the control (A).

What can you A/B test?

Marketing, performance, and growth teams have long used A/B testing to improve conversions aligned with business goals. Small optimizations to design, copy and layout can have a large impact on conversions when identified and executed on correctly.

Common A/B testing initiatives for eCommerce businesses:

• Product display images (moving vs still image)
• Value phrasing (free shipping vs not)
• Product catalogue layout (grid vs flex)
• CTA button color (green vs purple)
• Cart style (dynamic slider vs full page)

It’s important to only make incremental changes when testing your new variant. For example if you want to test a new variant of the product display page, only change one thing so it’s easy to attribute the results to that one change.

Why haven’t you heard about A/B testing checkouts?

Monolithic eCommerce providers do not allow for checkout customization, making A/B testing at the checkout level non-existent. These monolithic eCommerce platforms impose restrictions on checkout customization because they are not API-first, making the process of customizing a checkout incredibly complex. We’ve written previously about the pitfalls of templated checkouts and their assumptions around “one checkout fits all businesses”.

Below are 16 different brands using the exact same checkout to convert customers. How likely is it that the same checkout converts at its highest across all these different brands and businesses?

A/B testing at the checkout

At Commerce.js we make it easy for developers and business to build custom, high converting checkout forms and checkout integrations with our Checkout API.

With complete checkout control, you can design a checkout flow that works for your business, with the freedom to iterate on that design or flow over time. Checkout A/B testing is especially valuable for high volume businesses, where small improvements to checkout conversions result in revenue increases.

Because the checkout is the most important point of conversion, be sure your checkout is doing a good job at converting customers. In order to make improvements to your checkout conversion you need to be able to A) identify areas to optimize, and B) act on the insights gained. An API-first approach to commerce and checkouts means you can do both.

Below are a few ideas you could A/B test at the checkout level to see if they move the dial on conversions.

Vanilla vs branded checkouts

Continue to excite your customers about your products on the checkout page. A/B test the use of your brand colors, product images in the cart, and customer validation.

Single step vs multi step checkouts

Single page checkout forms are a great way to reduce clicks and friction for the customer. A/B test a single page checkout form that collects customer, shipping, billing and payment information on the same page.

Text size and accessibility

Consider A/B testing your checkout with different text sizes for input fields and page contrast. This can make a considerable impact on conversions if your customers find it hard to complete orders due to their eyesight.

Shipping address included

When only digital goods are in the cart, including a Shipping Address section at the checkout is added friction. A/B test what happens to conversions if carts with only digital goods have checkouts with no Shipping Address fields. Keep in mind Shipping Address can be used to combat fraudulent orders so keep an eye on this!

Global vs localized checkouts

One checkout doesn’t fit all, this is especially true for businesses with a global customer base as customer and address ordering can differ. A/B test if localizing your checkout based on the customers IP address increases conversions for your customers in different countries.

Guest vs membership checkouts

Customers are 35% more likely to convert if they don'd need to create an account to make a purchase. Is this true for your business? A/B test to see if a guest checkout converts better than your sign up / members checkout.

Setting up the A/B testing with Netlify

Let me show you how to setup an A/B test on our checkout using Netlify (note this Netlify feature is still in beta). We’ll be using one of our open source demos to do so. For this example, we will be testing if changing the background color of the checkout increases conversions but you could do this for anything on your checkout page. First, I made a copy of the demo store, and created a new branch. I called my new branch “different-checkout” and changed the background of the checkout to purple. We now have two branches: “master” and “different-checkout”.

Next, I added a new project in Netlify for it, and enabled branch deploys. Next, I pushed a commit to the “different-checkout” branch, which triggered a new deployment for that branch. Now that both versions have been deployed, click on “Split Testing” in the navigation bar in Netlify. Here you can control the amount of traffic that will be sent to each version of the website. For the sake of this example we’ve left it at 50/50, and clicked “Start test” to enable the traffic split. Netlify will now automatically split incoming user traffic between the two versions of the checkout.

Measuring the results

To measure the impact of the A/B test, use an analytics tracking plugin (e.g. Google Analytics) to track the user behaviour and sales performance of each of the two versions of your checkout. Use the data from the experiment to determine which of the checkouts results in the most conversions. More information on setting up analytics tracking.

You could also send a small amount of traffic (e.g. 10%) to an alternative site version, if I just wanted to test the waters with a new site design, home page, or a different set of products, etc. You could deploy an entirely different website to the split (as long as they’re in the same Git repository), or even add more branches for traffic splitting in three, four, or more directions.

Using extra fields for A/B test source tracking

In addition to analytics tracking, you can also add a hidden field that will display which checkout the customer used to purchase from, this will then show up in the actual order. Go to Settings > Data in the dashboard and add a “hidden field” in your custom data fields. Either use the ID provided, or get the extra field ID from your checkout token when you generate it. When capturing your order, include the extra field ID and use “A” or “B” as the value:

    commerce.checkout.capture(checkoutToken, {
      // ... capture payload
      extra_fields: {
        extr_Op1YoVpOdlXLv9: 'B',
      },
    }

Now when you view your order in the dashboard, you can see the value for the collected data field:

Testing

To test this, load up your website in an incognito browser session, or a number of different browser windows, and eventually you’ll see each request being split between the two versions.

Our split testing demo store is here

Finishing the experiment

Once your experiment is completed, you can click “Stop test” inside Netlify to disable traffic splitting and have 100% of your traffic directed back at your primary deploy branch.

Trust in your checkout

Don’t assume your checkout is optimized for your business and customers specifically. Use data from your customers to inform where enhancements can be made and ensure your checkout is serving your customers needs.

If you want to increase checkout conversions or increase the total order value per customer, experiment and observe your customer’s behaviour and then make iterative changes to your checkout.

If you want take control of your sales and start optimizing your checkout with A/B testing, make a start today. Need help with a specific commerce solution? Get in touch to discuss your project.