The right A/B testing tool is the one your team will actually use every week, on the pages that matter, with numbers you trust. Feature count matters much less than three things: how fast a non-developer can launch a test, whether results are measured on revenue, and whether the script slows your site. Get those right and almost any reasonable tool will pay for itself. Get them wrong and you’ll own expensive software that runs two tests a year.
I’ve sat through plenty of tool evaluations. The pattern is predictable. Someone builds a comparison spreadsheet with 60 rows of features, the team picks the tool with the most checkmarks, and six months later nobody’s logged in since March. So let’s skip the 60 rows and focus on what decides whether a testing program lives or dies.
Why the market got crowded
For years a lot of smaller teams defaulted to Google Optimize because it was free and tied into Google Analytics. Google shut it down, and the official sunset notice confirms that Optimize and Optimize 360 stopped working on September 30, 2023. That pushed thousands of teams to look for replacements at the same time, and the number of A/B testing software options grew fast.
More choice is good, but it makes evaluation harder. Tools now range from developer-first feature flag platforms to Shopify-only apps to enterprise suites with personalization, heatmaps, and surveys bolted on. They’re built for different people, so start with who’ll be using it.
Question one: who runs the tests?
Be honest about this. If your developers are already booked for the next two quarters, a tool that needs code for every variant will sit idle. Marketing-owned programs need a visual editor that handles real pages, including your actual theme, sticky headers, and mobile layouts.
If you do have engineers who want to test pricing logic, search ranking, or checkout behavior on the server, look at tools with server-side SDKs. Plenty of mature teams use both modes: visual tests for content and layout, code-based tests for logic.
Before you sign anything, do a simple trial. Ask the person who’ll own testing day to day to build one variant on your busiest product page without help. Time it. If it takes an afternoon and two support tickets, that’s your answer.
Question two: what does it measure?
A test that lifts clicks on “Add to cart” but lowers completed orders isn’t a win. For ecommerce, you want a tool that tracks orders and revenue natively, then reports revenue per visitor alongside conversion rate. Otherwise you’ll end up optimizing for the wrong thing.
The upside of measuring revenue well can be large. In a Harvard Business Review article, Ron Kohavi and Stefan Thomke describe a small change to how Bing displayed ad headlines. It sat in the backlog for months as a low priority. When someone finally tested it, it raised revenue by 12%, which they estimated at more than $100 million a year in the US alone. Nobody predicted that. The only reason anyone knew was that the test measured revenue directly.
If you’re new to how testing works mechanically, this explainer on what a website A/B testing tool actually does is a good primer before you start demos.
Question three: what does it do to page speed?
Most client-side testing tools add a script to your site. If that script loads slowly or blocks rendering, you get two problems. The page gets slower, and shoppers see “flicker,” where the original content flashes before the variant replaces it. Flicker can bias your results because the variant group sees a jumpier page.
Google’s Core Web Vitals guidance gives you concrete targets: Largest Contentful Paint within 2.5 seconds, Interaction to Next Paint at 200 milliseconds or less, and Cumulative Layout Shift of 0.1 or less. Measure your key pages before installing a tool and again after. If the numbers move the wrong way, ask the vendor how their script loads and whether they offer anti-flicker handling that doesn’t just hide the whole page.
The statistics question nobody asks in demos
Ask every vendor how their stats engine handles peeking. Some tools use fixed-horizon frequentist tests, which assume you won’t stop early. Others use sequential or Bayesian methods meant to stay valid when you check results often. Neither is wrong, but you need to know which one you’re using so you read the dashboard correctly.
Also ask whether the tool checks for sample ratio mismatch, which is when your planned 50/50 split comes out lopsided. It’s a red flag for broken tracking or bot filtering, and a good tool should warn you automatically.
A short evaluation checklist
Here’s the shortlist I’d actually use. Keep it to a page.
- A marketer can build and launch a variant on a real page in under an hour.
- Revenue, orders, and average order value are tracked without custom code.
- The script doesn’t push Core Web Vitals past Google’s thresholds.
- You know how the stats engine treats early peeking.
- It flags sample ratio mismatch.
- Pricing scales sensibly with your traffic, and you understand what counts as a “visitor.”
- You can target by device, traffic source, and new versus returning visitors.
Seven rows. If a tool clears all seven, the remaining differences mostly come down to taste and budget.
Shortlisting without the noise
Once you know your requirements, you can narrow the field quickly. Look at a few category leaders, a few ecommerce-focused tools, and anything your agency or developers already know well. There’s no single best AB testing software for everyone, only the best fit for your team, traffic, and stack. A ranked list of A/B testing platforms can save you a few hours of research, as long as you treat rankings as a starting point and not a verdict. Then run real trials on your own store. Demo environments always look clean. Your theme, your apps, and your mobile traffic are what matter.
A few habits make trials more useful. Agree on one test idea before the trial starts so every tool runs the same experiment. Install each tool on its own, not all at once, because stacked testing scripts slow pages down and can interfere with each other. And bring in whoever handles your analytics early. If a tool’s order counts don’t roughly match what your store platform reports, you want to find that out in week one, not after you’ve signed an annual contract. Small gaps between tools are normal. A gap that keeps growing usually means tracking is broken somewhere.
Where a store-focused tool fits
If you run a D2C brand on Shopify or a similar platform, it often makes sense to start with a tool built around ecommerce rather than a general web suite. CustomFit.ai is one example. It’s a no-code A/B testing, personalization and CRO platform for D2C and Shopify brands, it covers A/B, multivariate, split-URL and price tests, and it offers a 14-day free trial without a card, which is enough time to run one proper test on your own pages before you commit.