Most growth plans are written as forecasts. The quarter has a revenue target, the target has a traffic assumption behind it, and the traffic has a conversion rate taken from last year’s average. Everyone signs off, and the assumptions travel untouched until the quarter ends and the target is either met or explained away.
An experiment layer is the part of the plan that tests those assumptions while there is still time to act on them. For an ecommerce business, it is also the cheapest available growth, because it works on visitors who have already arrived. This article covers what conversion rate optimization actually is, how A/B testing produces evidence rather than opinion, what a Shopify store can and cannot test with the platform’s own tooling, and how to fold all of it into a planning cycle that is already full.
What CRO means in business terms
Conversion rate optimization (CRO) is the practice of improving the share of visitors who complete the action you care about. The action can be a purchase, an add to cart, a checkout start, or a custom event such as a newsletter signup. Stated that way it sounds like a web design task. In business terms, it is a revenue question: what does an average visit earn, and can that number be raised without buying more visits?
That reframing changes the metric. Conversion rate alone answers half the question, because it ignores what buyers spend. A discount raises the conversion rate and shrinks the basket at the same time. A premium bundle does the reverse, turning some buyers away while raising the value of those who stay. Judged on conversion rate alone, one of those two changes always looks like a win. Judged on revenue per visitor, which is total net revenue divided by unique visitors, both are priced at their true value.
What A/B testing is, and what it is not
An A/B test is a controlled comparison. The version already live, the control, is shown to one group of visitors. An alternative, the variation, is shown to another. Both groups are assigned at random and held to their version on later visits, so the only difference between the two sets of numbers is the change you made. The result is read on one metric the team chose before the experiment started.
Two things it is not:
· It is not a before-and-after comparison. Changing a page in March and reading April’s conversion rate measures the market, not the change. Ad spend moves, seasonality shifts, competitors run promotions, one product goes out of stock and changes what people buy instead. A test runs both versions at the same time so those forces hit both groups equally.
· It is not a collection of best practices. Lifting a design pattern from a case study tells you what worked for a different store with different traffic, a different price point and a different audience. It is a hypothesis, not a result.
Plan for a portfolio, not a single bet
Experiments are cheap individually and unreliable individually, which is exactly why they belong in a portfolio. In a meta-analysis of 1,001 A/B tests, 33.5% produced a statistically significant positive result, with a mean lift of 15.9% among the winners and a median of 7.5% (Analytics-Toolkit). Put simply: roughly two out of three experiments return no significant winner, and that is the expected cost of finding the one that returns 10% or more.
An inconclusive result is still an answer. It tells you the change was not worth more than the noise on your traffic, which stops the same idea being re-argued next quarter. What kills a programme is not losing tests, it is treating a leading variation as a winner after four hundred visitors and shipping it.
Four families of ecommerce experiments
For a store, the testable variables cluster into four groups:
1. Offer and price. Price points, bundles, subscription terms, discount depth.
2. Merchandising and content. Product page templates, collection layouts, landing pages, product images, titles and copy.
3. Fulfillment terms. Shipping thresholds, flat rates, delivery promises, free-shipping cut-offs measured against margin.
4 Checkout and post-purchase. Trust signals, upsells, cross-sells, payment options, account creation.
The list is not a priority order. The right first test is the one attached to the largest unresolved assumption in the plan, tempered by how much traffic that page receives. A test needs enough visitors to finish; a hypothesis about a page nobody visits is a project, not an experiment.
What a Shopify store can test natively, and what it cannot
Shopify has shipped its own testing feature, and it changes where the tooling line sits. Under Markets > Rollouts the platform now offers three rollout types, one of which, an Experiment, splits eligible visitors between a control and a treatment. The constraints are documented plainly: rollouts are available on the Basic plan or higher, but experiments require the Grow plan or higher; a rollout can change your online store theme, your checkout and accounts configuration, or product catalogs; Liquid template changes are not supported inside a rollout, and vintage themes cannot be tested at all (Shopify Help Center).
That is a reasonable fit for two decisions — a theme rebuild and a checkout configuration change — and it adds no monthly cost for stores already on Grow. Everything else still sits outside it. A price is not a theme setting, nor is a shipping rate, a product image, a single button, or a defined audience segment. Those need a testing tool that changes what a visitor sees at the element level, which on Shopify means an app.
The practical planning rule: name the experiment first, then choose the mechanism. A theme-level decision can ride the native path. A price, shipping or element-level question cannot.
Decide the rules before you see the result
Three parameters have to be fixed in advance, or the result will be read to fit whatever the room already believes:
· Sample size. The minimum number of visitors, and conversions, each variation has to collect before the difference is worth believing. Small samples swing on one large order.
· Significance threshold. The confidence a result must reach before it counts as conclusive, set by the team rather than discovered in the dashboard.
· Decision rules. What result means roll out, what means revert, and what means the test was too weak to settle it.
Add a fourth, non-statistical rule: a freeze on other changes to the pages in the test while it runs. Change two things at once and the result stops being attributable.
Choosing the tool that runs the experiment
The tool decision is secondary to the experiment decision, but it determines whether a programme survives contact with a real store. The failure points are mechanical rather than conceptual: keeping one visitor assigned to the same version across sessions, applying a test price consistently from the product page to the cart, the checkout, the order confirmation and the analytics, and reporting a lift with the confidence behind it instead of a raw percentage.
For Shopify stores, Elevate A/B Testing runs price, page, shipping, image, split URL and checkout experiments, with plans from $49 a month, a free trial and a 4.9 rating from 144 reviews on the Shopify App Store (read 2 October 2026). What matters when comparing tools is not the feature list but the denominator: what each plan charges on, which test types sit behind which tier, and whether price testing — the experiment most stores want first — is included or billed as an upgrade.
Where the experiment layer sits in the plan
Four habits keep experimentation from becoming a side project that dies in month two:
· One hypothesis backlog. Every unresolved assumption in the strategy goes in as a testable sentence with a metric attached.
· A fixed cadence. One test finishing per fortnight is a working pace for most small and mid-sized stores.
· A written log. What was tested, on which metric, the result, and the decision taken. A log is what stops the same idea being retested every year.
· One owner. An experiment program without an owner becomes a queue of ideas nobody closes.
The point of the layer is not to make the plan more scientific for its own sake. It is to convert opinion into evidence at a cost the business can afford, so that the next planning cycle starts from what is known rather than from what was assumed.
Key takeaways
· A plan is a set of assumptions with dates attached; an experiment layer tests them before the quarter ends.
· Judge outcomes on revenue per visitor, not conversion rate alone.
· Run both versions at the same time on randomly assigned visitors, never as a before-and-after comparison.
· Plan for roughly one significant winner in three tests, and treat inconclusive results as answers.
· On Shopify, native Rollouts covers themes and checkout configurations on Grow and above; price, shipping and element-level tests need an app.
· Fix the sample size, the significance threshold and the decision rules before the test starts.
