Skip to content
Specialist advertising for eCommerce brands
07572 224813hello@galaxyadagency.com
Galaxy AdvertisingECOMMERCE GROWTH SPECIALISTS

Service guide / Conversion Optimisation

Would Your Conversion Test Let You Reject the Idea?

Plan the comparison, safeguards and decision rules before changing a page, so a plausible idea gets a fair and useful evaluation.

Three test planning checks cover a challengeable idea, a suitable comparison and agreed decision boundaries.
Editorial diagram based on this article.View full-size diagram

The delivery summary goes live beside the purchase button, orders have a stronger week, and the team declares a win. In that same week, a popular product returns to stock and advertising introduces a new offer.

You might reasonably believe the summary helped. That comparison has not isolated its effect.

A useful test begins before the edit. Write down the explanation, the evidence that could challenge it and the limitations that could prevent a confident conclusion. The value is a better decision, rather than a scientific sounding announcement that makes an attractive idea difficult to question.

Explain what could make the idea wrong

Describe the mechanism you think matters. Suitable customers may hesitate because they cannot establish service contents before adding a product to basket. The delivery summary is meant to resolve that uncertainty earlier.

Then identify contrary evidence. Task reviews could show the information is already easy to find. Enquiries could concern unavailable delivery areas rather than missing details. Those observations should weaken your explanation, rather than somehow becoming further reasons to support it.

Keep the hypothesis precise enough to guide design. “Build trust” could mean almost anything. “Clarify whether the order includes room delivery for this product family” identifies information you can review and a customer question you can observe. It gives the proposed change a specific job.

Do the necessary repair without calling it an experiment

An inaccurate service statement needs correcting because it is inaccurate. A reproducible control failure needs repairing. You do not need to withhold either correction just to create a comparison.

Experiments concern an uncertain preference or causal explanation. Comparing two accurate presentations of the same service information is different from asking whether customers prefer a functioning page to a broken one.

Put that distinction in the work ticket. It protects customers and the analysis. A factual repair can affect commercial results, but that does not mean an aesthetic hypothesis has been proven through a controlled test. Describe what you did accurately before claiming what it established.

Choose a comparison your business can support

Ask the people responsible for experimentation whether a controlled comparison is practical given traffic, order volume and technical setup. Agree the allocation method, reliable measurement and a decision rule suited to the question.

Do not borrow a universal duration or sample size from an unrelated example. Required evidence depends on the outcome, expected differences, variability and the consequences of getting the decision wrong. Discuss those assumptions rather than replacing them with a convenient deadline.

If a controlled test is impractical, use a structured evaluation and state its limits. Task observation, technical verification and a contextual commercial comparison remain useful. They answer a narrower question than an isolated estimate of causal impact, so keep the conclusion within that boundary.

Make sure the comparison still means what you intended

Keep the information accurate and stable between versions. If one changes delivery eligibility as well as placement, you are no longer comparing presentation alone.

Record other relevant changes, including stock, prices, traffic sources and checkout behaviour. Some will be unavoidable. Decide how the team will respond instead of assuming the trading environment will politely stand still for your test.

Verify that the intended versions appear and their important controls work. Technical checking belongs before interpreting customer outcomes. Sophisticated analysis cannot rescue a comparison if the intervention itself was not delivered consistently. Establish what customers actually received before explaining their response to it.

Pick the outcome and the reasons to say no

Choose the principal outcome for its connection to the business question. Completed suitable orders may matter more than opening a panel. Use supporting observations to explain the route, without swapping them in whenever they look more favourable.

Set relevant safeguards. Simplifying a configuration panel should not increase incorrect orders. Giving a promotion more prominence should not hide essential conditions. An interaction increase accompanied by costly downstream problems can be a reason to reject the change.

Agree how to report uncertainty. An inconclusive result does not make the work worthless. It may show that the proposed difference is hard to distinguish under these conditions, pointing towards a narrower question or a better use of development capacity.

Decide when to stop and how to interpret it

Write down when you will review the comparison and what requires an immediate pause. A broken purchase route needs attention regardless of whether the planned evidence threshold has been reached.

Avoid repeatedly rewriting the decision rule after attractive intermediate movements. Have the analyst explain the approach before the team becomes attached to a preferred answer. If a later change is necessary, make the change and its reason visible.

At review, distinguish what happened from why it could have happened. A numerical difference, an implementation fault and a coinciding stock change are separate matters. They deserve separate consideration, even when one confident story would be easier to present.

Keep the version you decided against

Retain the rejected design with the decision record. The next team needs to see what was actually compared, rather than rely on somebody's memory of it.

Otherwise a narrow finding can gradually become an unsupported rule about every similar layout. Keeping the unsuccessful version gives that later discussion something concrete to check.

Leave the next team a defensible decision

Record the chosen action, supporting evidence and unresolved limits. You might retain the change, revise it, remove it or seek a different type of evidence. Explain why the response is proportionate to what you learned.

Preserve the versions and the reasoning. A future designer should understand why the delivery summary exists before removing it as clutter because its original advocates have moved on.

A good conversion test makes it possible to disagree with an appealing idea. When your business can reject its own hypothesis without rewriting the story, testing starts to produce professional judgement instead of a succession of changes that somehow always win.