Compare your Shopify store before and after a change

To compare your Shopify store before and after a change, mark the observation card done in Store Decode on the day you make the change. Store Decode then compares the same numbers for the 7 days before with the 7 days after, and again with the 14 days after. It is a date split of your own store’s data, not a test: anything else that changed in those weeks moved the numbers too.

Where to find it

1. Make the change in your theme the card points to, such as moving a section or swapping a slide.
2. On the same day, press Mark as done on the card on the Observations page. Explore’s section panel links to the card. The card moves under Done.
3. Wait. After 7 days, and again after 14, a report appears on Home under “Changes you marked done”, once both periods have enough sessions.
4. Press Got it on a report when you have read it. It leaves Home.

On the Observations page, the Done tab keeps every card you marked done in the last 60 days. Each shows its latest report: the number before, the number after, and the days and sessions on each side.

Store Decode also reports changes it finds by itself, with no card. A section that moved or was added counts once the new order has held on most views of the page for three days running. So does a new slide or banner image that has been the most viewed in its section for three days running, while the old one is rarely seen.

What the numbers mean

Each report counts all devices together. The before period is the 7 days ending two days before the change. The day just before is left out: for a change Store Decode finds by itself, that is usually the day the theme was published, with old and new mixed. The after period starts on the day of the change and runs 7 days for the first report and 14 days for the second.

Each report goes down this list and uses the first number that shows a clear difference:

1. Sales after a click per 1,000 sessions of the section, when it had at least 3 orders after a click and 300 sessions of its own on each side.
2. Clicks per 100 views, when the change was a new slide or banner image.
3. Views, when a section moved or changed and had 300 sessions of its own on each side.
4. The page’s add-to-cart rate, for the homepage, a collection page or a product page.

When none shows a clear difference, the report reads About the same on the first number it could compare.

So the report reads Higher after, About the same or Lower after, never “worked” or “failed”. “About the same” means the difference was too small to tell apart from the usual ups and downs. For Views, that is a change of under a tenth of the before value.

Each report on Home is one row: the change, its reading, the number before and after, and its sessions, device and the date you marked it. Under the rows Home prints one caption, for example:

The same numbers, 7 days each side of the date you marked it. Not a test: other things change too.

What else may have changed

A date split cannot separate your change from everything else in the same weeks. Before you read a report, think about:

• a sale, a discount code or free shipping that started or ended;
• a new ad campaign, an email send or a post that brought different visitors;
• a season, a holiday or a payday;
• another change to the same page within about 9 days, which falls inside the same periods;
• an app added or removed, or a theme update.

In the sample store

Tidewell Home is an invented sample store, and every number here is made up. The merchant changed its Why Tidewell section on 12 October, then compared the section’s clicks per 100 views themselves in Explore, on Mobile:

Why Tidewell, Mobile	Days	Mobile homepage sessions	Clicks per 100 views
Before	5 to 11 Oct	4,100	19.4
After	12 to 18 Oct	4,212	24.1

Clicks per 100 views went from 19.4 to 24.1, with 4,100 homepage sessions before and 4,212 after (Mobile · 7 days each side). That is what happened in those two weeks, not what the change did. A newsletter sent on 13 October would show up in the same after period.

This is a comparison by hand, not a Store Decode report. Had the change been marked done on an observation card, the report would have compared it by the rule in What the numbers mean, on all devices.

You could

The option Store Decode writes for a before and after report reads: “You could: review the change in your theme editor and compare again at 14 days.” The 14-day report then gives you a second, longer after period.

If the change was not on a card, compare by hand in Explore. Set the same device and the same number of days on each side of the change, and leave out the day you published it. Explore: Shopify page analytics, section by section shows where to set them.

Common questions

Is a before and after comparison the same as an A/B test?

No. An A/B test shows two versions to visitors at the same time and splits them at random. Store Decode shows one version before and another after, to different visitors in different weeks. That is why the report says it is not a test.

How long should I wait before comparing?

Store Decode waits 7 days for the first report and 14 for the second. After 28 days it stops reporting that change.

Why did no report appear?

Each side needs 300 sessions. A before period under that is never reported, because it cannot grow. A 7-day after period under 300 waits as “collecting” for the 14-day report. A change to a whole page other than the homepage, a collection or a product page has no number to compare, and the Done tab says “No before/after numbers for this change.”

Will the same card come back after I mark it done?

Not for 60 days. Once you mark a card done, Store Decode shows no new card of any kind for that section on Home or Observations for 60 days. The before and after report is the only follow-up, so the app never asks you to undo what it just suggested. Dismiss hides a card for 90 days and Snooze 14 days for 14.

Related

• Observations: what Store Decode noticed in your numbers covers the cards and their buttons.
• How Store Decode chooses an observation has the full rules, the 60-day cooldown among them.

A sample before and after is on the home page.