A/B testing two very different home pages - Confusing results


#1

Hi all, first message here and need some help understanding a/b test results:
Goal:
Reduce home page bounce rate
Increase number of viewed pages per visitor
Increase home page engagement (only 35% of the users would scroll on the home page)

Our theory about shy this was happening is as follow:
The hero image doesn’t really show what the company sells
The header is too tall, pushing important content below the fold
Content just below the header is not engaging enough

We redesigned the top part of the home page using an image showing our product, reduced its height by 30% and move important content right below it showing the warranties we offer on our product.

After one month of testing, the results were not what we thought at all. We used Google Optimize and after 30 days, the differences between the two versions were very small. In fact the new version of the page performed slightly worse than control.

Here are the two versions:


#2

Hello Gregory,
Welcome!

So what were you specifically testing? The new page performed worse in what way?


#3

Bounce rate higher, revenues were similar, time spent was lower:


#4

Bounce rates don’t mean much in the grand scheme of things. What other details do you have? Any heatmaps or session recordings? If the end goal is unchanged it probably means it just weeded out unqualified users who wouldnt have spent time buying anyway? So does it matter?
Also other factors could be any ad campaigns running - what details do you have in terms of analytics?
I think heatmaps/scroll/session recording playbacks data could help you a little more than just bounce rates IMO.
Hope you find what your looking for.


#5

for me, version one has a more prominent CTA, it is centred and isolated, very obvious to click. A more contrasting colour would make it stand out even more.

The risk-free aspect sounds like a decent enough angle, but it could be emphasised more. Without trying to sound too salesy, perhaps something along the lines of “best nights sleep guaranteed, or your money back. Start your 1 year trial now.”

Maybe a real life user quote will help sell the benefits better? Or has it won any awards? Has it got good ratings? If so display these prominently. Social proof will make the product more intriguing and entice the user to find out more


#6

Thank you @lykc those are great ideas. I’ll test changing the CTA and copy.
@ari_rahmati Thank you for your comment. I don’t have scroll data, GA doesn’t really show that. We’re switching service to VWO which provides heat maps and scroll maps.