SEO A/B Testing at Hunkemöller results in substantial business results

A/B testing is a well-known term for customer experience optimization (CXO) or ad optimization. By setting up a completely new test method in Hunkemöller’s SEO channel, we have proven that SEO A/B testing not only works well, but also produces substantial results.

Why test SEO A/B?

Search engines are an important source of traffic for almost all companies and organizations. On the one hand, traffic and therefore also search behavior is constantly changing. On the other hand, all changes to the website also have an impact. In order to anticipate these movements, a data-driven SEO strategy is important. With a data-driven SEO strategy, you can be confident that all changes that are implemented will also yield positive results. For an organization like Hunkemöller it is essential to make the right decision because of the competitive market and the entry of other parties. Data-driven SEO as discussed later in this article is an important key to future business success. The methodology of data-driven SEO answers the question:

“What will it bring us?”

Not only in organic traffic and visibility, but also in business metrics such as turnover.

Until recently, it was difficult or actually impossible to directly attribute the impact of a specific change to the organic traffic as an uplift. With our methodology for SEO A/B testing, we can say with statistical certainty whether the change has a positive impact and what this change will bring. So that decisions to implement changes site-wide are finally properly substantiated. This methodology makes it possible to test changes at a detailed level, in order to find the perfect mix in reaching the target group via the organic channel. In addition, SEO A/B testing can prevent you as an organization from wasting time and money on changes that ultimately do not contribute. Or what is the result of a delayed change.

How does SEO A/B testing work?

As said. With an SEO A/B test you can validate almost all SEO changes and optimizations before they are implemented site-wide. When we say almost every SEO optimization: we think for example of:

• Optimizing content elements such as: headings
• Optimizing metadata such as: page titles or meta descriptions
• Adding content
• Adding structured data
• The impact of client-side rendered vs. server side rendered content

The impact of Web Vitals optimization (site speed) A/B testing for SEO functions differently than CXO A/B testing. We do present two different versions of our content to users. But in an SEO A/B test, the control group and variant are divided within a group of pages that use the same template. That is different from CXO where one page with multiple variants (a control and at least one variant) is tested.

SEO A/B testing is page oriented instead of audience oriented, in the sentence that we test on a section of the site by splitting the pages (for each visitor, including Googlebot) instead of splitting the audience.

The pages are optimized to provide Google Search users with a better answer with the aim of making Google appreciate these pages better, resulting in an increase in more quality traffic.

Within the OrangeValley approach we use a forecasting model. This allows us to isolate and compare the variant and the (modelled) control.

Our approach takes into account external factors such as:

• Seasonal influences
• Google algorithm updates
• Competitor SEO Changes
• Other external macro factors

Any fluctuations such as updates to the Google algorithm will affect both the control group and the variant equally. We do not look at the trend of the traffic during the analysis, but we look at the performance difference between the modeledcontrol group and the variant. We cannot directly compare the variant and control due to possible differences in the variant and control groups, for example the number of pages or the absolute number of sessions per group.

In the following case we show that SEO A/B testing is a reliable and accurate way to get results with SEO changes.

Case Hunkemöller

The test concerns a change of the page title and was eventually implemented on 50% of a selected group of pages on hunkemoller.nl. Where the page titles were previously mainly optimized at keyword level, the focus of the test is on the intention of a Google user. In order to return a better and more appropriate result to a Google user.

The test started in early April and has been active for a total of 54 days. We looked at the following metrics to assess the test:

• Organic Sessions
• Organic clicks (non-branded)
• Organic Impressions (non-branded)
• Secondary Metrics
• Number of transactions
• Revenue

For each metric, we evaluate three things to get an idea of ​​the impact of the change:

• Test score (percent change including confidence interval)
• Significance score (statistical certainty, to know that the effect is fully traceable to the change)
• Daily and cumulative additional traffic or visibility (including confidence interval)

We then create an impact forecast to determine what it will yield when the optimization is implemented on 100% of the selected pages.

For this case, we zoom in on the result of organic non-branded clicks and impressions from Google search results.

Non-branded organic clicks

One of the metrics in the test is the number of organic clicks that come from search engines for searches without the Hunkemöller brand name. The non-branded traffic within SEO is the most important part for most organizations. We therefore view the result separately from searches related to the brand name.

In the image above you can see a development/progress of the variant (black line) compared to the forecast (blue lines). The forecast is based on historical data that we use to predict the expected traffic to the variant.

Already in the first week the lines diverge immediately. This confirms that the traffic to the variant group is outperforming what was predicted and immediately shows that the test is a success. Ultimately, based on non-branded clicks, we see an uplift in organic traffic of no less than 30%.

We use a set of control pages to give the model context to trends and external influences. So if something else changes during our test (eg seasonality), the model will detect and take that into account. The chart above shows seasonal influences. The good weather in May in the Netherlands drives up the demand for swimwear.

The graph above shows the cumulative effect of the traffic increase over our model. The shaded blue area represents a 95% confidence interval. When all three curves perform below (negative) or above (positive) the y=0 (cumulative drift) axis, the test is statistically significant.

On April 10, we had 95% confidence that the results were not caused by chance. At the end of the test, the significance score with a value of 99.9% was even higher.

Non-branded organic impressions
In addition to clicks, it is also relevant to look at non-branded impressions or organic visibility as a metric.

Following this metric shows that it takes a little longer for the lines to diverge. This is partly due to the fact that Hunkemöller has many number 1 positions in searches with many impressions. On the other hand, the positive effect on clicks and focus on a user’s search intent seems to pay off in better positioning and broader visibility. Ultimately, based on non-branded impressions, we see an uplift of 13%.

After a two-week run, we also had 95% confidence that the results were very positive and significant. We kept the test running for a significant amount of time after this to have a full picture of the impact of this change. At the end of the test, the significance score was 99.8%.

Result of the test

The page title change has achieved the expected/desired effect, namely; an increase in organic visibility and traffic. Based on all measured metrics, this test was positive and significant. Based on the findings from this test, the change has been implemented on all selected pages on hunkemoller.nl.

This case confirms that the page title is still a powerful SEO element. Responding to a user’s intent with a relevant page title works when it comes to presenting the best answer to a user.

The outcome of this test led us to subsequently set up similar tests for other countries where Hunkemöller is active. We know from experience that the results of a similar test can differ per site, industry and country. In addition, a schedule is being prepared to test other relevant SEO changes in a data-driven way.

Conclusion

Predicting the impact of an SEO change was virtually impossible. With the introduction of the SEO A/B test methodology, a proven, significant and reliable solution has been found to implement or perhaps postpone changes with a certain outcome. Hunkemöller’s case proved that the outcome was not only easy to predict, but also delivered great business results. The innovative audacity of the Hunkemöller team, combined with our SEO A/B methodology, has paved a completely new path to take their SEO to the next level.

Are you curious about how you can use SEO A/B Testing for your organization? OrangeValley has the knowledge and experience to support you in this process. We are happy to think along with you!