CRO EXAMPLES FROM EXPERTS

Daniël Granja Baltazar – CRO Specialist at VodafoneZiggo

Standing Up to a “Big and Bold” HIPPO

I was working at a digital agency and got a well-known Dutch luxury clothing brand as a client. During our introduction call I presented a few A/B test ideas based on data. The founder of the company dismissed all the ideas. He wanted to test “big and bold ideas” only.

I showed him a few tests that I ran for clients in the past quarter to convince him that subtle changes can have a big effect. At the end of the meeting he gave in: “But if this first A/B test does not work, we are switching to a big and bold test next.”

I agreed and proceeded to set up the A/B test. It was quite simple: adding the most used payment methods underneath the “add to cart” button. However, I personalized those payment methods based on the five biggest markets where we were going to A/B test this (The Netherlands, Germany, France, USA, and Canada).

The result? The A/B test saw a statistically significant uplift in conversion ranging from 2% to 6% (depending on the market). “Low risk culture countries” like Germany saw a stronger conversion uplift compared to “high risk culture countries” like the United States. Nevertheless this “simple” A/B test generated the client tens of thousands of euros of extra revenue each month.

After that A/B test the founder did not mention “big and bold A/B test ideas” ever again.

The Moment You Can’t But Become a CRO

Your first test is like your first love – you remember it forever. It really shaped me as a CRO. I was working for a Dutch museum which naturally wanted to sell more tickets online.

At the time there was a big debate going on internally on how to improve the exhibition pages on their website.

There were two camps:

1. The marketers wanted short pages with visuals and a big “buy” button.
2. The curators (who wrote the content) wanted lengthy and detailed pages to explain what the exhibition was about. And it was how the exhibition pages looked at the time.

I proposed to set up an AB test in Google Optimize (which was free – that always helps with convincing) to test which version would work best. The marketers and the curators both agreed.

I set up the AB test where I tested the detailed exhibition pages against shorter and more visual pages which briefly described the exhibition.

The result? Users spent twice as much time on the new exhibition pages and bought eight times more tickets for the exhibition compared to the control variant. That A/B test proved to me (and the Dutch museum) what the value is of A/B testing.

I’ve been a fan of experimentation ever since!

Jörg Dennis Krüger – The Conversion Hacker

Trust > Aesthetics (Learned the Hard Way)

There was a prestigious fashion e-commerce client, and the experiment was about introducing a minimalist redesign of their product pages. Our vision was to enhance the user experience and boost conversions. And we were quite confident in it.

Except, initially, this radical approach seemed to be a misstep — conversion rates fell sharply, putting us in a challenging spot with our client.

But, this became a turning point in my understanding of conversion optimization. We did a comprehensive analysis of user behavior and found something, something really important.

While the design was aesthetically pleasing, it lacked essential trust elements. Customers were engaged – but reluctant to complete purchases due to a perceived lack of trust in the new layout.

This taught me that the fundamentals of conversion optimization often trump subjective ideas of user experience and design. Trust, clarity, and reliability are paramount, sometimes outweighing purely aesthetic considerations.

By integrating strong trust signals like customer reviews, trust badges, and transparent return policies into the design, we not only recovered but surpassed our initial conversion rates.

Get a free strategy consultation with Jörg here.

The First Major Win is What Makes You Stay

The stage was set with a store specializing in personalized presents, and our audience: visitors from Google Shopping and organic Google search landing on a Product Detail Page (PDP).

Our hypothesis was simple yet ambitious: enhance product visibility to aid decision-making, thereby reducing bounce rates and elevating conversions. This led to a strategic shift: moving “recommended products” from the PDP’s bottom to the forefront. The outcome was nothing short of remarkable—a staggering 15% surge in sales, diminished bounce rates, and heightened user engagement.

This experience was transformative. It underscored the immense power of personalization—not just as a user experience enhancer but as a catalyst for significant business growth. It challenged traditional perceptions, proving that even minor modifications, when rooted in a profound understanding of user behavior, can yield extraordinary results.

This experiment redefined my professional trajectory, instilling a relentless pursuit of testing and optimization, particularly in personalization. It reinforced my commitment to data-driven strategies, continually adapting to empirical evidence. It urged me to delve deeper into the untapped potential of personalization in CRO, setting new benchmarks in the field.

Bryan Bredier – Expert CRO Maiit

Fired for Increasing CR by 60% (and Then Hired Again)

I recently had a very impatient client. They wanted to increase the conversion rate of their store but had a very short-term vision. They did not agree to leave time for tests to find the best combinations and wanted to change things after 24 hours. Sounds familiar, doesn’t it?

In the end, the collaboration quickly ended with the client removing access to the website for me. But they didn’t know how to restore the original website, so they left the modifications in place.

A week later, the client came back to us after noticing the changes had increased the conversion rate by around 60%. Turns out, our A/B test defined the best combination!

Guaranteed Hacks Work… Until They Don’t

Like most people at the beginning, I chased a lot of “hacks” that were “guaranteed” to improve the conversion rate of my first website. Well, they did!

With these tricks in my pocket, I was convinced that I could replicate it on any website – until I came across a site for which my tips did not work at all. I was puzzled. I dug deeper and realized that my approach was not good for a very basic reason.

Behind the screen, there are humans. And – surprise, surprise – different people do not react the same way. So, the approach in CRO (and in general in business 😅) must focus on understanding the human behind the screen more than on hacks.

Well, that was the moment when I understood the importance of putting the right tracking tools in place!😉

Paul Randall – Sr. Experimentation Strategist at Speero

Don’t Peek and Challenge Your Beliefs

Here are the two notable things I’ve learned from running experiments over the course of my career.

Many years ago, I would be looking at tests on a daily basis, but I began learning that a desired sample size is needed before you can make any conclusions.

It’s often stakeholders who will peek and want to stop a test early, but I’ve learned that communicating the number of users per variant helps them resist the urge.

Some testing tools still send out messages way too soon though!

So, don’t peek; have patience.

Another profound test also happened early in my career, as I transitioned from a UX practitioner.

We ran what seemed like quite a counter-intuitive experiment at the time, which was to move a lead generation form behind a preliminary step.

We did find that by not immediately showing the form, we were able to increase leads.

This pushed against many internally held beliefs about what a good landing page looks like, and it paved the way for even more experimentation.

Don’t be afraid to challenge your beliefs!

Brian Massey – Conversion Scientist™ at Conversion Sciences

CRO Stands for Continuous Redesign Of-the-website

We had been hired to work with a design team on a website redesign. Most website redesigns are what I call “all in” designs: the web development team spends months making thousands of decisions with the input of the executive team. No one knows how they’ve done until the new design launches. And there’s usually no going back if conversions drop.

We changed this process.

We redesigned the site one page at a time, testing our designs along the way. This meant that some pages had a new design while others had the old design. How many designers cringed at that point? I understand, but we were able to test away bad design decisions and keep good ones.

The redesign process had been expected to take more than six months. However, conversion rates had shot up starting in the second month and continued to climb each month as we worked our way through the site.

This is when I realized an important advantage of data-driven design: You get results much sooner, and the results are guaranteed. Today, we offer data-driven redesign services using experiments to back up our design decisions.

CRO is continuous website redesign.

Get a free quote for Conversion Optimization Services

Nick Phipps – Head of CRO at idhl

Tried Adding Products, Scrapped the Whole Merchandising System as a Result

During some research for an eCommerce client, we identified a large drop-off rate between category pages and product detail pages.

I dug a little deeper using session recordings and heatmaps to discover that users were only scrolling through the top few product cards on a listing. I thought that poor card design and a single-column layout were to blame.

So, we ran a few tests to try and show more products to each user. But all we got were inconclusive results, even on secondary metrics like click-through rates on the products.

This led to deeper research that helped us discover that the merchandising system (including the filters) was not fit for purpose. Users were struggling to find the products they were looking for, wandering around the website without any support.

Eventually, the system was replaced. The thing that stuck with me was to always keep iterating and always dig deeper during analysis. Go beyond the numbers.

Ensure you connect your testing tools to GA4 and behavior analytics tools like Mouseflow to gain a deeper level of detail that the numbers can’t give you.

What Happens When a Young Digital Marketer Decides to Try A/B Testing

I won’t surprise you if I say running my first AB test changed me for life. I can’t remember what it was or what the result was, but the path that it sent me down changed my career.

I didn’t really know anything about CRO or experimentation at that time. I was working in digital marketing, trying to find ways to improve some campaign landing pages. At some point, I’d read about A/B testing as a method to measure the impact of changes on a website, so I thought I’d give it a go.

I designed my experiment and used the visual editor (a harrowing experience) to implement the changes. I set some goals and I launched the test. It didn’t work… After lots of googling and banging my head against a wall, I managed to get it working and I remember thinking that I’d basically just become a developer!

That experience led me down the path to where I am today. I cringe at the thought of those early moments playing around in Optimize with no real strategy or understanding of CRO as a methodology, but it was the first step I needed to encourage me to learn all I could. It propelled my career into the most rewarding disciplines within digital (even though there may be a hint of biased opinion here).

Jesse Reynolds – CRO at ExtraHop

Competitors Don’t Always Know Best

For many SaaS companies, “free demo” is the main call-to-action and the accompanying form is the most important one. We identified that our form had a below average completion rate.

At the time, the form was a small overlay over our live demo experience. The form had five fields and presented only the bare essential info (form field titles, privacy links, etc.).

We opted to test a new page instead of the overlay and created a form that’s more in line with our competitors. We hypothesized this would meet user expectations and, by adding value props, it would ease their decision-making, increasing form completion rate.

The test utterly failed. We measured a 17% decrease in form completions and the number of qualified leads dropped 22%. Testing what our competitors were doing did not work for us.

Regrouping on the failed test, we examined what we knew about our users, specifically those interested in our demo. We realized we were approaching them the wrong way.

They weren’t the decision makers, they were those who would actually use the tool. And they were likely checking it out to report back to their leadership. We needed to create a positive experience for those users.
Setting up a new test, we focused on the benefits of the free demo in a “quid pro quo” approach using direct, “plain speak” (give us your email, and we give you access to the tool, plus 5 scenarios you can go through, etc.).

The test was a success, increasing form completion rate by 24% and qualified leads by 20%, even though it decreased total form submissions. We identified that we had failed to create clear journeys for our different users/personas and needed to add more clarity in our content about what the user could expect. The form now helps users understand what to expect and the results led to the decision to totally redevelop the website.

You Can Drop the “CR” in “CRO”

While I was working for an agency, one of our clients was a well known hotel brand. They had various tech limitations but were trying their best to incorporate A/B testing and optimization.

The team spent a good deal of time doing competitive research, looking deep into the data, thinking through design heuristics and best practices to come up with a super comprehensive deck of test ideas with thorough evidence and organized into one of the best presentations I have ever had the pleasure to work on.

And of course, the majority of the ideas were shot down, but one made it due to how simple it was. We suggested that, while users are searching for hotel rooms, the rooms should open in new tabs when clicked on.
Such a simple idea that was overlooked by the client in search of a “perfect” solution led to an increase of 12 million dollars in MRR.

It was a strong reminder that the user experience is the alpha and omega. Despite pressures to make the numbers go “up and to the right,” we have to focus on making the users’ experience better.

That’s why I like to avoid the term CRO and focus on just the “O”, Optimization. Maybe instead of CRO, we can coin Customer Experience Optimization to muddle things even more 😉

Ubaldo Hervás – Head of CRO at LIN3S

The Novelty Effect

I worked as a data analytics & CRO consultant in an American-based SaaS startup. My job here was to identify opportunities in the retention phase and to apply online controlled experiments to improve the core metrics.

Here’s one of those experiments. We found a negative correlation between the average time on specific features and the Churn Rate: The more time the users spent on particular product features, the fewer users uninstalled our application.

During the discovery phase, those features were probably the most important of our software-as-a-service in churn rate management terms. The hypothesis was that the churn rate would decrease if the users used this feature more.

We ran an A/B test, but the control and variation performed similarly during the first fourteen days. We thought it was a failure or an inconclusive experiment. However, during the third week, things changed.

We didn’t use churn rate as a primary metric, but focused on some engagement-related metrics. The variant started to perform much better, and the reason behind this was an effect called the “novelty effect.”

The “novelty effect” refers to the phenomenon where a change or something new on a website (such as a new design, feature, or product) initially attracts more (or less) attention and engagement from users, leading to a temporary increase/decrease in metrics.

The key learning here is that no Conversion Rate Optimization or Online Controlled Experiments standards exist. Most of the time, you will pause that campaign, and you shouldn’t (in that specific case). You will try your best to improve core metrics, but nobody can know what a variant’s behavior will be over a period.
When you realize the natural effect of the “novelty effect,” you become more humbled.

Research Cannot Be Optional

The success of a CRO initiative is 90% dependent on the quality of research. I learned that lesson when I started in this wild world of online experimentation and tried to base my experiments on best practices. After all, that’s what most people related to this area posted at that time: good practices, checklists, etc.
The results were equal to zero.

Then, I realized that you need to find meaningful pain points – and opportunities to offer meaningful solutions. Sometimes, finding good problems is more important than finding good solutions. From a CRO perspective, you need to research and prioritize those problems or opportunities and start working on them through experimentation (or not – experimentation is a means to an end).

There was no single experiment that told me that lesson. Several experiments with no business context, no key metrics detected, no stakeholders alignment, and no pain points detected were the best way to learn that throwing spaghetti at the wall to see what sticks is the best way to spend money, energy, and technical resources with no return.

Since then, research has not been optional in my CRO projects.

Juanma Varo – Head of Growth at Product Hackers

Experimenting on LinkedIn

In my opinion, a true CRO expert is one who deeply values and fully embraces the process, viewing it as their most potent asset. But one of the most transformative experiments I conducted was driven by sheer necessity, and there was no A/B testing involved.

My objective was to leverage LinkedIn as an efficient acquisition channel, aiming for low-cost conversions and sufficient volume for scalability. Despite experimenting with various ad copies, targeting different roles, companies, and industries, success was elusive. I was two months behind on my goals.

Then, I had a breakthrough. I set up a page that featured awards for various teams within an organization, including product managers, growth managers, designers, engineers, etc. I invited LinkedIn users to vote for their teams.

That led to an extraordinary engagement rate – more than tenfold compared to previous efforts. This initiative not only boosted engagement with our content but also significantly increased submissions to our programs.

Within weeks, I achieved the year’s submission goals, freeing me to explore new, innovative strategies to further enhance our results, all the while alleviating the stress and pressure I had been under.

Martin Greif – President at SiteTuners

The Ugly Face of Trust

I ran an A/B test for a VPN provider years ago that helped me understand the delicate balance between design aesthetics and the perception of trust and security.

We were testing a payment form. The control version had an outdated and overloaded interface provided by a payment processor. The variation was a custom-designed, visually appealing payment page that matched the website’s overall, much more modern design.

Contrary to expectations, the control version outperformed the beautiful custom design by a staggering three-to-one margin. This outcome was measured across key metrics: conversion rate, sign-ups, and revenue. Apparently, when it comes to payment interfaces, the design appeal mattered much less than trustworthiness – in this case, the fact that the interface was unmistakably linked to a recognized payment processor.

Customers were reassured by the familiar, albeit dated, appearance of the control version, interpreting it as a sign of legitimacy and protection for their credit card information.

Armed with these findings, we then ran experiments across the website. We placed trust elements such as privacy seals, reviews, and ratings near every call to action, bolstering the site’s credibility.

Of course, we kept the original payment page, but we also made enhancements to all pages leading to the payment processor, ensuring they communicated trust and security more effectively. And we made sure to include user reviews wherever possible to build trust even further.

This test told me once and for all that every design should accommodate the users’ psychological need for security. I returned to it many times when working on subsequent projects, remembering the importance of trust signals.

There’s an ongoing and evolving challenge in digital marketing and design: balancing the aesthetic appeal with the need for trust and security. And in some instances, the ugly is the winner.

Maria Luiza de Lange – CRO Lead at Tele2

Small Changes for Great Results

There were a couple of small tests that were very successful and that changed my philosophy when it comes to experimentation.

One example was a test where we added a simple phrase “tap to zoom” under the images in the image gallery on the PDP for Electrolux.se.

We could see from the data that the more consumers interacted with images the more they converted. And that portion was much lower on mobile devices than on desktops. By increasing the interaction with images on mobile devices through adding just one simple phrase, we were able to increase conversion.

Another example is the one we did with the mobile navigation on tele2.se. Again, we just added words “menu” and “search” under the respective icons – and that significantly increased interaction with these elements.

Both of these prove that no matter how small the change, it can still have an impact. Kind of repeating Ron Kohavi’s mantra: “Test everything, even the bug fixes”.

I also noticed that tests like these tend to energize and fascinate the whole team. They are easy to understand and bring good results.

Simone Schiavo – Sr Customer Lifecycle Manager at Leadfeeder

Curiosity and The Small Things

If you ask Simone Schiavo what is at the core of what drives him, he’ll tell you it’s curiosity. In his nearly decade-long career, Simone has had his share of challenges and ‘Aha!’ moments. The one thing that never changed was his ability to apply his curiosity to solve problems and repeat successes.

In the early days of his career, Simone had to learn the hard way that not everything needs to be a random experiment. While working at an online newspaper company and running an A/B test on their pricing page, he noticed a fall in the checkout metric.

The issue? Everyone was running their own experiments in a silo. During his experiment, he forgot to exclude paid traffic. At the same time, the PPC team was running a different campaign that wasn’t bringing in a lot of qualified leads. Simone says he learned that we shouldn’t think of experimentation or conversion as a stand-alone function. Don’t create silos, but instead, think of experiments as a way to empower running processes.

He had his career-changing experiment more recently, while working in the eCommerce space. The company received a lot of traffic, so the team could test a lot and reach statistical significance quickly. What did they do?

Instead of doing one huge experiment, the team focused on executing many smaller changes like removing the home page slider and reducing core page load times. Quite a few of them worked! Simone became convinced that by rolling out smaller changes, you don’t have to focus on huge breakthrough wins to see big results. Instead, aim for smaller wins that compound into something bigger.

Yes, you need to understand the basics of data and code to be successful in this field, but staying curious and applying that curiosity to what your team is working on is essential.

Chris Marsh – CRO Specialist for eCommerce at Dash of CX

The Importance of Not Giving Up

I’ve failed in many ways over the years. The very first test I was tasked with launching needed to launch at a specific time, to coincide with a newsletter. Unfortunately, back in 2017, the tools were quite clunky, and I thought I’d launched it… but I hadn’t.

Thankfully, the client was understanding, and the agency trusted their gut to keep me. Since then, I’ve become very detail-focused, diligent, and have been called a “QA madman” on some occasions!

For one client, we learned how important prices and discounts were to their audience, so we tested various ways to showcase the discount on the product page. Multiple variations and iterations failed: showing a % discount, $ off, or both – all failed.

Then, we tried other solutions and a wildcard idea – displaying the message “Now on sale” instead of a discount figure. And that finally produced a winning solution.

This taught us the importance of iterating. And, when possible, it’s great to run an A/B/C/D test – you’ll try more ideas and interesting solutions and have more chances to win!

Johann Van Tonder – CEO at AWA Digital

Wow, That “Talking to Customers” Thing Actually Works!

“Fix them, sell them, or kill them.” – this was the brief from my boss at a global internet investment firm.

I’d been given a small portfolio of stagnant online businesses. It was the early days of eCommerce: we all copied each other, but nobody knew what they were doing.

I did what people in corporate normally do – throw money at agencies and consultants. Heaps of money. No results.

Almost out of desperation, we started talking to customers. We had no formal methodology, just got on the phone with them and found out about the problems they were facing that brought them to our products.

The insights were pretty unexpected, certainly not in line with ideas from our internal brainstorming sessions.

I remember the reluctance with which we started implementing some of the changes. I also remember the disbelief in the results – real money in the bank, not just conversion rates or some fuzzy metric.

A/B testing tools were a new invention back then. The main reason we started testing was to speed up the deployment cycle. In the process, I discovered that some of the “no brainers” didn’t actually make any difference.

It’s such a simple discovery, but it fundamentally changed the trajectory of my career and life. The power of understanding the world of your customer, coming up with theories about how to improve their experience, and validating that thinking – this approach became such a revelation! To this day, thousands of experiments later, I still have that feeling of magic: “Wow, this actually works!”

Ruben de Boer – Owner at Conversion Ideas

Embracing the Scientific Approach to Everything

I ran a series of experiments for a furniture e-commerce shop in 2016 – and they changed my approach forever.

I had been in CRO for many years, but this was the first time I adopted a fully hypothesis-driven approach with many insights from scientific literature. I changed how I documented tests and insights to learn more about the user’s needs and motivations.

It led to a very successful CRO program, with many winners and key insights for the whole business. These insights were, for instance, also used in offline marketing.

Ever since I have firmly believed in a hypothesis-driven approach to CRO (instead of random testing based on so-called best practices) to truly learn from all experiments and to have excellent documentation.

Also, scientific literature became one of my most essential sources for optimization research. I even created a free online course, ‘CRO Process and Insights in Airtable,’ so others can benefit as well.

Fortunately, soon after, Online Dialogue approached me to become a consultant on their team. As they had been working this way at a world-class level for many years, it was a no-brainer for me to accept their offer.

Furthermore, in my personal optimization (like health, fitness, and sleep), I am now more focused on books and teachers who take science into account to get better results for myself.

Anthony Mendes, Head of Strategy at Get Social Labs

A Win Is Building Paths Through Failed Experiments

Most failures with experiments turn into successes one way or another. We’re led to believe that successes often take shape as a growth hack or a home run experiment – but that’s not true.

CRO as a whole has taught me that to succeed is to build many small paths through failed experiments.

No one knows a thing – until it’s tested. And even those tests can vary dramatically depending on who runs them!

Example: there’s a recent experiment that I ran for a SaaS client.

We created a new version of a campaign where we changed the motivation for the user to take action.

At first, we thought it’d be a failure, as the number of daily leads we were getting was lower than in the control version.

But this month, we found out that the conversion rate of the variant’s leads into sales was six times higher. The reason behind this is that leads arrived at sales reps primed for a specific conversation to happen.

Monique Poché – SaaS Conversion Copywriter

Change That Copy

I had a client reach out to me regarding a sales campaign that wasn’t working. It was a Facebook ad that took people to a landing page for a free quote on credit repair.

They had tried everything – from changing the Facebook audience to switching around the messaging. Nothing worked.

When they approached me, it didn’t seem like there was a whole lot more that I could do… until I looked at the copy. It was very impersonal and didn’t touch on their target audience’s pain points at all.

I reworked the messaging both on the ad and the landing page, even though my client didn’t think it would work given their recent failures.

By matching the messaging better between the ad and the landing page, and driving home the pain points without going overboard, my efforts were a success. The ad went from 30 impressions in two weeks to 131 in 6 hours. The landing page went from about a 5% conversion rate to 57.5%.

That’s how I learned that the copy and messaging can always be improved!

And it made me realize that it’s not a bad idea to suggest tweaking the copy regardless of where my clients are at in the testing/monitoring phase.

A fresh set of eyes can do wonders when it comes to optimizing conversions. Some people get so immersed in what they’ve always done, they might be missing a key element that makes all the difference.

This philosophy has definitely shaped my career as I help people optimize copy through audits and rewrites. It often enhances their results – and I’m always learning something new throughout the process.

Get your free audit by Monique here.