Split Testing ExpressionEngine – Multivariate and A/B Split Tests Tutorial

(Setup Time: Approx. 15 minutes + time spent designing split test)

Today I was asked:

“Is it possible to do A/B Split Testing and Multivariate Testing in ExpressionEngine with ZenTester?”

Yes – it’s very easy to set up a split test with ZenTester.

In fact, if you want to get the most out of ExpressionEngine, you should be split testing – otherwise, you’re leaving your site unoptimized, and missing out on potential signups, sales and profits!

Here’s how to do it:

Step 1: Sign up for ZenTester

If you don’t already have an account with ZenTester, you can register for an account here (link opens in a new window so you can refer back to this page).

It’s quick, easy, and much cheaper than any other set of tools with similar features.

Step 2: Create a test

After you have created your ZenTester account, log in and hit “Create New Test”.

And then select either an “A/B Split Test” or “Multivariate Test”:

I recommend Multivariate split testing, because it will let you test many variables on the one page at the same time.

It also lets you use ZenTester’s neat “WYSIWYG” editor – letting you set up split tests as quickly and as easily as editing a Word Document – so you don’t need to set up a second version of each page you want to test inside ExpressionEngine.

Plus, multivariate tests can tell you more about specifically which variables are improving your conversions.

Here’s a brief video demo I did on a recent webinar:

Once you have finished, save your test.

Then select the type of conversion goal you want to track.

(Most of the time, it’s as simple as selecting “Visitors should visit a specific page” and then entering the URL of your “thank-you” page – i.e. the “thank you for your purchase page” for a sale or your “thank you for signing up” page for a signup.)

If you need help, here’s a tutorial about setting up conversion goals.

Step 3: Copy the ZenTester tracking code

Once you’ve set up your test, ZenTester will give you a single line of javascript code to insert into your website.


This one line of javascript code handles all tracking and testing seamlessly – including making a quick change to the content on your webpage so that visitors see the right version of your split test. This is how you’re able to set up split tests without editing a whole bunch of code inside ExpressionEngine each time. (Like magic, huh!)

If you miss it along the way, you can always go back and find the code by going into “Your Tests”, clicking on the test you set up, and then click “Get Code”.

Step 4: Find your ExpressionEngine Templates

So now that you’ve got your ZenTester tracking code, the next step is to get it into your ExpressionEngine template.

Log into ExpressionEngine, and then go to “Design”, then “Templates”, then “Template Manager”.

Here you should see the templates for your site.

Step 5: Edit The Template

Select the “Template Group” for the page you want to split test, and click on the name of a template to edit that template.

If there is more than one template here, you will want to find the templates that are used for your thank-you page, and the page you will be split testing at the minimum.

  • If you do not have the ZenTester code on the page you are trying to split test, visitors won’t see the split test variations, and you won’t track any stats.
  • If you do not have the ZenTester code on your thank-you page, you won’t see any conversions recorded for your split test.

Alternatively – insert the ZenTester code into the right area of every template in every template group. It will save time next time you go to set up a new split test.

Step 6: Paste in the ZenTester Tracking Code

Paste the one-line ZenTester javascript code into the template, just before the closing head tag (i.e. just before the </head> ).

When you’re done, save your changes to the template by hitting “Update and Finished”.

Step 7: Unleash the Beast!

Now that everything’s set up, it’s time to unpause your test and let it run!

Go back into the test you set up in the ZenTester control panel (to get back to your test, click “Your Tests”, then click on the test you set up.)

Next, click “Configure”.

Scroll to the bottom of the page and click “Start Test”.

…And Watch The Results Roll In

Now that you’re done, check back occasionally to see the results come in!

Winning Like a Wimp

Here’s an odd question…

Who do you think designs stronger, taller and more stable towers made of spaghetti?:

  • Lawyers
  • Business school graduates
  • Kindergarteners
  • CEOs

If you guessed “kindergarteners”, you’re right – and they’re more successful at designing successful towers because they have a skill that you and I have been trained not to have.

I’ll come back to this in a minute – because it has some very interesting lessons for marketers.

Sometimes your cleverest marketing strategies crash and burn.

It sucks to spend months working on something trying to “get everything right before you release it” only to see it fail.

All that time and money you spent developing the strategy (or webpage or widget) goes to waste - and you can’t help but think about the opportunity cost… the time and money that could have been spent better elsewhere, getting a better result…

I hate risk.

And I hate the idea that I could work on something for months, and all my good work might as well have been flushed down the toilet.

This is why I work to win like a wimp.

Wimps don’t bolt into battle with bravado – focussing on winning at all costs – only to end up knocked out on the tiles, lying in a pool of their own bad decisions.

Wimps always avoid having “too much skin in the game” – only ever risking the minimum that they can get away with, but using this to get the maximum possible results.

Here’s the 3-step philosophy wimps use to win on the web – getting better, faster results from every web design and marketing asset they create:

  • MVP
  • Test
  • Iterate

MVP stands for “Minimum Viable Product”.

This doesn’t necessarily mean “create a new product”. It means “do the minimum amount of work you can, and then release your work”.

Most people do the exact opposite of this – they design a website and spend hundreds of hours making tiny and insignificant tweaks to their design to “get everything right” before they’re willing to release their work and see how it flies.

The next phase is to Test.

If you think you’re onto something good – it pays to prove it, quickly.

Step away from your ego for a moment. Sometimes everyone’s cleverest, most creative marketing strategies crash and burn.

So it’s important that you discover this as quickly as possible if you want to avoid wasting time and effort working on the “wrong things”, getting bad results, and looking like an ass.

This is why you develop an MVP, and then test it. Because it helps you to assess the success of your strategy sooner.

If you’re testing a web page, multivariate testing helps you to get more in-depth insights into WHY your strategy works better or worse – which you can use in the next phase…

The final phase is to Iterate

Based on what you learn from testing, start iterating.

Go back and make MVP-style tweaks and changes to your strategy based on what you’ve learned… Then go back and prove those changes through testing.

…And then do it again.

Here’s Why Kindergarteners Beat Lawyers, Business School Graduates and CEOs…


Practical Split Testing Strategies Webinar

STOP! – Before you focus any more time into SEO, pay-per-click promotions, affiliate programs, joint ventures, email marketing or any sort of advertising, you need to watch this webinar.

Lately, I’ve been seeing far too many people saying the easiest, most profitable, most effective and fastest way to increase your sales is to simply get more people to your website.

I’m a bit believer in attracting more qualified leads…

…But if you only focus on traffic, you miss out on all of the best traffic-generation opportunities. (I know this sounds odd, but take a look at our latest webinar to see what I mean.

Webinar Summary

In this webinar, we cover three topics:

  1. The Snowball Effect: How one split test lead to 300,000+ new signups…
  2. 5 Simple Strategies for Faster, Easier and More Profitable Split Testing, and;
  3. An over-the-shoulder look at exactly how we set up split tests inside ZenTester.

Below I’ve included the full webinar recording, plus 3 edited versions covering each of these topics individually (in case you only want to focus on one or two of these three topics.

ZenTester Webinar: Practical (Full) [43 minutes 24 seconds]


ZenTester Webinar: Practical (Part 1: The Snowball Effect: How one split test lead to 300,000+ new signups…) [14 minutes 32 seconds]


ZenTester Webinar: Practical (Part 2: 5 Strategies for Easier, Faster, More Profitable Split Tests) - [ 8 minutes 28 seconds]

ZenTester Webinar: Practical (Part 3: Split Testing Demo) - [19 minutes 47 seconds]


New ZenTester Redesign! Tell us what you think

We’re excited to announce that we’ve finally given the ZenTester website a much needed facelift and brought it into the modern world. The primary reason we wanted a redesign was the old site was hard coded in PHP, which was fine and worked well, however it meant that whenever Brent wanted to add new content or change pages on the site he had to ask me to make the changes.

This doesn’t seem like a big deal but when our software is changing and improving as rapidly as it is the website can quickly get out of date and having this unnecessary friction holding us back was unacceptable. I’m also pretty obsessed with efficiency and ensuring everything is as effortless and fast as possible and as such this was a pretty big pain point for me.

So we’ve moved everything across to WordPress which makes things much easier to manage.  I’d love to hear your feedback on the new design and if you notice any broken links / images / pages or something that just isn’t right please let me know.

There are also a number of amazing plugins available that can take care of many of the normal annoyances that come with running a website, here are some of my favourites:

  • WordPress Backup to Dropbox – I discovered this plugin a few months ago and absolutely love it. It automatically backs up all your files and database to Dropbox so that if your server ever completely dies you have a recent backup available and can get up and running again in a few hours. If you are on a premium dropbox plan with unlimited revisions of files it gets even better as if something strange happens with your site you can go back in time in dropbox and see what’s recently been changed to find the cause of the problems.
  • Bulletproof Security – Although WordPress’s security is quite good these days Bulletproof Security adds another layer on top of that to ensure your blog stays completely free of hackers (or makes it not worth their while to attack you in the first place). It’s also quite simple to get up and running, only minimal configuration is required.
  • Amazon S3 for WordPress with CloudFront - A more obscure plugin that certainly isn’t for everyone but is fantastic for us. What it does is automatically uploads all your blog images and media to your Amazon Cloudfront account and serves them from there rather than from your web host. Amazon Cloudfront is a super fast Content Delivery Network that syncs your content with servers all around the world. This means that whenever a visitor comes to your site instead of downloading all the media from your server which is probably on a slow shared host thousands of kilometres away it grabs all the content from the closest Cloudfront datacenter. All up this means your site will be much snappier to load and your visitors will have a much better experience using it.
  • Redirection - Redirection is super useful if you’re ever moving a site from raw PHP to WordPress (as we’ve done) or changing the structure or permalinks of your site. It allows you to redirect any url on your site to any other url using a 301 redirect which Google loves and your SEO will be intact. It also gives you a log of any pages that are 404′ing so you can redirect these 404 url’s to the correct location or just send visitors that go to them back to the homepage of your site.

We also run a lot of other common plugins such as All in one seo, W3 Total Cache and Google Analytics that are pretty much standard installs on all WordPress blogs anyway so I won’t repeat them here.



Starting a New Site

Recently, we’ve received a few emails from people who are starting new websites, asking for tips on how to apply split testing principles to their new site.

This video looks at:

  • The value of split testing a new website;
  • The “Snowball Effect” – and how it benefits your business in the early stages;
  • What you should be testing early;
  • Where to get great ideas about what works (and what doesn’t);
  • How to design a high-converting site from scratch (and then improve it);
  • How to save significant amounts of money on graphic design;
  • Why it’s valuable to use the money you save to invest in test variations;
  • And much, much more!

Creating Irresistible Offers

One of the biggest reasons why offers fail to convert is because they do not meet customers’ needs.

Irresistible offers are offers that not only help meet your customers needs, they typically provide a high conversion rate to serve as a solid foundation for future split tests.

In this video, Brent shares a simple case study for discovering your customers needs, then using their feedback to maximise your sales, and profit-per-sale.

Creating and Testing Eyeball-Sucking Headlines

It’s no wonder why people split test headlines first.

They attract attention, they’re easy to split test, they’re prominent and they can lead to huge conversion gains.

BUT it can be deceptively difficult to write a great headline.

(In fact, top copywriters spend years honing their skills here.)

The 30-Second Trick for Writing Great Headlines

A closely-guarded secret in the copywriting community is that copywriters don’t always write from scratch.  In fact, they often take inspiration from other proven sales copy.

These headlines are taken from some of the highest performing pieces of sales copy ever written:

  • Do you make these mistakes in English?
  • How to create financial advertising that sells…
  • Amazing Secret Discovered By One-Legged Golfer Adds 50 Yards To Your Drives, Eliminates Hooks and Slices… And Can Slash 10 Strokes From Your Game Almost Overnight
  • Never Forget Anyone’s Name Again As Long As You Live!
  • They all laughed when he sat down at the piano! …But when he started to play…
  • Are we being lied to?
  • Only Way Left for the Little Guy to Get Rich
  • The Ultimate Tax Shelter
  • How To Eat Your Way Out Of Fatigue
  • The lazy mans way to riches
  • At Last! A Plastic Surgeon’s Diet
  • Do You Have The Courage To Earn Half A Million Dollars A Year?
  • How A Fool Stunt Turned Me Into A Star Salesman
  • Are The Foods You Are Eating Today Starving Your Brain?
  • Is It Immoral To Make Money This Easily
  • Free! 30 Ingenious “Super Money-Makers” You Can Use Tomorrow! (Note: There is evidence to suggest “free” might affect deliverability in emails.)
  • Don’t Pay One Penny Until This Course Turns You Into A Human Computer
  • Don’t Spend A Cent On Ink Cartridges Until You Read This Closely-Guarded Industry Secret
  • Will This Be The Most Explosive Turn Around In The History Of The Stock Market?
  • Weird New Sonic Lure Caught Fish Like Crazy!
  • The Sex Food So Potent Priests Were Forbidden To Eat It
  • 71 Year Old Man Has Sex 5 Times A Day

How To Get (And Test) Killer Testimonials

Testimonials have the potential to significantly improve the conversion rates you get from your website.

But not all testimonials are created equal.

In this video, Brent shares 3 questions that elicit three types of very powerful testimonials.

These questions gather testimonials that are:

  • Feature focussed;
  • Outcome focussed, and;
  • Recommendation focussed.

Brent then shares what to do with the testimonials next, and several practical examples of what to do in a testimonial split test.

Why Bother Split Testing Testimonials?

You might be wondering “If testimonials are so great at increasing conversions, why do you need to split test them?”

In a recent split test, a ZenTester user discovered they achieved the highest conversion rates when they had a lot of testimonials, or no testimonials – and only having a few testimonials actually damaged conversions.

In another test, a 34% increase in conversion rates was achieved by having relatively inconspicuous testimonials on a page. (Typically, testimonials work best when they are displayed prominently.)

Some formats (with photos, with bolding and highlighting, in a separate box, in the sidebar, in the main page text) work better on different sites.

And some types of testimonials will resonate with your customers better than others.

In the same way that not all testimonials are created equal, different layouts, formatting, and types of testimonials will convert at different rates.

This is why it’s valuable to test your testimonials, rather than simply trusting that they will improve conversions.

ZenTester Split Testing FAQs

Sometimes, getting away from the keyboard is the best thing you can do when you’re working online.

It gives you a chance to meet some really cool people, hear what people think about your products or services, and discover the information you might not be communicating effectively.

Recently, I had the pleasure to speak at an event for around 200 incredible entrepreneurs from all over Australia and New Zealand – an event run by best-selling author Dale Beaumont for his Business Blueprint group members.

After hearing how ZenTester worked, a large number of people decided to take the plunge and become ZenTester users themselves – including a lot of people who were struggling with other split testing tools like Google Website Optimizer and Visual Website Optimizer.

But I was also asked a lot of great questions about how ZenTester works, and how to get the most value out of split testing.

In this video, I take the opportunity to answer these questions (and more), including:

  • What platforms does ZenTester let you split test? (WordPress, Joomla, Drupal, Magento, or plain HTML sites)
  • How many landing pages can I run split tests on?
  • How many split tests should I run at once?
  • How often should I be split testing?
  • How long does it take to complete a split test?
  • How can I speed up my split tests?
  • Where should I begin split testing?
  • How should I set up complicated split tests?
  • AND What to do if you’re not ready to begin split testing yet…

I hope this video answers some of your questions too!


P.S. – Don’t you just hate it when the YouTube preview image catches you making a funny face?

How Split Testing Increased Airline Ticket Sales by 591% – In A Down Market

When the Global Financial Crisis hit in 2008, it took a toll on many businesses – but none more than businesses who sold luxury items, like travel, jewellery, furniture and cars.

The GFC was a sudden (and brutal) reality check for many online businesses working in these industries, who saw their sales slide backwards almost overnight when consumers stopped spending – and airline websites were some of the worst hit.

The subject of this split testing case study – – was in just this position. But one clever split test (designed by Blast Advanced Media) increased airline ticket sales by 591%.

So how did Blast boost the sales of a luxury item (air tickets to remote tourism destinations) by 591% – and beat all of their previous online sales records – in a down market?

Well, first – they improved the usability of their site…

Why Usability Matters…

When a user visits your site for the first time, they need to quickly “learn” how to use the site to get what they want.

It’s a new (sometimes confusing) experience that takes effort.

The problem is human beings are designed to be efficient with our energy. We automatically calculate the short-term reward of getting something we want, and the short-term effort it will take to get the reward.

If the effort is greater, we usually lose interest (get lazy, do something else, try another option, etc).

Online, that means potential customers click the back button and leave your site.

If the Short Term Pain is greater than Short Term Gain…

…People do something else instead.

Because this is such a powerful driver of human behaviour, it makes sense to spend time focussing on improving your website in these areas:

  • Removing obstacles and “pain points” that may be decreasing your conversions, and;
  • Creating easier, more valuable, more obvious conversion goals and offers.

This is exactly what Blast Advanced Media did to reverse the effects of the Global Financial Crisis on’s website, and increase conversions on airline ticket sales by 591%, practically overnight.

What Was Tested

Blast made it easier for visitors to to purchase by providing them with a clear call to action that was relevant to their needs – and put it in a prominent place of the page.

Here’s what the original page looked like:

Here’s what the tested page looked like, after the new call-to-action box was added:

And here’s a closer look at the box that was added to the page:

In short, the test:

  1. Created a call to action with obvious benefits to the user(by focussing on the great discounted fares on offer);
  2. Focussed on the user’s needs by showing the fares to the specific destination that the customer was reading about, and;
  3. Made it easier to proceed with a booking by placing the call to action right beside the information visitors were reading.

(If they had heatmaps running on their tests, I’m sure they’d also see that the placement was also right next to the busiest parts of their site design – where they get the most clicks, and the most mouse movements – also making it easier for users to click.)

The conversion goal for this test was airline ticket bookings.

The Results

In this test, the new version resulted in a 591% increase in conversions – customers booking airline tickets with

A 591% increase in ticket bookings meant nearly 6 times more sales for

And for businesses that have mainly fixed costs (like most websites, and airlines), a 6x jump in sales revenue can mean a big leap in profits.


It’s no fluke that Blast Advanced Media found a winner with this test. After all, they focussed on time-tested principles when assessing what to improve on their site:

  1. Creating a call to action with obvious benefits to the user (by focussing on the great discounted fares on offer);
  2. Focussing on the user’s needs by showing the fares to the specific destination that the customer was reading about, and;
  3. Making it easier to proceed with a booking by placing the call to action right beside the information visitors were reading.

And by focussing on these time-tested principles of website optimization, they achieved an incredible split test result – a 591% improvement to sales from these pages, practically overnight.

(As a side note: A second similar variation that was tested also achieved a staggeringly high increase in conversions, although not quite as high. A 497% increase – illustrating how important these principles were in improving’s website!)

About This Test

This test was designed by Kayden Kelly, CEO of Blast Advanced Media. (A big thank you to the team at Blast for permission to use the test results in this case study.)

You can find more about this test, and how other companies have significantly increased their conversion rates through split testing, by following our blog for more (and more details on how to get our entire library of great split testing case studies!)

Page 1 of 3123»

    No Twitter Messages