Web

3 A/B Testing Tools Compared

Jacco Blankenspoor

Published April 11, 2014

Tweet Subscribe

In my recent article that covered Google Analytics alternatives, someone asked a question in the comments about A/B testing. In response, two tools were mentioned: Optimizely and Visual Website Optimizer. In this post, I’m going to review these in addition to another one called Google Analytics Content Experiments, which offers simplified (but free) A/B testing functionality as part of the Analytics suite.
If you’re new to A/B testing in general, I recommend Kerry Butters’ articles Are Most Winning A/B Test Results Misleading? and The Designer’s Guide to A/B Testing, both of which include more basic info on what A/B testing is and why it’s important.
Now let’s go on to the reviews.

Optimizely

Let’s start by looking at Optimizely. They offer a decently priced entry plan with 2,000 visits per month, which is enough to run a few tests (or one large test). Optimizely requires you to insert a snippet of code in your header (which gets confirmed by email, which is nice) after which you can use their dashboard. You just add a new experiment, and a guide takes you through all the necessary steps.
I prepared an experiment to test whether Google Adsense link units should be put below (how it currently is) or above the navigation links. You can see the real page here to get an idea.
I can easily select the related code and make the switch. This is just a test, but if it was a real analysis I would also need to generate new Adsense code to measure the impact in earnings. But for now it’s the concept that matters.

If you run a very popular site, you can add one or more conditions to filter your traffic (like coming from a specific URL), as shown below:

You can also set up goals to show up in your report, but I find the standard set very limited. You can compare on the basis of clicks, pageviews, or custom events that you have to set up yourself with JavaScript.
In their sales pitch, Optimizely states that you don’t have to be a coder to get it working. But if you want to do some serious tracking, some coding is still necessary. I think a more visual approach would be preferred. Coding shouldn’t be necessary after you’ve already placed their tracking script in your header.
Let’s say you want to compare different positions for a newsletter signup box, measuring the signup rate for each position. With Optimizely, you will have to insert some JavaScript code behind your form in addition to the tracking script. This seems redundant.
What about the results from my test? Well, this is the report I was shown.

The results are still being collected, but this doesn’t really tell me anything since it isn’t clearly defined what these conversions are (probably clicks). This is just too limited.
Optimizely allows for a lot of integration with heatmap tools like CrazyEgg and ClickTale, and with various analytics tools (including GA). I think at these pricing levels at least some of that functionality should be included out of the box.
Optimizely looks impressive on paper and can certainly be a helpful tool if you don’t mind coding things. But I wish it was more click-and-play so you can stay focused on the testing.
Optimizely offers a free 30-day trial.

Visual Website Optimizer

Now let’s look at Visual Website Optimizer (VWO), which claims you won’t need to code at all. Their pricing is quite steep, but there’s a free trial. VWO begins by taking you through the steps needed to set up your experiment. After that you are given the code to insert, but you can also use plugins for most popular systems like WordPress and Magento.
I made the same change to my AdSense link units as with the Optimizely test by editing the HTML. There are a few more ways to modify your page. The easiest way for my example would be using the “Rearrange” function, but that wasn’t working with the AdSense code.

After selecting and changing the HTML, I need to choose what I wanted to test. As you can see, I can be very specific about what to track when it comes to clicks on a link (if the test code allows for it). And there is a way to track signup forms. As with Optimizely, you can filter your traffic based on a whole range of conditions.


A nice feature of VWO is that it comes with an integrated heatmap, thought it’s not as advanced as a dedicated heatmap tool. Too bad you can’t compare both heatmaps in your A/B tests. And they aren’t very advanced either, since the coloring doesn’t really tell you much. But it’s a nice idea that could improve with a little more development.

VWO comes with a summary report and a detailed report, but to be honest these should be integrated with each other. The detailed report just adds some graphs and filters, but nothing really in-depth. The reports you do get are enough to help you in analyzing your results and use as a basis for further actions.

You get a little more information with VWO than with Optimizely. Engagement in this case is measured by overall clicks. Unfortunately I wasn’t able to track the actual clicks on the AdSense ads in the end, due to it being in an iframe.
Visual Website Optimizer gives the click-and-play experience I was missing with Optimizely, and gives you enough ways to measure how visitors respond to changes. The heatmap functionality is nice but not very useful in it’s current form. And even though reporting is a bit more than basic, it still feels too simple.
Visual Website Optimizer offers a free 30-day trial.

Google Analytics Experiments

To conclude this tools comparison, I will have a look at Google Analytics Content Experiments (GACE), previously known as Google Website Optimizer. GACE is a free A/B testing tool that’s part of Google Analytics. It only allows for pure split-testing, and you have to make the different variations yourself (i.e. set up new pages).

To use GACE, you start by setting up your experiment and defining your objectives (which can also be predefined goals in GA, which is very helpful). You can also run experiments on Adsense results. Then you just have to define two or more testing pages (one being the original), insert some code it’s ready. You can even use your sales funnels and start split-testing these.
On one of my other sites, I have published a Liquid Web review. As you can see, there are prominent blue boxes displayed with a clear call to action. Changing colors requires you to copy the original page, and make the modifications yourself (like I did here). Also, you need to make sure your alternate page isn’t indexed by itself.
After you set up your experiment, you will immediately experience a major downside of GACE: You have to wait for 1-2 days to see results, unlike the two other tools, which are real-time. This means you can’t instantly act on the results coming in. With A/B testing it will definitely help to test over a combination of days, but if you are testing minor tweaks you’ll want to do a few variations per day so you can be sure which changes affected your conversions.
GACE doesn’t allow you to filter your traffic when setting up your experiment, you can only choose which percentage of your traffic should be included in the test.
Another (unnecessary) disadvantage of GACE is that it changes your URL to track the experiment:

While this doesn’t present major problems, it just isn’t very clean and also not very convenient when someone wants to bookmark your link (like I do a lot in Evernote).
GACE does give you the best reporting results in this comparison, because it allows you to compare based on different metrics. Even the ones you didn’t initially set up are shown, but a winner is defined on the metric you put in as objective.

The Content Experiments functionality is very well integrated in the whole GA suite, which offers you some benefits over the other tools. When it comes to the testing itself, it’s rather limited and more time consuming to set it up. It also requires you to be patient. But it gives you an easy way to start with A/B testing without spending any money. And the AdSense integration is very valuable if you run an ad-supported site.

Conclusion

In summary, even though these three products seem to offer the same functionality, the way they handle things is very different.
Optimizely gives you lots of possibilities if you don’t mind coding some stuff together, while Visual Website Optimizer is more click-and-play. VWO comes with an extra tool for heatmap tracking, which is a nice feature but too limited. You can use both tools for both basic (changing elements) and advanced (filtering) experiments. But when it comes to reporting there is still room to improve. Documentation for both products is excellent, even if you just want to be educated on the different concepts of A/B testing.
With both products, it still feels like I’m missing important information though. Let’s take visitor origin for example. I can exclude certain segments of visitors so I can tell where they are not coming from, but wouldn’t it be nice to see if UK and US visitors are responding the same way to a change, and see that in a chart? Now it requires you to run two separate tests, and use a separate analytics tool. Maybe I’m asking too much here, but I believe there’s lot of potential in these tools since they already know so much about your traffic.
Both offer a free 30-day trial, so I encourage you to run some tests yourself.
Google Analytics Content Experiments is fully integrated with the rest of the GA suite, but is rather limited in the way tests are performed. It also requires more manual actions to make the adjustments, and you need to be patient before seeing results. But once you’ve set up a nice set of goals or funnels, it’s a great (and free) tool to use.
If you’ve used any of these or know of another tool, please let us know in the comments.

Jacco Blankenspoor

Jacco Blankenspoor is a website developer from the Netherlands, and is currently involved in job sites built on WordPress. You can see him in action at his current pet sites: Heathrow Airport Jobs and Airport Jobs, a job engine built on WordPress.

Occam's Razor
by Avinash Kaushik

Digital Marketing and Analytics Blog

Experiment or Die. Five Reasons And Awesome Testing Ideas.

57 Comments | Print | PDF

"Experiment or die, there is no try."
That was my call to action, Yoda inspired, last week to a group of international C-level executives. And I meant every word of it.
There is a tendency to think experimentation and testing is optional. Ouch!
I fundamentally believe that is wrong. For a few simple reasons:
# 1 It's Not Expensive!
You can start for free with a superb tool: Google's Website Optimizer. It is packed with enough features that I have no qualms flogging it (even though I work closely with the team!).
If you want to help our economy and pay for your tools then that is absolutely fabulous. Both Offermatica and Optimost are pretty nice options.

[Just don't fall for their bashing of all other vendors or their silly claims, false, of "superiority" in terms of running 19 billion combinations of tests or the bonus feature of helping you into your underwear each morning.
You'll be lucky if you can come up with 5 combinations, and it is not that hard to put on your underwear.
Look for actionable uniqueness. For example I am quite fond of the fact that with Offermatica you can "trigger" tests based on behavior. That is nice, well worth paying for.]

# 2 Six And A Half Minutes. That's it!
Tom has tried this with many many Marketers, and its so true: If you have two different pages you want to test, it takes six and a half minutes for you to configure, test (QA) and launch a A/B test.
[Please read that literally, as it is written. You have two pages already. 6.5 mis to: Configure. QA. Launch.]
You have six and half minutes right?
I cannot recommend enough the wisdom of starting with a A/B test.
You will start fast, you will find enough problems in your company, you can show easy wins.
Aim to get to the thing vendors are selling, MVT, but start with A/B regardless of the tool you use.
# 3 Show 'em You Are Worth It.
There is a lot of pressure on all of us to prove our worth and make significant improvements to our web business.
ClickStream analysis with Omniture or Google Analytics or ClickTracks is well and good, testing will get you on the path of taking having a direct impact faster.
By the nature of it Testing is action oriented, and what better way to show the HiPPO's that you are awesome then by moving the dial on that conversion rate in two weeks?
# 4 Big Bets, Low Risks, Happy Customers.
Very few people appreciate this unique feature of testing: You have an ability to take "controlled risks".

Let's say you want to replace your home page with pictures of naked people, yes in the quest of engagement . : ) Naked people are risky, even if they are holding strategically placed Buy Now buttons.
So run a test where only 10% of the site traffic sees version B (naked people).
You have just launched something risky, yet you have controlled the risk by reducing exposure of the risky idea.
Stress this idea to your bosses, the fact that testing does not mean destroying the business by trying different ideas. You can control the risk you want to take.
# 5 Tags, CMS, Reports & Regressions: All Included!
Pretty much all Testing tools are self contained, simple to launch (A/B is brain dead easy, MultiVariate needs your brain to be awake – that's not hard is it?), they contain all reporting built in and the data is not that hard to understand.
So you don't have to worry about integrations with analytics tools, you don't have to worry about rushing to get a PhD in Statistics to interpret results and what not.
You will hear super lame arguments about mathematical purity or my factorial is better and the other guy's whatever. Ignore. It will take you a while to hit those kinds of limits. And the nice thing is by then you'll be smart enough to make up your own mind.
What's important is you start. Do that today. Think of this as dating and not a marriage. You are allowed to make mistakes. You are not going to marry the first guy you run into. Don't take that approach here.
So agree with me? This is attractive? Right?
Think about it this way. If your analytics career is flagging then testing is the Viagra you need to take.
Seriously.
: )
So as my tiny gift for you here are five experimentation and testing ideas for you. I'll try to go beyond the normal stuff you hear at other sources.
# 1 Fix The Biggest Loser, Landing Page. (& Be Bold.)
Now all that is well and good. But the sad thing in a common mistake people make is get excited and then go try to test Add To Cart buttons. Or three different hero images on the home page.
That's all well and good. But honestly that's not going to rock your boat. [Remember you are on Viagra!]
For your first test be bold, try something radical, bet big. I know that sounds crazy. But remember you can control risk.
If you start with a A/B test with some substantial difference then you can show value of testing faster because you'll get a signal faster, you'll start the emotional change required to embrace testing across the organization.
My favorite place to start, is the Top Landing Pages report (or Top Entry Pages if that's what your vendor calls it) from your web analytics tool.
Find the biggest loser, the one with the highest bounce rate.

Click and look at the sites sending traffic to this page, look at the keywords driving traffic to this page. That will give you clues about customer intent (where people come from, and why).
Come up with two different (bold) ways to represent that page and deliver on that customer intent.
Your first A / B / C test.
# 2 Test a Single Page vs. Multi Page Checkout.
One of the highest ways to improve conversion is to reduce Cart & Checkout Abandonment rates. Take money from people who want to give you money!
Some websites have a one page checkout process: Shipping, billing, review and submit.
Some have it on four pages.
I have seen both work, you never know, it really depends on the types of visitors you attract.

So if you have a single page why not try the multi (if your abandonment rate is high, say more than 20% :). Or vise a versa?
I have seen very solid improvements in these tests.
Or here's a bonus. Many shopping cart (or basket to my British friends) pages have a Apply Coupon Code box. This seems to case people to open Google and search for codes. So why not move this coupon code box to the Review Before Submit page?
It won't send those who don't have a coupon code looking for one, and by the Review Order page they are way too committed. For those that have a coupon code they can still apply it.
In both these scenarios you are helping your organization find value quickly by touching a high impact area.
And remember, this works for lead submission forms and other such delights.
# 3 Optimize the Number of Ads & Layout of Ads.
Ad supported sites are numerous. And the there is so little restraint, the core idea seems to be let's slap as many ads on the site as we can.
More ads = more clicks = more revenue.
Usually this is never tested.

[I can't read Norwegian so this could be wrong, but I counted a total of 19 ads on this page! Ten above the fold. Important point: American sites are just the same.]
So test the number of ads you should have on a page. Its not that hard. It can be a simple A/B test or a MultiVariate test.
In a memorable test the client actually reduced the number of ads on the page by 25% and the outcomes improved by 40%. I kid you not, 40%. And guess in which version customers were happier.
There is a built in assumption there that you are simply not selling impression, in which case pile on the ads in the pages. You are not being held accountable for outcomes so enjoy the ad party.
Here's a bonus idea.
There are sites were the ad is in the header, it takes up the whole header and is the first thing that loads. I have only seen one case where that worked.

The header takes up 30% of the space above the fold on a 1024 resolution.
So if that is you why not try a test with the header ad and without? See which one improves overall conversion / outcomes?
The other bonus idea is to try different ad layouts. Most people have banner blindness, top of the page and in the middle of the content (as in Yahoo news).
Why not try different layouts and formats? If not to see which one works the best then to just annoy your customers? :)
# 4 Test Different Prices / Selling Tactics.
You can of course test different pretty images, why not try to reinvent your business model using testing?
A company was selling just four products. But the environment got tough, the competitors got competitive. How to fight back? Some "genius" in the company had an idea "Why don't we give our cheapest product, currently $15, away for free?"
CMO says: Radical idea. CEO says: Are you insane? CFO says: No way!
Now it did present a fundamental challenge, no one like to give revenue up. And people worried about how successful it would be, what would be the revenue impact, why would anyone buy a non-free version etc etc.
Rather than create prediction models (with faulty assumptions!) or giving up in face of the HiPPO pressure, the Analytics team just launched a A/B test. And they controlled for risk (after all the CFO did not want to go bankrupt) by doing a 95% control and 5% version A test.

Perhaps unsurprisingly the free version of the product sold lots of copies.
That was not surprising.
What was surprising was that free helped shift the sku mix in a statistically significant way, i.e the presence of free caused more people to buy the more expensive options. Interesting. [In a delightfully revenue impacting way!]
The other positive side effect was to cause lots of new customers to be introduced to the franchise, as they "purchased" the free version. Lovely.
Here are some bonus ideas.
If you give discounts try 15% off vs $10 off (people tend go for the latter! :)).
Try $25 mail in rebate vs $7 instant rebate (or change amounts to suit).
You get the idea.
# 5 Test Box Layouts, DVD Covers, Offline Stuff.
Let's say you are launching a new product or a dvd or something similar. You want to figure out what layout might be more appealing to people in stores.
You could ask your mom to pick a version she likes.
You could ask your agency to ask a few people.
Or you could launch a test online and see which version is rated highest by your website visitors!
I have done tests for DVD covers and the results were surprising.
Or here's another idea…
You are a multi channel customer. You sell bikinis. Now you want to sell Accounting Software. Why not try it on your website before you reconfigure your stores?
Or you are Wal-Mart and it is expensive and takes a long time for you to put new products in your stores. That makes it risky to start stocking the "on paper hideous but perhaps weirdly appealing" Zebra Print Occasional Chairs in your store. What if it bombs?
Well why not add it to your site, see if it sells. If it gets 15 positive customer reviews (!!), then you know you have a winner on your hands.
The actual launch process is faster, you can reduce risk, and you don't have to rely on just your company employees (the fashion mavens) from picking winners and losers.
All done.
I hope that you'll find both compelling reasons for starting experimentation and I have managed to stretch your mind beyond "honey let's start testing shopping cart buttons".
There is so much you can do. This recession season buy your CEO the gift that keeps giving, a experimentation and testing tool.
Here's a summary for you. . . .
Five reasons for online Experimentation & Testing:

#1 It’s Not Expensive!
#2 Six And A Half Minutes. That’s it!
#3 Show ‘em You Are Worth It.
#4 Big Bets, Low Risks, Happy Customers.
#5 Tags, CMS, Reports & Regressions: All Included!

Five off the beaten track Experimentation & Testing ideas:

#1 Fix The Biggest Loser, Landing Page. (& Be Bold.)
#2 Test a Single Page vs. Multi Page Checkout.
#3 Optimize the Number of Ads & Layout of Ads.
#4 Test Different Prices / Selling Tactics.
#5 Test Box Layouts, DVD Covers, Offline Stuff.

Ok now its your turn.
What are the reasons your company is not jumping on the awesome testing bandwagon? If it did, what finally convinced them? If you are doing testing, care to share some of your ideas? Anything off the beaten path you have tried? Any massive failures?
Please share your feedback, insights and stories.
Thank you.
PS:
Couple other related posts you might find interesting:

• Experimentation and Testing: A Primer
• Build A Great Web Experimentation & Testing Program

Like this post? Share it:

Best Metrics For Digital Marketing: Rock Your Own And Rent Strategies.

Occam's Razor

by Avinash Kaushik.

Best Metrics For Digital Marketing: Rock Your Own And Rent Strategies

46 Comments | Print | PDF

In a world of infinite choice, the ability to pick critical few metrics to focus on is, well…, critical. It is the difference between plodding along, or winning big.
But choosing what to focus on is extremely hard.
You have to have a deeper understanding of the business, an expansive knowledge of what is possible (and not just inside Google/Adobe Analytics), and, this is so, so, so important, the ability to balance the now and the near future.
We all, myself included, fail so often because of the difficulty inherent in those three elements. In many cases we don't have sufficient knowledge for one and three, even if we are ok with two. And that's not because we stink. It is because often we are just the data girls/boys.
I recently had an opportunity to recommend to a group of CEOs everything they should measure for everything they should do with digital. You can imagine, in a world of infinite choice (!), this is a non-trivial challenge. I wanted the end product to be a handy-dandy list they can use to check if their employees are focused on the right things.
In this post I want to share that one-page list of the best metrics for digital content, marketing and business success with you. I hope the metrics will be helpful, but more than that I hope that the thinking behind whittling down from the infinite to the finite will be of value. If you get the think, you can definitely go out and create your own list of the critical few.
Ready to rock and roll?
Context #1: A Reminder About the Importance of ABO.
I believe in always focusing on the complete customer-business journey. In earlier posts on this blog, I've simply defined the journey as Acquisition, Behavior and Outcomes.
Each stage focuses on a unique facet of the digital business…

Are we able to efficiently find the right people and influence them? Can we deliver an incredible experience that blows the mind of our current and future customers with its sheer beauty, intelligence and simplicity? Do the customer and the company both end up with a win-win (they get joy and task completion, we get some current revenue and possibly a long term repeat customer)?
Most of the time when I look at the dashboards and reporting efforts in companies, they are usually obsessed with one of these pieces or maybe even a couple. Rarely is there an insane obsession with all three. Some of this is because we are organized in silos, no incentive to think end-to-end. At other times it is because of a lack of skill or the ability to analyze the complete journey.
Focus on all three with everything you do with digital analytics. Strive for a a balance across all three in terms of people, processes, and analytical focus.
One good way to accomplish that is to ensure you have an optimal org design, and that your Digital Marketing and Measurement Model exemplifies this balance.
Context #2: Own + Rent = Great Digital Success
I believe that if you want to win in digital, with your ABO strategy, you have to do Own really well and you have to do Rent really well.
[Bonus reading: Rent or Own? The Optimal Digital Investment Strategy]
Your primary owned channel is your website, mobile or desktop. It could also include your blog, your recommendation engines, your customer forums, and all kinds of other digital existences that you own outright. You own the domain. You own the customer relationships. You own decisions around content, creativity and evolution. You. Everything.

Your primary rented channels, at the moment, are your social existences. On Instagram, Pintrest, Google+, Vimeo, Weibo, Vkontakte and others. You are able to contribute content on these channels, you are able to engage with the audience, you are able to control the conversation/your presence a bit. But you don't own anything else (data, relationship, creativity, etc. etc.).
If you want to win on the web, you have to do both Own and Rent really well. You pretty much don't have a choice. But if you are going to suck at something, sucking at rent is preferable.
Best Digital Metrics: Own Existences/Strategies.
Close your eyes and just imagine the number of metrics you can use to measure the success of your marketing efforts. 25? 50? 68? More?
It would not surprise me. We have so many, and it is not that they are terrible. It's just that there are so many and I set myself the challenge to just focus on two metrics each for acquisition, behavior and outcomes. Just two.
Can you feel my pain? How can we just pick two? #omg
For acquisition my goal was to pick two metrics that would incentivize our teams to behave in the most optimal manner. I abandoned CPC (I hate that metric, it incentivizes short-term thinking of the worst kind). I even abandoned CPA (not enough focus on what makes the ads good). And many others.
I settled on Clicks and Assists.
Clicks because in this case I'm more obsessed about performance marketing. Clicks will get you and I to focus on ad copy, bids, other creative elements, precise targeting (or retargeting), ad placement (quality score for search!), and so many other things that go into making sure that our ad does not simply show up to the party but also entices a physical reaction (Click!).
Assists because most marketers continue to exhibit suicidal tendencies by sticking with last-click attribution. The best way to stink at digital today is to do last-click attribution (and the analytics vendors don't help, last-click is default… they want us to go out of business!).
I want you to focus on % Assists as your standard metric for marketing success (best place to find it is in the Google Analytics MCF Assisted Conversions report). Reward campaigns (owned, earned or paid) that deliver the person who converts right away AND the campaigns who deliver the people who convert after a few days. Assists incentivizes the right behavior by your marketers.
[Bonus read: Multi-Channel Attribution Modeling: The Good, Bad and Ugly Models]
For Behavior, I picked Bounce Rate and Visitor Loyalty.
Bounce Rate to encourage an obsession with landing pages. Far too often with our ads we write checks that our site can't cash. Far too often the Marketer's are paid on ad metrics and not what happens on the site. I want to fix that. Hence Bounce Rate. Create great landing pages, ensure there is clear continuity between your ad, link, email campaign, social contribution, press release, and the place you are sending the traffic to.

Visitor Loyalty because I've grown less fond of "repeat visits" with the passage of time. Repeat visits is too low of a bar. Visitor Loyalty empowers a focus on one person and the quest to impress them so much that they want to come to your site (content or ecommerce) again and again and again.
Best practice: Look at your Visitor Loyalty report, pick the number that is most optimal for your business (say, more than 5 times in 30 days) and set that as your target. Now measure like crazy!
I hope you noticed that the behavior I'm encouraging with Visitor Loyalty is the reason why % Assists is such an important metric as well.
You get people to stay for more than one page during their first visit… and get them to visit your site multiple times… how can you not win!
[Bonus read: See-Think-Do: A Content, Marketing, Measurement Business Framework]
Finally for Outcomes, I choose my absolute favorites: Macro and Micro-Outcomes.
The best websites in the world have a portfolio of outcomes (in case your customers are not ready to jump into bed with you right away!). Some of these outcomes add value to the business in the next 30 days, some over 90 days and some over nine months. The portfolio empowers you to solve for every type of customer, and let them move at the pace they are most comfortable with.
You are not going to be successful if your owned digital existence does not have a portfolio of outcomes, here's an example for a casino/destination…

I do find it a smidgen irritating that so many Analysts and Marketers focus on single-session conversions for just one outcome (order or lead or something else). C'est tres terrible!
Macro-Outcomes help you measure immediate success, over 30 days. Notice even this is a small collection and not just one conversion.
Micro-Outcomes ensure that there is long term thinking in your company, the best kind of thinking, and that your employees undertake efforts to drive business value from every visit to the site.
Six simple critical few metrics that balance for the short and the long term, that incentivize the right behavior by our peers and agencies. And all standard metrics in any decent web analytics tool.
[Bonus read: Excellent Analytics Tip #24: Obsess About Real Business Profitability]
Let's focus on the other owned asset we have as a company, mobile.
If you've been to one of my keynotes recently you might have heard me recommend a mobile only strategy, and not a mobile first strategy. We have so much progress to make in that area. With mobile becoming the first (or even primary) digital touch-point for our customers, we have to have an insane amount of focus in this area.
[If you are short on ideas for mobile greatness: Mobile Marketing 2015: Rethink Customer Acquisition, Intent Targeting.]
For mobile Acquisition, our critical few metrics stay the same. Clicks and Assists.
Commercial user behavior on mobile is still transient rather than persistent, so you might initially get a lot more value with a focus on Clicks. After that, having thought long and hard about this choice, I still recommend keeping Assists. One day when cross-device tracking/analytics is easier (it is way too hard today), you'll be ready.
For Behavior, I have Bounce Rate and Time Spent.
Bounce Rate is even more important for Mobile, the back button is hard coded into the mobile device! (At least for Android.) You stink and they'll be gone in 0.05 seconds.
Again, because of the transient nature of mobile behavior, I made a switch from Visitor Loyalty to Time Spent. In most cases people will visit your site for a specific purpose (even more so on phones than tablets), create experiences that answer questions and get them to spend the time it takes them to have a successful visit.
The exception is a content site (my beloved New Yorker as an example). For their mobile site, I would keep Visitor Loyalty.

And one more time, because of current user behavior, and continued difficulty to transact on mobile devices, my recommendation is to focus on See and Think Micro-Outcomes.
A gigantic reason for under-investment in mobile innovation is the belief by businesses that they can't make money fast ("look, low conversions!"). That is a big mistake. Hence my recommendation to focus on Micro-Outcomes. Those metrics will incentivize the right behavior.
Though we segment Own into two pieces, you'll notice we only ended up with just seven metrics. Focus FTW!
Now let's go see what we can do for Rent.
Best Digital Metrics: Rent Existences/Strategies.
Our goals, participation options and desired outcomes all change for our Rent existences. Much less conversion driven, much more brand marketing driven, and much more longer term focused in terms of impact and outcome.
Most businesses and Marketers have sub-optimal results from their Rent efforts when they are primarily conversion driven, primarily focused on performance marketing and primarily looking for short-term impact.
In choosing the metrics, those elements were top of mind for me. (And I encourage you to keep that strategic view and purpose in mind as you choose your own.)
I've broken Rent channels into permanent and temporary. YouTube is an example of a permanent outpost. This is because for most videos uploaded on YouTube often over 70% of the views will come 30 days after you upload them. Your presence and contributions have a long life. While on temporary outposts like Google+ or Facebook your content, by most measures, "lives" for less than 60 mins – with a small trail of activity for 24 hours and then poof (!) it's gone. For Twitter, it is much, much, shorter than 60 mins/24 hrs.
Yes, on permanent and temporary channels your contributions (text, links, videos, contests, pimping) are always there. It is good for a little SEO. It does have other value (like people randomly bumping into your Twitter page). But almost all engagements your business will have, if any, will be on the latest content. Keep that in mind as you choose what to focus on.
For Acquisition, advertising you buy on YouTube, the metrics to use would be Impressions and Click-thru Rate.
Impressions because in this case a big objective is to solve for brand marketing, so getting a high share of voice is important. (Doing TV on YouTube!)
Click-thru Rate to give a tiny amount of incentive to extend the conversation beyond just the initial impression. CTR will encourage a focus on any commercial purpose of the ad, and creation of a great experience (on YouTube or your Owned channels).

For Behavior, we focus on Views and Completes.
Views because the primary front of our existence are our delightful non-boring videos. We want to encourage consumption of that content to enhance brand perception and move the brand equity up a few points.
But Views by themselves can drive sub-optimal behavior in Marketers. We counter-balance that with Completes . It will encourage a focus on video content, its ability to retain attention and, if we rock, will drive the outcome metrics.
For Outcomes, YouTube is not focused on conversions/short-term outcomes. We focus on driving organic Views (and Complete!) via Social Amplification of content. There is nothing that works better for brand marketing then getting other people to take your content, slap their own name on it and help you reach their friends/lovers/followers – an audience you might not otherwise reach.
And we focus on one of the most precious thing you can get from an audience on any rent channel: Attention! In this case measured via the growth in Subscribers . In study after study we have seen that your Subscribers are the ones who view your content on YouTube first, and their ability to help you out with the initial social amplification is the difference between a pin dropping in a forest or lighting the initial flames of, wait for it…. wait for it…. wait…. viralness!
Subscribers are your biggest fans, you earn more of them, your life becomes exponentially easier over time because you don't have to keep renting audiences via advertising (though please buy advertising, people who work at YouTube have bills :)).
[Bonus Read: Only three things work on YouTube, learn more about them... YouTube Marketing And Analytics: A Primer For Magnificent Success]
Home stretch, measuring our Temporary rent existence/strategies.
This is a bit harder because while there is some commercial intent we can tap into (regardless of if we are a news site or an ecommerce/lead gen site), it is extremely difficult to grab attention (very short half-life of your contributions). You have to deal with things like EdgeRank, or the faster-flowing-everyday river that is your customer's Twitter stream.
I must have mentioned somewhere in this post that picking your critical-few is hard work!
For Acquisition, advertising we buy on Twitter, Facebook etc, my choices were Impressions and Likes/+1s.
Impressions, see above section on our permanent presence.
Likes/+1s is interesting, is it not? This was really hard. Why would you advertise on your temporary social channels? It turns out there is almost no commercial intent (or at least 43 times lower than email campaigns, see this graph – and click Follow!). My recommendation is to focus on getting outcomes on your social presence. Measure Likes/+1s at a post/brand page level. Pure brand marketing with a pinch of engagement on your rent channel.

For Behavior, we are going to focus on Amplification and Applause.
Amplification Rate measuring the number of Shares (Retweets, Forwards etc) on each post. Our goal is to create content that allows you to reach your second-level network. And when people amplify your content, they do so with their own brand endorsement. #awesome
Applause Rate measures the number of Likes (or +1s, Favorite clicks) on each post. It is very hard to know when you are creating content (text, links, videos, images etc) of value. Applause rate is your audience telling you what they like more and what they like less. Focus on the more content and not on the less content. :)
For Outcomes, we are going to focus on Conversation and Owned Micro-Outcomes.
Conversation Rate measures the number of audience comments (or replies) on each post. We are renting these spaces, creating amazing content, doing great brand marketing while participating in and initiating great conversations. So, let's just measure that. Do we create content that creates conversation? If not, why are we on social channels? Great outcome to measure.
Micro-Outcomes measures the outcomes that deliver value to your business over 90 days and over nine months. (For more, please see the Own section above.) This should incentivize our teams to focus on creating long term value on temporary rent channels.
[Bonus read: Best Social Media Metrics: Conversation, Amplification, Applause, Economic Value]
[Bonus read: Facebook Advertising / Marketing: Best Metrics, ROI, Business Value]
If you want to measure the social metrics mentioned above automatically, including Micro-Outcomes, you can do so using the True Social Metrics tool.
Best Digital Metrics: Comprehensive View.
Here's the complete picture from my keynote, the best digital metrics to measure the effectiveness of our ability to find the optimal customers (current, future), influence them and deliver win-win outcomes…

While it looks like a lot, I've recommended just fifteen metrics to measure everything you could possibly to with digital for your business. Not bad, right?
Please click on the image above to see a higher resolution version, or if you want to download it.
Closing Thoughts.
We are blessed with Big BIg BIG data, and many, Many, MAny, MANY metrics! It is our ability to identify the critical few spectacularly relevant metrics to focus on that will bring personal glory to us and professional glory to our business. (Mostly it is about personal. :))
In choosing metrics you have to make trade-offs, some are painful. More than learning about the best metrics available to us, I hope you've learned about the thought process behind each choice. Why something was chosen? What behavior it will drive in our teams/clients?
And one more thing… please remember that this is a journey and not a destination. The Web Metrics Lifecycle Process is Define > Measure > Analyze > Action > Improve or Eliminate. Rinse and repeat. Make sure you remember that (and follow the process!) before you spend so much time in discussion and choosing that the optimal time to deliver insights has passed you by.
Good luck!
As always, it is your turn now.
Does your current dashboard optimally cover Acquisition, Behavior and Outcomes? When discussing a framework to ensure you have the right metrics, does your company use the Own and Rent philosophy? Would you recommend a different set of metrics for ABO for mobile? What do you use to measure the success of your YouTube efforts? How about the temporary social channels like Google+ or Facebook? Is there a metric you consider your absolute BFF that is missing from the choices above?
Please share your advice, insights, questions, critique, ah-ha moments via comments below.
Thank you.

Like this post? Share it:

10 Email Marketing Campaigns Analyzed to Increase Email Conversions

Chris Hexton
Vero Blog

10 Email Marketing Campaigns Analyzed to Increase Email Conversions

April 2013 

 

It’s time for some email marketing examples from the real world.
I’ve selected 10 emails that came through my inbox recently and am going to focus on conversions and how each campaign performs.
For each of the campaigns listed I’ve pulled together a screenshot as it arrived in my inbox, a list of things the campaign does well and a series of ideas or hypotheses the creators could test in order to improve conversions.
These examples should give you some inspiration for your next campaign.
Each of these campaigns was sent to me publicly. No email is perfect and there are always things to test. The best way to get better is to try and try again!

1. KISSmetrics’ blog update

Thoughts

This email is short, sweet and gets to the point. It’s a good email: it gives enough of a flavour of the blog post to leave you wanting more.
There are lots of reasons this is a good campaign yet it also feels a little ‘cold’. The KISSmetrics blog is full of genuine, quality posts and this blog update email doesn’t to a great job of conveying this.
The primary call to action (to read the post) could also be bolder: it is overpowered by the call to action to sign up to KISSmetrics.

Testing ideas

1. Add a button: Try making the “Click to continue reading…” a button.
2. Highlight the blog post: Make the font-size of the heading and the blog post text larger than the font-size of the surrounding links by a few points.
3. Conflicting CTAs: Move the ‘email us any time’ into the footer – right now there are really three competing calls to action in the middle of this campaign.
4. Include an image: Include an image from the post (if there is one). Many of the posts on the KISSmetrics blog feature great images and examples so why not include one of these in the email?

Site: KISSmetrics

2. DealsDirect

Thoughts

As far as ‘sale’ emails go this is a fantastic one.
Usually they are cluttered, mention 1,000 products in the one email and, as such, have no clear call to action.
This campaign does focus on a primary CTA (“Shop Now”) and ensures the entire image is clickable. You can’t see it up above but the ‘alt tags’ on their images are superb: the email is navigable even without loading the images.
Finally, I really like the use of the social call to action under the “Shop Now” link.

Testing ideas

1. Target the iPhone CTA: I’ve never opened a DealsDirect email on my iPhone so it looks as though the bottom link is being shown to everyone. It would be interesting to test only showing the ‘Download the iPhone app’ to primarily iPhone users. Why? This would obviously increase the CTR for that particular link (more targeted) and may also increase the CTR on the primary call to action to “Shop Now” by removing the iPhone app link for irrelevant users. Here’s a great case study where this held true:
2. Remove the second CTA: Remove the secondary CTA (the link to the rugs). Would the overall effect on sales be bigger if this were removed? There is a possibility this secondary CTA is affecting overall CTRs – but, then again, it might mean the total sales from this one email are higher.
3. Smaller footer: Tidy up the footer. There is a lot of relevant information in the footer but it is a bit cluttered / long-winded. Remove some of the clutter!

Site: DealsDirect

3. Scottevest

Thoughts

Every day Scottevest do a sale. I like that the email shares the technical aspects of the clothing, exactly what Scottevest is known for.
The header, the body image and the footer link all point to the daily sale page, meaning there is less ‘link leak’ as customers can click-through to the place you want them to see.

Testing ideas

1. No real CTA: A big red button would be much better than the subtle ‘Daily Sale’ link at the end of the email. Sure, it’s hard not to click something in this email but we all love a good button – it’s UX 101 – so why not include one!
2. Whitespace: There are only two paragraphs but they look daunting: they’re chunky. Despite telling a good story I’d shorten or split up these paragraphs. It might also make sense to include some of the images up top throughout the body, to break things up. Using bold and italics goes a long way too.
3. Make the header smaller: Get straight into the action! Regulars will see this email every day, there isn’t a need to have such a large ‘branding’ section at the top of each email. Get to the content.

Site: Scottevest

4. OkCupid

Thoughts

It’s pretty hard to critique this campaign. It is a simple automated email and it’s purpose and call to action are very clear. The includsion of the re-assuring pionts at the bottom of the email is an excellent idea. This ‘Question and Answer‘ format has worked wonders for people in the past.
Including the face of the girl/guy in question is always a solid conversion strategy: people relate to people. Making the face, name and button clickable is also a solid move.

Testing Ideas

1. Button colour: OkCupid has very strong branding colours (pink, blue and light blue) but it’d be worth testing the colours of this button. Would green work better than blue?
2. Re-iterate the CTA: Make “Go get ‘em” clickable at the bottom of re-iterate the CTA by including a second button / some other link.
3. Language on the button: Perhaps ‘Check her out’ doesn’t resonate with everyone. This could even be inferred from the questions each user has answered. By building individual profiles based on what answers customers have provided the CTA could be changed on a customer-by-customer basis. This would be very powerful.

Site: OkCupid

5. Slideshare

Thoughts

Like the OkCupid email I imagine this email is effective at increasing engagement for Slideshare as it is triggered by a distinct event and has a solid goal: to get customers to follow other customers.
It’s short, simple, to the point and the inclusion of the face / display picture is, again, effective. Rock on Slideshare and trigger-based emails!

Testing Ideas

1. That button: Slideshare could test both the size and colour of the button. They could also test the copy, e.g. ‘Follow Fernando’ or ‘Follow them back’. Follow is logical copy but encouraging click-throughs by highlighting the benefits or influencing psychology is always worth testing. In this case, including the name or suggesting reciprocation would play on our guilt!
2. Clean up the template: The spacing on the template is a little weird as is the inclusion of ‘The Slideshare Team’. Sometimes the inclusion of a footer that signs off ‘The Team at XYZ’ can in fact seem colder than not including one at all. This can have a negative impact.
3. Make the benefit MORE obvious: Why should I follow Fernando back? Has he done something for me? Is his content good? The primary motivator in this instance is likely to be ‘guilt’ – i.e. wanting to return the favour. Another example might be highlighting his followers. If Fernando has 1,000 followers it’s highly likely I will want to return the favor: this is someone with influence!
4. Secondary CTA: I’m never a big fan of multiple calls to action but the inclusion of the very small ‘Follow your LinkedIn contacts on Slideshare now’ is an extremely interesting idea. This is actually a very powerful feature for Slideshare / LinkedIn. They should consider testing this CTA as the primary CTA in different versions of the email.

Site: Slideshare

6. Warby Parker

Thoughts

A smooth email from the gang at Warby Parker. Lots done well here: a clear call to action, individual links to both male and female glasses / ranges, repeated CTA, a simple footer and a beautiful, on-brand image at the top. THe template is very clear an structured as well. On the whole it’s hard to suggest too much here.

Testing Ideas

1. Male vs. Female: Having two separate versions of this campaign could lift click-throughs. Although there are male and female versions of each pair of frames WP could test the images (colours) shown and the links would be more direct.
2. Move the ‘View in browser’ link: Something across the board, having this link at the top can be useful but can also lead to this, at least in Gmail:…not always ideal, it’s worth considering if the trade-off is worth it for you.
3. The image map is not ideal: Due to the way they’ve used the images in this email, when images are turned off you see the following:…this could be avoided if individual images were used and ALT tags were applied. In this way the user has a chance to understand what the email is about even without enabling images.

Site: Warby Parker

7. Canon

Thoughts

On the whole this email feels a bit ‘salesy’. The colours, the quality of the image and the style of the product image all feel like something fresh out of a brochure.
There are positives here too. The email has one, clear call to action, you can click all of the images as well as the CTA button at the bottom. I also like that the footer is kept reasonably clean by linking to the T&Cs rather than trying to cram the contest conditions into the footer (though this does depend on your local laws).
Having an image of what I can win is also very clear and a great way to display the benefits of entering, always important when trying to get customers to engage.

Testing Ideas

1. False personalization: I’m all for personalization, on many fronts, but using ‘Chris Hexton’ (first and last names) feels wrong and is clearly false personalization. This articles gets into detail on better ways to personalize emails using event-based rules to give you ideas outside of using your customers’ names!
2. Add urgency: It’s great that Canon include that the deal ‘ends by midnight’ but they don’t make a big enough deal of it! Have some HUGE numbers saying “XX Hours To Go” or similar. Include this at the top and the bottom.
3. The image at the top: Experimenting with the image used at the top might be worth while. Including a smaller version of the product could still get across the prize in question whilst including an image with people might lead to a more relatable campaign and, in turn, higher click throughs.

Site: Canon

8. Hipmunk

Thoughts

Hipmunk have a great brand, so it’s great to see them using their mascot in the title. This familiarity is important for brand building and for driving engagement with tis email. If there was ever an implementation of ‘big buttons’, this is it. The calls to action are really clear in this email and that is something we don’t see enough. I also like that they re-link to each of the platforms when they mention the names of each social network underneath the icons.

Testing Ideas

1. An explanation: This email felt like it came ‘out of the blue’ a little. I have no real idea why I received it at this particular time, what triggered it, etc. For this reason I think the success of the email might be enhanced by providing a reason to follow Hipmunk. A contest, some announcement I want to engage with, a tip or trick, etc. Give me a reason to convert.
2. Too many options? Having a single icon such as ‘Follow us on Facebook’ might lead to a higher level of engagement. It’d be worth testing each platform (against a clearer benefit) and seeing which gets the highest response in this form of email. It would also allow them to drive engagement on each platform independently by sending more emails over time. Don’t confuse the customer.

Site: Hipmunk

9. CafePress

Thoughts

Another good example of the sort of email campaign that is usually extremely cluttered, over the top and confusing. This email is effective at conveying the 80% off sale (on top of already reduced prices), it has a good quality image featuring real people that are just like me and the colours are all very ‘friendly’.
Again, most of the images and buttons (i.e. 80% of this email) are clickable. The inclusion of the social links is also savvy.

Testing Ideas

1. What is in the footer? It feels like there is wayyyyy too much text down there. Keep it simple!
2. Make the button more distinct: The button ‘Shop Now’ could be in higher contrast with the rest of the image. Although the entire image is a link buttons really ‘anchor’ the call to action. Embrace this.
3. Two deals feels confusing: It feels confusing to have the second link and mention of ’10% off’ at the bottom. 10% off what? On top of the 80%? On top of the already reduced prices? I think this detracts from the email and goal. The same goes for Cafepress groups. It’s not a bad idea to include this, but it’s quite unclear what it is, if I’m not already familiar!
4. A/B test popular searches: The popular searches is a quality and logical set of links to include here: it provides a shortcut to the items on sale. It would be interesting to A/B test this with specific images of products or, even better, with unique HTML / product listings per-user.

Site: CafePress

10. Mixbook

Thoughts

A long-form email, these can be tricky to get right. What I really like about Mixbook’s campaign is that the template is very clear and structured and the call to action is obvious. THe big ’30% off everything’ is excellent and each of the sections clearly outlines a different product to which you can apply this offer.
The ‘gift guide’ concept is actually helpful and a great reason to use the long format. Plus there are people’s faces EVERYWHERE. This is fantastic. The use of urgency with ’5 days only’ is also clear, having used big font, though perhaps it could be a more legible color.

Testing Ideas

1. More obvious individual calls to action: The little arrows next to each product in the gift guide could be bigger, to encourage click throughs.
2. CTA text: “Get Started Now” is quite clear but could perhaps be clearer and focus more on the benefits: “Get your Mom something great” or “Make your Mom’s Day” for example.
3. Test the subject: I wonder if a subject with a higher focus on the benefits would work more effecitvely. Similar to the CTA, focus on ‘Get your Mom something awesome this Mother’s Day’ might work better than focusing on the included gift guide with ‘Mother’s Day Gift Guide’ which is a ‘feature based’.

Site: Mixbook

What do you think?

What would you test on these campaigns? What do you love?
Reviewing the work of others’ email marketing campaigns gives lots of great ideas for testing your own so always be on the lookout!