Occam's Razor
by Avinash Kaushik

Three Amazing Web Data Analyses Techniques For Analysis Ninjas

29 Comments | Print | PDF

Day in and day out we stare at standard tables and rows and convert them into smaller or scarier tables and rows and through analysis we try and move the really heavy beast called the "organization" into action.
It is hard.
This blog post has three ideas I've learned from other smart people, ideas that help surprise the "organization" with something non-normal and get it to take action. Each idea is wonderfully simple, and yet in its own sweet way makes us, Analysis Ninjas, think harder and deliver insights better.
Let's go.
Compute Actual Cost Per Acquisition Post-Facto Including Micro-Conversions.
I know that is confusing. Stay with me.
This idea, 100% of it, comes via my friend David Hughes. [He passed away recently and I miss his friendship, and our collaborations, tremendously.] From this post: Improve Search Marketing Conversion Rates through Email Registration. I'm going to redo the tables, just to make them fit the width of this blog.
David's idea is simple and genius.
Today when we measure our Cost Per Acquisition (CPA) for our campaigns (Search, Email, Affiliate, whatever), we just think of the macro-conversion and, perhaps worse, we think only of that session / visit.
Let's assume we are running www.macys.com and we got 1,000 Visitors to come to our site via a display advertising campaign. As dutiful Reporting Folks we will send this table out to reflect performance of that campaign.

$16.7 CPA might sound huge (or not depending on your margin), but on the surface it seems a lot. The flaw in this report of course is in assuming that all 1,000 visits were in play (wanted to convert / buy something). This is rarely the case.
I have repeatedly evangelized identifying all the jobs the site is trying to do (macro AND micro conversions) and then quantifying their economic value to the business. On the Macy's website of the 970 non-converting visitors, some might have signed up for a free account, some for email alerts or coupons, some opened a wedding registry etc.
If some of those 970 Visitors completed some micro-conversions, then shouldn't the CPA be on that basis rather than just the 30 orders above?
Simplifying the scenario a bit. if some of those 970 submitted an email address / signed up for price alerts and converted later then shouldn't the cost-per-acquisition include those future sales?
Say some of the Visitors did just that. What was the acquisition cost of each sign-up?

Nothing. Nice.
What will Macy's do next? Send the 100 folks the price alert they signed up for!
And what will come of it? Sales, of course.
This is a reasonable picture that will emerge.

So we got 30 orders from the original visits, and another 20 by re-targeting users via permission-based email.
What does the CPA of our original affiliate marketing campaign look like now?

A more respectable $10 compared to the original $16.7.
An immediate implication is that if at a CPA of $16.7 you were profitable, then you can communicate to your Senior Leaders that you were actually even more profitable since the final CPA is now only $10. And if you find yourself in a aggressive marketing siutation then you could even increase the bids on your display campaigns to get even more Visitors. Thanks to your clever micro-conversion and re-targeting strategy!
Lessons:

1. It is important to think in terms of micro-conversions, beyond your main objective. For the 98% of people who won't convert on your site, do you have a way of engaging them again in the future?
2. It is critical to have a robust re-targeting strategy (as in our case above). Hopefully it will be intelligent, relevant to the customers and non-torturous.
3. If you do #1 and #2 then be a dear and ensure you compute the "final CPA" of your original campaign (search or email or affiliate or social or whatever).
4. You can't do the above analysis inside Google Analytics (or even Site Catalyst or the base versions of WebTrends or CoreMetrics). You'll use Excel or a simple database (or possibly the data warehouse versions of Omniture, CoreMetrics, WebTrends).

<sidebar> Some of you might be excitedly yelling "Attribution!" at the screen. For now, just immerse yourself in the simplicity of the analysis above. I won't cover attribution here but if you have Web Analytics 2.0 jump to page 358 for my thoughts. Also remember in this case at least it was deliberate re-targeting of the initial pool of people.
</sidebar>
Command Attention, Valuable Action, By Stating Raw Numbers.
This idea comes via Kaiser Fung, from this post: Further thoughts on the Facebook business model.
In a blog post with thoughts about a graph from WebTrends, that shows click-through rates (CTRs) and cost per clicks (CPCs) on Facebook. Kaiser made this simple insight:

"What does a 0.01% CTR mean? Yes, that's 100 clicks per 1 million ads shown to Facebook users."

Let me restate that astonishing number. If your ad shows 1,000,000 times, you get 100 clicks!
And of course that's clicks, not conversions.
It caused my eyes to open wide.
That is astonishingly low.
Somehow when someone tells you "Facebook's ads CTR is 0.01%" you don't quite get it. I mean, it does not feel pathetically minuscule, as it should.
I have championed the contextual use of raw numbers to deliver insights, especially when using Averages, Percentages and Ratios. [See: Actively Avoid Insights: 4 Useful KPI Measurement Techniques]
Yet the 0.01% number did not make the impact on me it should have. And that is exactly the problem when you present conversion rates (also pathetically low on every single website on the planet) and other such metrics.
So make sure you show raw numbers.

The first number might not get your management team to take any action; it just does not evoke any feeling.
The second set of numbers might get someone to scream: WTH!
They might ask:

1. Are we showing the wrong ads on Facebook?
2. Are we using any intelligent ad targeting strategy or just randomly showing ads?
3. If we double our budget to 2,000,000 impressions is there even relevant inventory (desired demographic / users) on Facebook for us?
4. Would it be worth it?
5. Why do we suck so much? Is it us? Is it Facebook?

All really great questions — ones that you have to find answers to as a Marketer and an Analysis Ninja. Answers that will help your company improve your Facebook advertising strategy, or quit.
Lessons:

1. Big numbers are not your enemy, especially if they lead to tiny numbers.
2. Resist the urge to turn everything into an average, percentage, ratio or a compound metric.
3. At least one person will tell you that 1,000,000 impressions is great branding. Tell them that branding is measurable and you can prove them right or wrong so you don't have to take their word for it. [See: Brand Measurement: Analytics & Metrics for Branding Campaigns]

Makes sense? If not please share your thoughts using the comment box below.
Either way, remember that your job is to divert people from becoming lulled into a false sense of everything's okay. Scare them into paying attention and asking you tough questions.
Face Reality By Computing "Convert-able Audience" & "Real Conversion Rates."
This idea comes via Thomas Baekdal, from this post: Converting Traffic to Subscribers.
In it, Thomas postulates that even if you have 1,000,000 Absolute Unique Visitors to the website, that does not mean that your possibly "convert-able" audience is a million.
Some people will visit once and never again. That was not an audience that would have converted, ever. For example, the link above is to Baekdal Plus. I pay $49 per year to access that premium content because it is so good. Many of you may not want to pay for content on the web. So for Thomas, not all the Visitors from the above link are actually in play for conversion. [Though I wish they were.]
So it is imprudent to count those folks; better to only count returning Visitors.
Then, some content attracts traffic, other content actually is "valuable and will convert people into subscribers." Thomas's guidance is to only count the latter in the in play for conversion bucket.
Now you can calculate the "convert-able" audience. In Thomas's example here's how his picture looks:

(1,000,000 less the 63% one-time Visitors) less the 20% valuable traffic = 74,000.
Possible convert-able audience = 74,000.
Real audience you even have a remote chance of converting: 74k.
So small, right? After starting with a million.
I rarely see Web Analysts doing this simple exercise and educating their Senior Leadership of this harsh truth. We assume every single person who will visit www.tesco.com is there to convert. Every single person who visits www.etsy.com is there to buy. Our conversion rate calculation, Orders/Visits (bad version) or Orders/Visitors (better version), reflects that, sub optimal, mental model.
We show our leaders that we suck more than we actually do by computing conversion on the basis of All Visits (bad version) or All Visitors (better version).
If Thomas has 3,700 conversions in a month, we would normally report that as 0.37% conversion rate. [(3700/1000000)*100]
Of course, the reality is that the conversion rate was 5%. [(3700/74000)*100]
Not that 5% is orgasmically higher. But it is more reflective of the truth than 0.37%.
You would take one set of actions with 5% and a completely different set with 0.37%.
Compute your "convert-able audience." Please.
Use whatever common-sense approaches you can find.
In a post written in Nov 2006, I presented a similar thought (though in a different context than Thomas). [See: Excellent Analytics Tip #8: Measure the Real Conversion Rate & "Opportunity Pie"]
My graphics were a lot less sexy in comparison to Thomas's.

The idea was to get you to identify your "Real Conversion Rate", by identifying your "Opportunity Pie."
My recommendations were:
Throw out everyone who bounced, just for now, and also if you use log files (ohh those were the days!), then throw out "visits" by robots / junk. That gives you a rough idea of your "Opportunity Pie" (convert-able audience).
Or
If you have a qualitative survey deployed (with the three greatest survey questions ever), then throw out the percentage of Visitors who do not state their Primary Purpose as visiting your website to "buy" or "research products and services" (I generously assume we can convince the latter bucket to buy through amazing marketing on the site). So now you know just the people for sure in play and possibly in play.
This second path will also give you a great rough idea of your "Opportunity Pie" (connect-able audience).
My recommendations were different from the ones Thomas is using. But both reach for the same goal: To get you to understand that not every single Visitor will convert, and you should know, even roughly, how many are in play / convert-able.
Perhaps you'll come up with your own rules. You might throw out everyone who was there to check Order Status. Or those that logged into their account to update settings. Or those that only visited the /blog/ directory. Or the Social Media of course they will never every buy but eat our bandwidth daily digging diggers!
As long as they pass the common-sense filter, go for it. You'll be earning your Analysis Ninja chops, and delivering something extremely valuable to your management team (even if they perceive it to be a cold bucket of water on their faces, the first time).
Lessons:

1. Don't scam your Senior Management by lulling them into believing every Visitor is convert-able.
2. Ignore the standard Conversion Rate definition in Google Analytics, Omniture, WebTrends, CoreMetrics, whatever else you are using. Focus on People. (Unless your business model is that everyone must convert, and does convert, on every Visit.)
3. You might get resistance when you first present the "real conversion rate" or "convert-able audience" metrics. Worry not. Charge forward. Good will come.

After the initial shock, your Management team, if they are smart, which I am sure they are, will ask you this: "So what can we do with the majority of the traffic on our website that is not convert-able?"
Preen proud as a peacock; this is your moment of greatness. Tell them why having thoughtful micro-conversions is so important on the site. Tell them you are going to compute the micro-conversion rate for the non convert-able audience. Tell them that with some of the non convert-able audience you'll hence establish a longer term relationship: with some you'll just hope to create delight and make them your recommenders, and with others still you'll do re-targeting and use David's method (all the way up top of this post) to reduce cost-per-acquisition.
All really great business outcomes.
In a nutshell. the goal is not to abandon a majority of your traffic. The goal is not to just ignore all the bouncers (fix that, tips here: Six Tips For Improving High Bounce / Low Conversion Web Pages). The goal is not to be depressing. The goal is to face reality, give it a hug and then figure out how to kick things up several notches.
Are you Ninja enough to accept that challenge?
Of course you are. You read this blog! : )
Know that I'm rooting for you.
Okay, now it's your turn.
Does your company do re-targeting to captured email addresses? If not, why not? If yes, then do you compute real CPA? Have you computed your "convert-able audience?" Is it 100% of your website Visitors? When was the last time you used raw numbers to shove a dose of reality in front of your Senior Leaders? Are there other techniques you've used that worked?
Please share your Analysis Ninja tips with the rest of us Ninjas-in-training using the comment box below.
Thanks.

Like this post? Share it: