Occam's Razor
by Avinash Kaushik

Kill Useless Web Metrics: Apply The "Three Layers Of So What" Test

50 Comments | Print | PDF

Data, data everywhere yet nary an insight in sight.
Is that your web analytics existence?
Don't feel too bad, you share that plight with most citizens of the Web Analytics universe.
The problem? The absolutely astonishing ease with which you can get access to data!
Not to mention the near limitless potential of that data to be added, subtracted, multiplied, and divided to satiate every weird need in the world.
You see just because you can do something does not mean you should do it.
And yet we do.
Like good little Reporting Squirrels we collect and stack metrics as if preparing for an imminent ice age. Rather than being a blessing that stack becomes a burden because we live in times of bright lovely spring and nothing succeeds like being agile and nimble about what we collect, what we give up, and what we deliberately choose to ignore.
The key to true glory is making the right choices.
In this case its making right choices about the web metrics we knight and sent to the battle to come back with insights for our beloved corporation to monetize.
A very simple test can allow you to figure out if the metric you are dutifully reporting (or absolutely in love with) is gold or mud.
It is called the Three Layers of So What test. It was a part of my first book, Web Analytics: An Hour A Day.
What's this lovely test?
Simple really (occam's razor!):
Ask every web metric you report the question "so what" three times.
Each question provides an answer that in turn raises another question (a "so what" again). If at the third "so what" you don't get a recommendation for an action you should take, you have the wrong metric. Kill it.

This brutal recommendation is to force you to confront this reality: If you can't take action, some action (any action!), based on your analysis, why are you reporting data?
The purpose of the "so what" test is to undo the clutter in your life and allow you to focus on only the metrics that will help you take action. All other metrics, those that fall into the nice to know or the highly recommended or the I don't know why I am reporting this but it sounds important camp need to be sent to the farm to live our the rest of their lives!
Ready to rock it?
Let's check out how you would conduct the "so what" test with a couple of examples.
Key Performance Indicator: Percent of Repeat Visitors.
You run a report and notice a trend for this metric.
Here is how the "so what" test will work:

"The trend of repeat visitors for our website is up month to month."
So what?
"This is fantastic because it shows that we are a more sticky website now."
(At this point a true Analysis Ninjas would inquire how that conclusion was arrived at and ask for a definition of sticky, but I digress.)
So what?
"We should do more of xyz to leverage this trend." (Or yxz or zxy – a specific action based on analysis of what caused the trend to go up.)
So what?
If your answer to that last "so what" is: "I don't know… isn't that a good thing… the trend is going up… hmm… I am not sure there is anything we can do… but it is going up right?"

At this point you should cue the sound of money walking out the door.
Bottom-line: This might not be the best KPI for you.
Let me hasten to point out that there are no universal truths in the world (though some religions continue to insist!).
Perhaps when you put your % of Repeat Visitors KPI to the "so what" test you have a glorious action you can take that improves profitability. Rock on! More power to you!

Key Performance Indicator: Top Exit Pages on the Website.
[Before we go on please know that top exit pages is a different measurement than top pages that bounce.]
You have been reporting the top exit pages of your website each month, and to glean more insights you show trends for the last six months.

"These are the top exit pages on our website for the last month."
So what? They don't seem to have changed in six months.
"We should focus on these pages because they are major leakage points in our website."
So what? We have looked at this report for six months and tried to make fixes, and even after that the pages listed here have not dropped off the report.
"If we can stop visitors from leaving the website, we can keep them on our web site."
So what? Doesn't everyone have to exit on some page?

The "so what" test in this case highlights that although this metric seems to be a really good one on paper, in reality it provides no insight that you can use to drive action.
Because of the macro dynamics of this website, the content consumption pattern of visitors does not seem to change over time (this happens when a website does not have a high content turnover – like say a rapidly updating news site), and we should move on to other actionable metrics.
Here the "so what" test not only helps you focus your precious energy on the right metric, it also helps you logically walk through measurement to action.

Key Performance Indicator: Conversion Rate for Top Search Keywords.
In working closely with your search agency, or in-house team, you have produced a spreadsheet that shows the conversion rate for the top search keywords for your website.

"The conversion rate for our top 20 keywords has increased in the last three months by a statistically significant amount."
So what?
"Our pay-per-click (PPC) campaign is having a positive outcome, and we should reallocate funds to these nine keywords that show the most promise."

Okay.
That's it.
No more "so what?"
With just one question, we have a recommendation for action. This indicates that this is a great KPI and we should continue to use it for tracking.
Notice the characteristics of this good KPI:
#1: Although it uses one of the most standard metrics in the universe, conversion rate, it is applied in a very focused way – just the top search keywords. (You can do the top 10 or top 20 or as many "head keywords" as it makes sense in your case, just be aware this does not scale to the "mid" or "tail".)
#2: It is pretty clear from the first answer to "so what?" that for this KPI the analyst has segmented the data between organic and PPC. This is the other little secret: no KPI works at an aggregated level to by itself give us insights. Segmentation does that.

Key Performance Indicator: Task Completion Rate.
You are using a on-exit website survey tool like 4Q to measure my most beloved metric in whole wide world and the universe: task completion rate. (You'll see in a moment why. :)
Here's the conversation…

"Our task completion rate is down five points this month to 58%."
So what?
"Having indexed our performance against that of last quarter, each one percent drop causes a loss of $80,000 in revenue."
So what? I mean in the name of thor, what do we do!
"I have drilled down to the Primary Purpose report and most of the fall is from Visitors who were there to purchase on our website, the most likely cause is the call to action on our landing pages and a reported slowness in response when people add to cart."
Good man. Here's a bonus and let's go fix this problem.

Nice right?
Notice in this case you have a inkling to the top super absolutely unknown secret of the web analytics world: If you tie important metrics to revenue that tends get you action and a god like status.
Keep that in mind.
So that's the story of the "so what" test. A simple yet effective way of identifying the metrics that matter.
This strategy is effective with all that we do, but it is particularly effective when it comes to the normal data puke we call the "management dashboard". Apply the "so what" test and you'll make it into a Management Dashboard.
Closing Summary:
Remember, we don't want to have metrics because they are nice to have, and there are tons of those.
We want to have metrics that answer business questions and allow us to take action—do more of something or less of something or at least funnel ideas that we can test and then take action.
The "so what" test is one mechanism for identifying metrics that you should focus on or metrics that you should ditch because although they might work for others, for you they don't pass the "so what" test.
And killing metrics is not such a bad thing. After all this is the process that has been proven to work time and time again:

More here: Web Metrics Demystified.
Ok now it's your turn.
Do you have a test you apply to your web metrics? What are your strategies that have rescued you during times of duress? What do you like about the "so what" test? What don't you like about it? Do you have a metric that magnificently aced the "so what" test?
Please share your comments, feedback and life lessons via comments.
Thanks.
PS:
Couple other related posts you might find interesting:

• Six Web Metrics / KPI's To Die For
• 4 "Useless" KPI Measurement Techniques
• Five Rules for High Impact Web Analytics Dashboards

Like this post? Share it:

Best Metrics For Digital Marketing: Rock Your Own And Rent Strategies.

Occam's Razor

by Avinash Kaushik.

Best Metrics For Digital Marketing: Rock Your Own And Rent Strategies

46 Comments | Print | PDF

In a world of infinite choice, the ability to pick critical few metrics to focus on is, well…, critical. It is the difference between plodding along, or winning big.
But choosing what to focus on is extremely hard.
You have to have a deeper understanding of the business, an expansive knowledge of what is possible (and not just inside Google/Adobe Analytics), and, this is so, so, so important, the ability to balance the now and the near future.
We all, myself included, fail so often because of the difficulty inherent in those three elements. In many cases we don't have sufficient knowledge for one and three, even if we are ok with two. And that's not because we stink. It is because often we are just the data girls/boys.
I recently had an opportunity to recommend to a group of CEOs everything they should measure for everything they should do with digital. You can imagine, in a world of infinite choice (!), this is a non-trivial challenge. I wanted the end product to be a handy-dandy list they can use to check if their employees are focused on the right things.
In this post I want to share that one-page list of the best metrics for digital content, marketing and business success with you. I hope the metrics will be helpful, but more than that I hope that the thinking behind whittling down from the infinite to the finite will be of value. If you get the think, you can definitely go out and create your own list of the critical few.
Ready to rock and roll?
Context #1: A Reminder About the Importance of ABO.
I believe in always focusing on the complete customer-business journey. In earlier posts on this blog, I've simply defined the journey as Acquisition, Behavior and Outcomes.
Each stage focuses on a unique facet of the digital business…

Are we able to efficiently find the right people and influence them? Can we deliver an incredible experience that blows the mind of our current and future customers with its sheer beauty, intelligence and simplicity? Do the customer and the company both end up with a win-win (they get joy and task completion, we get some current revenue and possibly a long term repeat customer)?
Most of the time when I look at the dashboards and reporting efforts in companies, they are usually obsessed with one of these pieces or maybe even a couple. Rarely is there an insane obsession with all three. Some of this is because we are organized in silos, no incentive to think end-to-end. At other times it is because of a lack of skill or the ability to analyze the complete journey.
Focus on all three with everything you do with digital analytics. Strive for a a balance across all three in terms of people, processes, and analytical focus.
One good way to accomplish that is to ensure you have an optimal org design, and that your Digital Marketing and Measurement Model exemplifies this balance.
Context #2: Own + Rent = Great Digital Success
I believe that if you want to win in digital, with your ABO strategy, you have to do Own really well and you have to do Rent really well.
[Bonus reading: Rent or Own? The Optimal Digital Investment Strategy]
Your primary owned channel is your website, mobile or desktop. It could also include your blog, your recommendation engines, your customer forums, and all kinds of other digital existences that you own outright. You own the domain. You own the customer relationships. You own decisions around content, creativity and evolution. You. Everything.

Your primary rented channels, at the moment, are your social existences. On Instagram, Pintrest, Google+, Vimeo, Weibo, Vkontakte and others. You are able to contribute content on these channels, you are able to engage with the audience, you are able to control the conversation/your presence a bit. But you don't own anything else (data, relationship, creativity, etc. etc.).
If you want to win on the web, you have to do both Own and Rent really well. You pretty much don't have a choice. But if you are going to suck at something, sucking at rent is preferable.
Best Digital Metrics: Own Existences/Strategies.
Close your eyes and just imagine the number of metrics you can use to measure the success of your marketing efforts. 25? 50? 68? More?
It would not surprise me. We have so many, and it is not that they are terrible. It's just that there are so many and I set myself the challenge to just focus on two metrics each for acquisition, behavior and outcomes. Just two.
Can you feel my pain? How can we just pick two? #omg
For acquisition my goal was to pick two metrics that would incentivize our teams to behave in the most optimal manner. I abandoned CPC (I hate that metric, it incentivizes short-term thinking of the worst kind). I even abandoned CPA (not enough focus on what makes the ads good). And many others.
I settled on Clicks and Assists.
Clicks because in this case I'm more obsessed about performance marketing. Clicks will get you and I to focus on ad copy, bids, other creative elements, precise targeting (or retargeting), ad placement (quality score for search!), and so many other things that go into making sure that our ad does not simply show up to the party but also entices a physical reaction (Click!).
Assists because most marketers continue to exhibit suicidal tendencies by sticking with last-click attribution. The best way to stink at digital today is to do last-click attribution (and the analytics vendors don't help, last-click is default… they want us to go out of business!).
I want you to focus on % Assists as your standard metric for marketing success (best place to find it is in the Google Analytics MCF Assisted Conversions report). Reward campaigns (owned, earned or paid) that deliver the person who converts right away AND the campaigns who deliver the people who convert after a few days. Assists incentivizes the right behavior by your marketers.
[Bonus read: Multi-Channel Attribution Modeling: The Good, Bad and Ugly Models]
For Behavior, I picked Bounce Rate and Visitor Loyalty.
Bounce Rate to encourage an obsession with landing pages. Far too often with our ads we write checks that our site can't cash. Far too often the Marketer's are paid on ad metrics and not what happens on the site. I want to fix that. Hence Bounce Rate. Create great landing pages, ensure there is clear continuity between your ad, link, email campaign, social contribution, press release, and the place you are sending the traffic to.

Visitor Loyalty because I've grown less fond of "repeat visits" with the passage of time. Repeat visits is too low of a bar. Visitor Loyalty empowers a focus on one person and the quest to impress them so much that they want to come to your site (content or ecommerce) again and again and again.
Best practice: Look at your Visitor Loyalty report, pick the number that is most optimal for your business (say, more than 5 times in 30 days) and set that as your target. Now measure like crazy!
I hope you noticed that the behavior I'm encouraging with Visitor Loyalty is the reason why % Assists is such an important metric as well.
You get people to stay for more than one page during their first visit… and get them to visit your site multiple times… how can you not win!
[Bonus read: See-Think-Do: A Content, Marketing, Measurement Business Framework]
Finally for Outcomes, I choose my absolute favorites: Macro and Micro-Outcomes.
The best websites in the world have a portfolio of outcomes (in case your customers are not ready to jump into bed with you right away!). Some of these outcomes add value to the business in the next 30 days, some over 90 days and some over nine months. The portfolio empowers you to solve for every type of customer, and let them move at the pace they are most comfortable with.
You are not going to be successful if your owned digital existence does not have a portfolio of outcomes, here's an example for a casino/destination…

I do find it a smidgen irritating that so many Analysts and Marketers focus on single-session conversions for just one outcome (order or lead or something else). C'est tres terrible!
Macro-Outcomes help you measure immediate success, over 30 days. Notice even this is a small collection and not just one conversion.
Micro-Outcomes ensure that there is long term thinking in your company, the best kind of thinking, and that your employees undertake efforts to drive business value from every visit to the site.
Six simple critical few metrics that balance for the short and the long term, that incentivize the right behavior by our peers and agencies. And all standard metrics in any decent web analytics tool.
[Bonus read: Excellent Analytics Tip #24: Obsess About Real Business Profitability]
Let's focus on the other owned asset we have as a company, mobile.
If you've been to one of my keynotes recently you might have heard me recommend a mobile only strategy, and not a mobile first strategy. We have so much progress to make in that area. With mobile becoming the first (or even primary) digital touch-point for our customers, we have to have an insane amount of focus in this area.
[If you are short on ideas for mobile greatness: Mobile Marketing 2015: Rethink Customer Acquisition, Intent Targeting.]
For mobile Acquisition, our critical few metrics stay the same. Clicks and Assists.
Commercial user behavior on mobile is still transient rather than persistent, so you might initially get a lot more value with a focus on Clicks. After that, having thought long and hard about this choice, I still recommend keeping Assists. One day when cross-device tracking/analytics is easier (it is way too hard today), you'll be ready.
For Behavior, I have Bounce Rate and Time Spent.
Bounce Rate is even more important for Mobile, the back button is hard coded into the mobile device! (At least for Android.) You stink and they'll be gone in 0.05 seconds.
Again, because of the transient nature of mobile behavior, I made a switch from Visitor Loyalty to Time Spent. In most cases people will visit your site for a specific purpose (even more so on phones than tablets), create experiences that answer questions and get them to spend the time it takes them to have a successful visit.
The exception is a content site (my beloved New Yorker as an example). For their mobile site, I would keep Visitor Loyalty.

And one more time, because of current user behavior, and continued difficulty to transact on mobile devices, my recommendation is to focus on See and Think Micro-Outcomes.
A gigantic reason for under-investment in mobile innovation is the belief by businesses that they can't make money fast ("look, low conversions!"). That is a big mistake. Hence my recommendation to focus on Micro-Outcomes. Those metrics will incentivize the right behavior.
Though we segment Own into two pieces, you'll notice we only ended up with just seven metrics. Focus FTW!
Now let's go see what we can do for Rent.
Best Digital Metrics: Rent Existences/Strategies.
Our goals, participation options and desired outcomes all change for our Rent existences. Much less conversion driven, much more brand marketing driven, and much more longer term focused in terms of impact and outcome.
Most businesses and Marketers have sub-optimal results from their Rent efforts when they are primarily conversion driven, primarily focused on performance marketing and primarily looking for short-term impact.
In choosing the metrics, those elements were top of mind for me. (And I encourage you to keep that strategic view and purpose in mind as you choose your own.)
I've broken Rent channels into permanent and temporary. YouTube is an example of a permanent outpost. This is because for most videos uploaded on YouTube often over 70% of the views will come 30 days after you upload them. Your presence and contributions have a long life. While on temporary outposts like Google+ or Facebook your content, by most measures, "lives" for less than 60 mins – with a small trail of activity for 24 hours and then poof (!) it's gone. For Twitter, it is much, much, shorter than 60 mins/24 hrs.
Yes, on permanent and temporary channels your contributions (text, links, videos, contests, pimping) are always there. It is good for a little SEO. It does have other value (like people randomly bumping into your Twitter page). But almost all engagements your business will have, if any, will be on the latest content. Keep that in mind as you choose what to focus on.
For Acquisition, advertising you buy on YouTube, the metrics to use would be Impressions and Click-thru Rate.
Impressions because in this case a big objective is to solve for brand marketing, so getting a high share of voice is important. (Doing TV on YouTube!)
Click-thru Rate to give a tiny amount of incentive to extend the conversation beyond just the initial impression. CTR will encourage a focus on any commercial purpose of the ad, and creation of a great experience (on YouTube or your Owned channels).

For Behavior, we focus on Views and Completes.
Views because the primary front of our existence are our delightful non-boring videos. We want to encourage consumption of that content to enhance brand perception and move the brand equity up a few points.
But Views by themselves can drive sub-optimal behavior in Marketers. We counter-balance that with Completes . It will encourage a focus on video content, its ability to retain attention and, if we rock, will drive the outcome metrics.
For Outcomes, YouTube is not focused on conversions/short-term outcomes. We focus on driving organic Views (and Complete!) via Social Amplification of content. There is nothing that works better for brand marketing then getting other people to take your content, slap their own name on it and help you reach their friends/lovers/followers – an audience you might not otherwise reach.
And we focus on one of the most precious thing you can get from an audience on any rent channel: Attention! In this case measured via the growth in Subscribers . In study after study we have seen that your Subscribers are the ones who view your content on YouTube first, and their ability to help you out with the initial social amplification is the difference between a pin dropping in a forest or lighting the initial flames of, wait for it…. wait for it…. wait…. viralness!
Subscribers are your biggest fans, you earn more of them, your life becomes exponentially easier over time because you don't have to keep renting audiences via advertising (though please buy advertising, people who work at YouTube have bills :)).
[Bonus Read: Only three things work on YouTube, learn more about them... YouTube Marketing And Analytics: A Primer For Magnificent Success]
Home stretch, measuring our Temporary rent existence/strategies.
This is a bit harder because while there is some commercial intent we can tap into (regardless of if we are a news site or an ecommerce/lead gen site), it is extremely difficult to grab attention (very short half-life of your contributions). You have to deal with things like EdgeRank, or the faster-flowing-everyday river that is your customer's Twitter stream.
I must have mentioned somewhere in this post that picking your critical-few is hard work!
For Acquisition, advertising we buy on Twitter, Facebook etc, my choices were Impressions and Likes/+1s.
Impressions, see above section on our permanent presence.
Likes/+1s is interesting, is it not? This was really hard. Why would you advertise on your temporary social channels? It turns out there is almost no commercial intent (or at least 43 times lower than email campaigns, see this graph – and click Follow!). My recommendation is to focus on getting outcomes on your social presence. Measure Likes/+1s at a post/brand page level. Pure brand marketing with a pinch of engagement on your rent channel.

For Behavior, we are going to focus on Amplification and Applause.
Amplification Rate measuring the number of Shares (Retweets, Forwards etc) on each post. Our goal is to create content that allows you to reach your second-level network. And when people amplify your content, they do so with their own brand endorsement. #awesome
Applause Rate measures the number of Likes (or +1s, Favorite clicks) on each post. It is very hard to know when you are creating content (text, links, videos, images etc) of value. Applause rate is your audience telling you what they like more and what they like less. Focus on the more content and not on the less content. :)
For Outcomes, we are going to focus on Conversation and Owned Micro-Outcomes.
Conversation Rate measures the number of audience comments (or replies) on each post. We are renting these spaces, creating amazing content, doing great brand marketing while participating in and initiating great conversations. So, let's just measure that. Do we create content that creates conversation? If not, why are we on social channels? Great outcome to measure.
Micro-Outcomes measures the outcomes that deliver value to your business over 90 days and over nine months. (For more, please see the Own section above.) This should incentivize our teams to focus on creating long term value on temporary rent channels.
[Bonus read: Best Social Media Metrics: Conversation, Amplification, Applause, Economic Value]
[Bonus read: Facebook Advertising / Marketing: Best Metrics, ROI, Business Value]
If you want to measure the social metrics mentioned above automatically, including Micro-Outcomes, you can do so using the True Social Metrics tool.
Best Digital Metrics: Comprehensive View.
Here's the complete picture from my keynote, the best digital metrics to measure the effectiveness of our ability to find the optimal customers (current, future), influence them and deliver win-win outcomes…

While it looks like a lot, I've recommended just fifteen metrics to measure everything you could possibly to with digital for your business. Not bad, right?
Please click on the image above to see a higher resolution version, or if you want to download it.
Closing Thoughts.
We are blessed with Big BIg BIG data, and many, Many, MAny, MANY metrics! It is our ability to identify the critical few spectacularly relevant metrics to focus on that will bring personal glory to us and professional glory to our business. (Mostly it is about personal. :))
In choosing metrics you have to make trade-offs, some are painful. More than learning about the best metrics available to us, I hope you've learned about the thought process behind each choice. Why something was chosen? What behavior it will drive in our teams/clients?
And one more thing… please remember that this is a journey and not a destination. The Web Metrics Lifecycle Process is Define > Measure > Analyze > Action > Improve or Eliminate. Rinse and repeat. Make sure you remember that (and follow the process!) before you spend so much time in discussion and choosing that the optimal time to deliver insights has passed you by.
Good luck!
As always, it is your turn now.
Does your current dashboard optimally cover Acquisition, Behavior and Outcomes? When discussing a framework to ensure you have the right metrics, does your company use the Own and Rent philosophy? Would you recommend a different set of metrics for ABO for mobile? What do you use to measure the success of your YouTube efforts? How about the temporary social channels like Google+ or Facebook? Is there a metric you consider your absolute BFF that is missing from the choices above?
Please share your advice, insights, questions, critique, ah-ha moments via comments below.
Thank you.

Like this post? Share it: