Occam's Razor
by Avinash Kaushik

Kill Useless Web Metrics: Apply The "Three Layers Of So What" Test

50 Comments | Print | PDF

Data, data everywhere yet nary an insight in sight.
Is that your web analytics existence?
Don't feel too bad, you share that plight with most citizens of the Web Analytics universe.
The problem? The absolutely astonishing ease with which you can get access to data!
Not to mention the near limitless potential of that data to be added, subtracted, multiplied, and divided to satiate every weird need in the world.
You see just because you can do something does not mean you should do it.
And yet we do.
Like good little Reporting Squirrels we collect and stack metrics as if preparing for an imminent ice age. Rather than being a blessing that stack becomes a burden because we live in times of bright lovely spring and nothing succeeds like being agile and nimble about what we collect, what we give up, and what we deliberately choose to ignore.
The key to true glory is making the right choices.
In this case its making right choices about the web metrics we knight and sent to the battle to come back with insights for our beloved corporation to monetize.
A very simple test can allow you to figure out if the metric you are dutifully reporting (or absolutely in love with) is gold or mud.
It is called the Three Layers of So What test. It was a part of my first book, Web Analytics: An Hour A Day.
What's this lovely test?
Simple really (occam's razor!):
Ask every web metric you report the question "so what" three times.
Each question provides an answer that in turn raises another question (a "so what" again). If at the third "so what" you don't get a recommendation for an action you should take, you have the wrong metric. Kill it.

This brutal recommendation is to force you to confront this reality: If you can't take action, some action (any action!), based on your analysis, why are you reporting data?
The purpose of the "so what" test is to undo the clutter in your life and allow you to focus on only the metrics that will help you take action. All other metrics, those that fall into the nice to know or the highly recommended or the I don't know why I am reporting this but it sounds important camp need to be sent to the farm to live our the rest of their lives!
Ready to rock it?
Let's check out how you would conduct the "so what" test with a couple of examples.
Key Performance Indicator: Percent of Repeat Visitors.
You run a report and notice a trend for this metric.
Here is how the "so what" test will work:

"The trend of repeat visitors for our website is up month to month."
So what?
"This is fantastic because it shows that we are a more sticky website now."
(At this point a true Analysis Ninjas would inquire how that conclusion was arrived at and ask for a definition of sticky, but I digress.)
So what?
"We should do more of xyz to leverage this trend." (Or yxz or zxy – a specific action based on analysis of what caused the trend to go up.)
So what?
If your answer to that last "so what" is: "I don't know… isn't that a good thing… the trend is going up… hmm… I am not sure there is anything we can do… but it is going up right?"

At this point you should cue the sound of money walking out the door.
Bottom-line: This might not be the best KPI for you.
Let me hasten to point out that there are no universal truths in the world (though some religions continue to insist!).
Perhaps when you put your % of Repeat Visitors KPI to the "so what" test you have a glorious action you can take that improves profitability. Rock on! More power to you!

Key Performance Indicator: Top Exit Pages on the Website.
[Before we go on please know that top exit pages is a different measurement than top pages that bounce.]
You have been reporting the top exit pages of your website each month, and to glean more insights you show trends for the last six months.

"These are the top exit pages on our website for the last month."
So what? They don't seem to have changed in six months.
"We should focus on these pages because they are major leakage points in our website."
So what? We have looked at this report for six months and tried to make fixes, and even after that the pages listed here have not dropped off the report.
"If we can stop visitors from leaving the website, we can keep them on our web site."
So what? Doesn't everyone have to exit on some page?

The "so what" test in this case highlights that although this metric seems to be a really good one on paper, in reality it provides no insight that you can use to drive action.
Because of the macro dynamics of this website, the content consumption pattern of visitors does not seem to change over time (this happens when a website does not have a high content turnover – like say a rapidly updating news site), and we should move on to other actionable metrics.
Here the "so what" test not only helps you focus your precious energy on the right metric, it also helps you logically walk through measurement to action.

Key Performance Indicator: Conversion Rate for Top Search Keywords.
In working closely with your search agency, or in-house team, you have produced a spreadsheet that shows the conversion rate for the top search keywords for your website.

"The conversion rate for our top 20 keywords has increased in the last three months by a statistically significant amount."
So what?
"Our pay-per-click (PPC) campaign is having a positive outcome, and we should reallocate funds to these nine keywords that show the most promise."

Okay.
That's it.
No more "so what?"
With just one question, we have a recommendation for action. This indicates that this is a great KPI and we should continue to use it for tracking.
Notice the characteristics of this good KPI:
#1: Although it uses one of the most standard metrics in the universe, conversion rate, it is applied in a very focused way – just the top search keywords. (You can do the top 10 or top 20 or as many "head keywords" as it makes sense in your case, just be aware this does not scale to the "mid" or "tail".)
#2: It is pretty clear from the first answer to "so what?" that for this KPI the analyst has segmented the data between organic and PPC. This is the other little secret: no KPI works at an aggregated level to by itself give us insights. Segmentation does that.

Key Performance Indicator: Task Completion Rate.
You are using a on-exit website survey tool like 4Q to measure my most beloved metric in whole wide world and the universe: task completion rate. (You'll see in a moment why. :)
Here's the conversation…

"Our task completion rate is down five points this month to 58%."
So what?
"Having indexed our performance against that of last quarter, each one percent drop causes a loss of $80,000 in revenue."
So what? I mean in the name of thor, what do we do!
"I have drilled down to the Primary Purpose report and most of the fall is from Visitors who were there to purchase on our website, the most likely cause is the call to action on our landing pages and a reported slowness in response when people add to cart."
Good man. Here's a bonus and let's go fix this problem.

Nice right?
Notice in this case you have a inkling to the top super absolutely unknown secret of the web analytics world: If you tie important metrics to revenue that tends get you action and a god like status.
Keep that in mind.
So that's the story of the "so what" test. A simple yet effective way of identifying the metrics that matter.
This strategy is effective with all that we do, but it is particularly effective when it comes to the normal data puke we call the "management dashboard". Apply the "so what" test and you'll make it into a Management Dashboard.
Closing Summary:
Remember, we don't want to have metrics because they are nice to have, and there are tons of those.
We want to have metrics that answer business questions and allow us to take action—do more of something or less of something or at least funnel ideas that we can test and then take action.
The "so what" test is one mechanism for identifying metrics that you should focus on or metrics that you should ditch because although they might work for others, for you they don't pass the "so what" test.
And killing metrics is not such a bad thing. After all this is the process that has been proven to work time and time again:

More here: Web Metrics Demystified.
Ok now it's your turn.
Do you have a test you apply to your web metrics? What are your strategies that have rescued you during times of duress? What do you like about the "so what" test? What don't you like about it? Do you have a metric that magnificently aced the "so what" test?
Please share your comments, feedback and life lessons via comments.
Thanks.
PS:
Couple other related posts you might find interesting:

• Six Web Metrics / KPI's To Die For
• 4 "Useless" KPI Measurement Techniques
• Five Rules for High Impact Web Analytics Dashboards

Like this post? Share it: