Occam's Razor
by Avinash Kaushik

The Difference Between Web Reporting And Web Analysis

86 Comments | Print | PDF

Someone asked me this very simple question today. What's the difference between web reporting and web analysis?
My instinct was to use the wry observation uttered by US Supreme Court Justice Potter Stewart in trying to define
po rn: "I know it when I see it."
That applies to what is analysis. I know it when I see it. : )
That, of course, would have been an unhelpful answer.
So here I what I actually said:

If you see a data puke then you know you are looking at the result of web reporting, even if it is called a dashboard.
If you see words in English outlining actions that need to be taken, and below the fold you see relevant supporting data, then you are looking at the result of web data analysis.

Would you agree? Got an alternative, please submit via comments.
I always find pictures help me learn, so here are some helpful pictures for you. . .
This is web reporting:
And so is this, even if it looks cuter:

And while you might be tempted to believe that this is not web reporting, with all the data and the colors and even some segments, it is web reporting:

See the common themes in all the examples above?
The thankless job of web reporting, illustrated vividly above, is to punt the part of interpreting the data, understanding the context and identifying actions to the recipient of the data puke.
If that is your role, then the best you can do is make sure you have take the right screenshots out of Site Catalyst or Google Analytics, or charge an extra $15 an hour and dump the data into Excel and add a color to the table header.
So what about web analysis?
The job of web analysis mandates a good understanding of the business priorities, creation of the right custom reports, application of hyper-relevant advanced segments to that data and, finally and most importantly, presentation of your insights and recommended action using the locally spoken language.
See the difference? It's a different job, requires different work, and of course radically different skills.
Examples of web analysis? I thought you would never ask. . .
This is a good example of web analysis:

[And not only because it is my work! Learn more about it here: Action Dashboard.]
Notice the overwhelming existence of words. That's not always sufficient, but I humbly believe always necessary.
When you look to check if you are looking at analysis or reporting look for Insights, Actions, Impact on Company. All good signs of analysis.
Here's another example of really good web analysis:

[Click on the image above for a higher resolution version.]
Ignore how well or badly the business is doing. Focus on approach taken.
Here are some things that should jump out. . . . A deliberate focus on only the "movers and shakers" (not just the top ten!).  Short table: just the key data. Most of the page is taken up with words that give insights and specific actions to take.
Another example that I particularly like, both for the style of presentation and how rare it is in our world of web analytics. . .

[Click on the image above for a higher resolution version.]
No table, no rows, no pies. And yet data holds center stage with clearly highlighted actions.
Normally, we all do the column on the left (it might look different, but we have it). Unfortunately we don't appreciate is the power of the middle column ("segmentation reveled"). That is super important because it gives the recipients exposure to the hard work that you have done and in a very quiet ways increases their confidence in your work. Guess the outcome of that? They take the actions you are recommending!!
Analysts constantly complain that no one follows any of their data-based recommendations.  How do you expose your hard work? In a garish Las Vegas show girl fashion where all the "data plumes" are, unsexily in this case, hanging off the body? Or, in quite concise ways? Only one of those two work.
One more? Okay here you go. . .

[Click on the image above for a higher resolution version.]
Diana has loads of observations, supported by visuals (sometimes it really helps to show the search results or the emails or the Facebook ad) with highlights (actually lowlights) in red, and finally recommendations.
And note the tie to outcomes (another common theme in all examples above). In this case, the search improvements are tied to the increase in donations I can make because of sales of my book. 1.5 extra smiles per month! (All my proceeds from both my books go to charity.) A good way to get attention from the "executive" and get him or her to take action.
Do that. A lot. Be creative. Yes it is hard work. But then again glory is not cheap, is it?
Exceptions to the rule.
Not every output you get from your Analyst, or "Analyst" :), with loads of words on it, instead of numbers, will be analysis. Hence my assertion that "I know it when I see it." Words instead of data pukes is just a clue, read the words to discern if it actually is analysis or a repeation of what the table or graph already says!
In the same vein not every output that is chock full of numbers in five size font, with pies and tables stuffed in for good measure, is a representation of web reporting. It is hard to find the exceptions to this rule, but I have seen at least two in nine years.
Top 10 signs that you are looking at / doing web analysis.
Let's make sure this horse is really and truly dead by summarizing the lessons above and using a set of signs that might indicate that you are looking at web analysis. . .

#1. The thing that you see instantly is not data, but rather actions for the business to take.
#2. When I see Economic Value I feel a bit more confident that I am looking at the result of analysis. Primarily because it is so darn hard to do. You have to understand business goals / outcomes (so harrrrrd!) and then work with Finance to identify economic value, and then you have to configure it in the tool and then apply advanced segments, and then figure out how things are doing. That is love. I mean that is analysis! Or at least all the work that goes into being able to do effective analysis.
#3. In the same vein, if you see references to the Web Analytics Measurement Model (or better still, see it in its entirety on one slide up front), then you know that the Ninja did some analysis.
#4. Any application of algorithmic intelligence, weighted sort, expected range for metric values (control limits), or anything that even remotely smells of ever so slightly advanced statistics is a good sign. Unknown unknowns are what it's all about!
Also mere existence of statistics is not sufficient. All other rules above and below still apply. :)
#5. If you see a Target mentioned in the report / presentation, then the Analyst did some business analysis at least. See the top right of the picture immediately above.
#6. Loads and loads and loads of context! Context is queen! Enough said.
#7. I have never seen web analysis without effective data/user segmentation. I think this statement is in both my books. . .  "All data in aggregate is crap." Sorry.
#8. If there is even a hint of the impact of actions being recommended then I know that is analysis. It is hard to say: I am recommending that we shift this cluster of brand keywords to broad match. It is harder to say: I am recommending. . . and that should increase revenue by $180,000 and profit by $47,000. Look for that.
#9. If you see more than three metrics in a table you are presented with then you might not be looking at analysis.
#10. Multiplicity! If you see fabulous metrics like Share of Search (competitive intelligence) or Task Completion Rate (qualitative analysis) or Message Amplification (social media) then they are good signs that the Analyst is stepping outside Omniture / WebTrends. I would still recommend looking below the surface to ensure that they are not just data pukes, but the good thing is these are smarter metrics.
User Contributions:
#11. From Carson Smith: If someone looks at your analysis / report / presentation / dashboard and has to ask "and… as a result?", then it might be reporting. What happened should be obvious.
[I love applying the "Three Layers of the So What" test to any analysis I present or see. I ask "so what" three times. If at the end of it there is no clear action to be taken then I know it is just web reporting, not matter how great it looks or how much work went into it. Ask "as a result?" or "so what?" to your work!]
#12. From Chuck U: 1) If it can be automated, it's probably not analysis 2) If your data warehouse team says they can automate it for you, then it's definitely not analysis. [#awesome! -Avinash]

Can you think of other signs? Please share your suggestions via comments. I'll add the best ones to this list.
In the list above, and in the examples in this post, you see my clear, and perhaps egregious bias for business analysis and business outcomes and business actions and working with many parts of the business and business context. But I've always believed that if you and I can't have an impact then why are we doing what we do?
I hope you've had some fun learning how to distinguish between web reporting and web analysis. It is a fact of life that we need both. The bigger the company, the more they want data pukes, sorry, reporting.
But if you have "Analyst" in your job title then you perhaps now have a stronger idea of what is expected of you to earn that title. If you have hired a "web analysis consultant" and are paying them big Rupees then you know what to expect from them. Don't settle for data pukes, push them harder. Apply the rules above. Send their "analysis" back. Ask for more. Raise your expectations!!
I hope now "you'll know it when you see it," and have more datagasms!
Okay, it's your turn now.
How would you answer the question about the difference between web reporting and web analysis? What signs do you look for when evaluating the work of your Analyst or Consultants?
Please share your thoughts via comments below.
Thanks.
PS: In case you are curious here's the current official definition of po rn, as outlined in Miller v. California:

(a) whether the 'average person, applying contemporary community standards' would find that the work, taken as a whole, appeals to the prurient interest,
(b) whether the work depicts or describes, in a patently offensive way, sexual conduct specifically defined by the applicable state law, and
(c) whether the work, taken as a whole, lacks serious literary, artistic, political, or scientific value.

Like this post? Share it: