This post is an adaptation of my presentation at the 2011 HOW Interactive Design Conference in San Francisco. I included a variety of slide transitions in my talk, so in some cases below, I’ve consolidated slide groups into animated GIFs. Keep your eye out for those so you don’t miss any details.
Do you remember your last yearly physical? Mine was sometime before the Obama administration, so I guess that’s what I mean by “yearly.” I should go every year, and you’d think that would be a reasonable goal to set—just one visit per year—but I just can’t seem to stick to it. Let’s be honest, going to the doctor is a drag. And I don’t make it much easier on myself by having a doctor that is a 35 minute drive away. (There must be some psychological significance to that.) Anyway, the physical was what you might expect: I was stripped, probed, poked, weighed, and even had a blood test. The works.
After the exam, my doctor and I had the following exchange:
Me: Should I go ahead and schedule next year’s appointment?Doc: No, that’s not necessary. Come back in a few years.Me: Really? I thought yearly was kind of, like, the bare minimum.Doc: Nah, stuff doesn’t really start happening until you’re forty.
Stuff.
Stuff? What does that even mean? I started scoping the place for diplomas.
A few days later, I got the results of my blood work in the mail. I stood there staring at the page for a while, trying to make sense of a grid of numbers. See, I don’t know what’s normal, and the report didn’t help me figure that out, either. It failed to tell me the basic story I needed it to: You are healthy, or You are not. So, why’d they even mail me this thing?
And that, right there, is the connection I want to make with measurement. Many of you have been handed reports just like this. Instead of telling you the story you need to hear, they deliver you too much information and, usually, not enough value. In fact, there’s a particular report that is common in the corporation that seems to do this every time: the KPI report.
Here’s an example for you to see; a page from an actual KPI that one of my clients paid for a few years ago:
Ugh. Doesn’t it just make you hurt all over?
I thought it was so important that you see this that I was willing to contaminate this beautiful page by including it. Now, I’m not saying there’s nothing of value in a KPI report. It’s just that after 90 pages just like this—seriously, that’s how long this particular KPI was—anyone in their right mind will start to tune out.
I like smart people. More than that, I like watching other people get smarter. But sometimes smart people create things that don’t come close to sharing their true intelligence. The KPI report is one of those things, which is why I’d like to suggest that we refer to it from now on as the “keeping people ignorant” report. Next time your manager blusters:
“Where’s my KPI report?“
you’ve got a ready response:
“Oh, you mean the…”
Actually, maybe don’t say that. Or, if you do, don’t say you heard it from me.
Seriously though, the KPI report represents all that is wrong with how we typically do measurement. The story I told you about my dysfunctional relationship with healthcare bears striking resemblance to what I’ve seen over and over again of website measurement.
Dysfunctional Measurement
Dysfunctional measurement has the following characteristics:
1. It’s outsourced. You’re paying someone else to do it for you because, for whatever reason, you believe you can’t do it for yourself. As a result…
2. It’s irregular. When you rely upon someone to do something else for you, it typically doesn’t get done the way it should. And when you’re paying for it, it probably isn’t getting done as often as it should. But when it does get done…
3. It’s too quantitative. Think about it. A third party cannot know enough about your business to ask the right questions—the questions you probably already are asking. They can give you stats, but stats aren’t always answers.
I want to show you how to dismantle that system. To do that, we have to address each piece.
Outsourcing isn’t all bad, by the way. There are many analytics experts out there that have deep expertise that is certainly worth seeking out from time to time. But that’s the key—from time to time. In between those engagements, you should be routinely measuring on your own. It’s the same principle with health: go see your doctor routinely, but don’t ignore your health in between. So if we start with #1 and make website measurement our responsibility, we can begin to right the ship.
But, it’s not just about frequency, either. I don’t think that going to my doctor more often would make it more appealing—or even more valuable. After all, if I found a doctor that only weighed and measured me every time I had an appointment, you’d probably suggest I find a new doctor. But that’s pretty close to the approach we take by default with website measurement. We look at basic stats, and we look at the same ones over and over again. And generally, we’re looking for more—more visits, more pageviews, more conversions. I imagine that if I kept returning a bigger number with each visit, even a lazy doctor would eventually get concerned about me. So more isn’t enough. It’s not even about more, really.
It’s about having a holistic, functional system.
Functional Measurement
We can turn our dysfunctional system into a functional one by:
1. Doing it ourselves
2. Following a routine
3. Keeping it simple.
Or, put in natural language:
Functional measurement isn’t occasionally paying someone else to gather numbers for you. It’s regularly gathering data that provides enlightening, qualitative insights.
Sounds good, right?
I like it, but I have one problem. Qualitative sets off a bit of a subjectivity alarm for me. We need to define what qualitative means.
Qualitative measurement answers questions.
Specifically, the questions you should be answering through measurement are:
1. Who is using my site?2. Where are they coming from?3. What content are they consuming?4. How are they engaging with that content?5. What can I do to make their experience better?
So far, we’ve turned measurement upside down.
It’s not about numbers as an end; it’s about numbers as a means to an end. That’s the essence of measurement.
A while back, I discovered something that really helped me understand how to get down to the essence of things, something I later applied directly to measurement and was instrumental in shaping the process I’m sharing with you here. Have you ever heard of The Two Things?
The Two Things About Website Measurement
I was introduced to The Two Things by Glen Whitman, who has a page up on his website all about it. It begins with a story he told about meeting a guy at a bar. When his new acquaintance learned that Glen is an economist, he asked him, “What are the Two Things about economics?” Glen was surprised at this question, and the man clarified that “for every subject, there are really only two things you need to know. Everything else is the application of those two things, or just not important.”
Glen offered his two things about economics (“Incentives matter,” and ” There’s no such thing as a free lunch”), and then was hooked. He’s since gathered a huge list of two things’, covering a wide array of topics—from business to being a DJ.
So, to frame the rest of this series, I’d like to share with you the Two Things about web measurement. Since first thinking about this, I’ve revised them a few times, but I think I’ve settled here:
Thing #1: There are no independently meaningful metrics. It’s about combining them to answer questions.Thing #2: Anything can be a source of data.
I believe that if you stick to these two things as non-negotiable truths, you’ll get the most out of the measurement you do.
Next Time: I’ll focus on Thing #2 and take a look at one specific source: Google Analytics, and then in the final post, I’ll take a look at another source: Real, live users. Stay tuned!
Resources Recommended by Imprint
-
Available now: Print Magazine’s Guest Art Director Digital Collection
-
Milton Glaser and Mirko Ilic give a live DesignCast on The Design of Dissent. Register now!
-
Get an inside look at logo design from Chermayeff & Geismar
-
Design TV: Improve your design skills with help from experts in the design industry.