Tara Hunt’s getting discouraged with the lack of good metrics on the Web:
So, what do we know? What have we learnt? We just keep feeding the same damned system over and over. Numbers. Those numbers, they never lie, do they? They are scientific and logical and will tell us how successful or unsuccessful we are. Someone led us astray. Numbers do lie.
And points to a few good post on the subject.
Somehow I’m not surprised.
I would quote, for effect, that famous Ogilvy quote, “50% of advertising works, we just don’t know what 50″ and poke fun and say, But numbers online don’t lie, so go online.
Numbers don’t lie, numbers tell the truth. And my numbers tell me that 50% of the effort I put attracts an audience, I just don’t know which 50%. The Web does make a difference. Thanks to the Web I can measure how much I don’t know, in real time. I don’t need Ogilvy to tell me that.
Which brings me to a different post, this one by Joel Spolsky. It’s about management, but I’ll quote out of context:
Robert Austin, in his book Measuring and Managing Performance in Organizations, says there are two phases when you introduce new performance metrics. At first, you actually get what you wanted, because nobody has figured out how to cheat. In the second phase, you actually get something worse, as everyone figures out the trick to maximizing the thing that you’re measuring, even at the cost of ruining the company.
And on the Web?
Every system that puts a premium on measurement will be gamed. If you reward for numbers, people will find ways to improve those numbers. The question is, are you measuring the right numbers?
MySpace games the system big time. They could improve their UI, make it slicker, easier to use, waste less of their user’s time. But that will cut down the number of pages they serve. They don’t make enough money on these pages, but they sure made a nice sum on the promise of that traffic. In other words, they got compensates for numbers that were easy to inflate.
YouTube also games the system for exactly the same reason. Of all the videos that YouTube “serves†me, I watch maybe one in ten. But I don’t go to YouTube searching for these golden nuggets. I read feeds and those feeds embed videos. I don’t have the time to click on most of them, I just don’t care. But they do count as page hits.
If YouTube used links instead of embeds, I would be watching the same number of videos. But their page hits would be ten times lower. Closer to the reality of a potential revenue stream, but too low for the revenue stream they want to project to any potential buyer.
Those numbers are all real. They’re cold, hard, indisputable measures. What’s unreal is the implied expectations that all the attention will materialize into retention. It won’t.
Other favorite numbers. “Unique usersâ€. Those are measured by sessions that expire after 30 minutes. I myself contribute around 100 unique users to my blog every single month. “Real usersâ€. Those are people who at least once used the service, probably registered with an e-mail address. Most of them also used it at most once.
But they imply something, and they’re inflated for that very reason.
The one number that matters is money in the bank. Fiscally or metaphorically. Even if I don’t get paid to blog, I still like the attention, the ideas, the people I meet. Those are currency to me.
But when you’re building for the future, you can’t measure it yet. So you measure the now and make some projections. And it’s the now numbers that are inflated to make people look better. Sometimes, at the expense of the real number that matter down the road.
So for example your customer retention specialist, in his desire to earn the bonus associated with maintaining a customer, will drive the customer so crazy that the New York Times will run a big front page story about how nasty your customer “service†is. Although his behavior maximizes the thing you’re paying him for (customer retention) it doesn’t maximize the thing you really care about (profit).
That’s all basic management 101. If you know about management by measuring, you know the fallacy of it. The Web brings nothing new, or changes anything about that. As long as you’re measuring one number that means nothing, on the assumption that it affects some other number that means everything, people will game it.
The expectation that attention leads to retention is at fault. Keep measuring, just take anything you measure with a grain of salt.
Skunk Works » When optimum lies
Classifiche dei blog italiani