Why Internet Measurement is Not Quite as Exact As We Thought

October 5, 2009 at 3:10 PM 5 comments

This particular post from Media Post’s “Online Metrics Insider” weekly email came across my desk, and it does an excellent job of summing up why the job of an Internet Research Analyst can be so darn difficult at times. 

In theory, internet measurement should be the most exact form of audience measurement, especially when compared to the broad strokes estimates provided by Nielsen (Television), Arbitron (Radio), ABC Audit (newspapers), etc.  It is far too easy for non-research geeks to look at internet audience numbers and assume they are 100% to the decimal point correct.

But as the post below illustrates, there are many, many different elements that go into developing an internet audience metric, all which affect the bottom line.  Thus, while I have worked with Internet Analysts who like to report the internet figures down to the exact total (“We had 23,345,854 page views this month), I prefer to work with more broad terms (aka 23.3 million page views), accounting for the differences in measurement methodology.


~ JY


The Numbers Just Don’t Add Up
by Jim Sterne , Friday, October 2, 2009


A funny thing happened on the way to the CMO’s office.

Between the realization of an eye-opening, game-changing insight gleaned from advertising test results and Web behavior data, the report you’re gleefully ferrying to the C-Suite wilted, turns brown at the edges and starts to dribble a slimy substance with a conspicuous stench.

The CMO immediately develops a nose-squint. The VP of Corporate Communications has that “Oooo, you’re in for it!” look in her eye and the VP of Advertising nudges the Director of Direct Marketing and says sotto-voce, “The golden boy is about to find out his day in the sun has turned him to toast.”

The CMO points to (but does not touch)

        a traffic report from comScore

        a traffic report from Hitwise

        a chart from Compete.com

        an ad banner report from Atlas

        a traffic report from Omniture and

        another from Google Analytics

“It’s like the old joke,” she said with no humor at all. “If you take all the economists in the world and line them up end-to-end, they all point different directions. What the hell is going on with these numbers? Are we getting thirty two and a half million people on our Web site or forty-four million?”

The first time you ran into this nest of nettles, you hopped over to the white board and cheerfully explained all about

        cookie deletion

        cookie blocking

        multiple machine browsing

        multiple browser browsing

        multiple people on the same cookie

        non-human traffic

        dynamic IP addressing

        page caching

        javascript loading

        called pixel placement

You didn’t even get to the good stuff about comparing miles to gallons and how

        different tools using

        different date cut-off routines and

        different methods to capture

        different types of data to store in

        different kinds of databases with a

        different method of data cleansing and

        different slicing and dicing segmentation to produce

        different kinds of reports that ended up in

        different feed for integration into

        different datawarehouses

…before you were thanked for your help and shown the door — permanently.

You don’t fall for it this time.

This time you explain that the world of online marketing has been suffering from an delusion of precision and an expectation of exactitude.

You tell them that we live in a world of statistics and probabilities. We can’t count all the stars in the sky, so we don’t try. We don’t try to get an actual count of

        television watchers

        radio listeners

        magazine readers

        billboard readers

        bus poster readers

        floor sticker readers

        airline ticket jacket readers

        sandwich board readers

Instead, we count some and estimate the rest.

You share the good news that we can do this better than any of the above — and we’ve got some astonishing tools and techniques for dynamically targeting the audience and optimizing each one’s experience.

You say, “We get 36.3 million people coming to our Web site.”

The CMO lowers her half-glasses and gives you the look you last saw when caught using the office copy machine for party invitations. So you add, “With a 4% margin of error and it’s a benchmark we can compare month over month from now on.”

“So somewhere between 34 and a half and 38 million,” she says.

“Pretty much right between them, in fact.”

Disparagingly, she asks, “You really can’t give me a more accurate number of how many people saw this digital marketing masterpiece that costs me tens of millions a year?”

“I can tell you whether our digital visitors are more engaged with our brand, come back more often, buy from us and discuss our products with their friends. How many people buy our products who saw our ads on CNN and ‘Oprah’ that cost you hundreds of millions a year?”

The VP of Advertising makes himself visibly smaller.

“I came here to show you a way that could save four million dollars of search marketing while boosting online sales by 6 to 8%,” you say.

The scowl leaves the CMO’s face. The odor of dubious data dissipates. Her eyes narrow as she leans forward and says, “Show me.”

The numbers don’t have to be precise — just compelling.


Entry filed under: Online.

The Power of Yahoo!…TBO’s Story about Wii-Playing Police Officers Leads to a 2.8 Million Page View Day Nielsen’s Plan to Replace Live Only Ratings with Live Plus Same Day

5 Comments Add your own

  • 1. Article Marketing and E-mail Campaigns : Viral Marketing  |  October 9, 2009 at 5:14 AM

    […] Why cyberspace Measurement is Not Quite as Exact As We Thought […]

  • 2. Rochelle Williams  |  October 13, 2009 at 8:38 AM

    Methodology impacts all research measurement – internet, broadcast or print.
    Unlike broadcast (Nielsen) there is no single web analytics tool that has been chosen as the single standard platform on the web.
    This may change with the purchase of Omniture by Adobe in early September, 2009. http://www.adobe.com/aboutadobe/invrelations/adobeandomniture.html
    Omniture will soon be implemented in all Adobe products and the industry will be moving closer to standard metrics and methodology that should clear the decks of multiple vendors, methodologies, metrics and standards.

  • 3. Rochelle Williams  |  October 13, 2009 at 8:49 AM

    Why are the QuantCast and Omniture totals so different?( I don’t have current access to Omniture) but I suspect the totals are between 2.8 and 3 million – vs. 5.7-monthly users (TBO) for QuantCast are largely due to the fact that new pages have not been Omniture-coded, while QuantCast uses a single line of code to capture all click-throughs.


    • 4. Jennifer Yarter  |  October 13, 2009 at 2:28 PM

      You are right – Omniture was averaging about 2.8 million users up through August. But due to some high profile TBO stories that were featured on the home page of Yahoo.com, our uniques have been pushed above the 4.5 million mark for the past two months.

  • 5. Rochelle Williams  |  October 13, 2009 at 3:52 PM

    Congratulations on the web-exclusive Wii video -and Yahoo home page placement of other stories – the gift that keeps on giving. News video will always benefit from (web exclusive) – and turning viral.
    Another recent triumph – the USF story (I saw Judd Chapin and WFLA live on MSNBC)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 10 other followers

RSS Our Company Products’ Twitter Feeds

  • An error has occurred; the feed is probably down. Try again later.

%d bloggers like this: