Those who rely on Arbitron’s ratings as their report card want to believe ratings mean something, that good programming is rewarded with improving numbers, while bad programming is punished with declining numbers.
Unfortunately that doesn’t seem to happen very often. You’ll have a good trend, then a bad one, maybe another good one, and after that who knows.
We call these seemingly random rating swings wobbles. Wobbles make it hard to tell what’s going on. How do you know when an upward trend is real? When should you panic if the numbers start going south?
To be confident the station’s efforts are working, you need ratings consistency, the opposite of wobbles. What we call Trend Persistence.
We decided to investigate wobbles, and what we found out may trouble you.
Our analysis suggests that monthly PPM trends are directionally meaningless. Trends change direction frequently and randomly.
Furthermore, PPM trends are no more consistent than diary based trends. Both are inconsistent indicators of the direction of a station.
As it turns out, the best way to predict your next trend is to bet against the last one. If last month the station went up, bet that this month the station will go down, and you’ll be right four out of five times.
Even more troubling is the fact that even a persistent trend can be a poor predictor of the health of a station. Four, five, and even six month trends can reverse course and leave a station right where it started.
We wanted to look at a market where PPM had been in place for a while because ratings seem to move around a lot soon after a market makes the transition to PPM.
We looked at a total of 253 6+ full week station trends just like you find in the trade publications.
We compared changes in share from month to month, counting the number of times a station’s ratings moved either up or down in the same direction two, three, four, and more times.
The graph above shows the percentage of times stations trended in the same direction. In four out of five instances, one month’s trend reverses the next month. If a station has an up month, for example, 79.8% of the time it will be followed by a down month.
Stations rarely trend in the same direction for two months. Only one in every ten trends (9.9%) persist for the second month.
Three month trends are even rarer. Our analysis found that stations move in the same direction (up or down) for three straight months only 7.1% of the time.
We created a simulated metro we called Farmville. In Farmville, the roll of the dice determines whether stations move up or down.
We ran repeated simulations using a similarly sized ratings data-set filled with entirely random ratings.
Totally random Farmville trends looked a lot like PPM ratings.
In Farmville, 78% of trends reversed the next month, compared to 80% for the PPM market. See the graph above to compare.
Significantly, the differences between Farmville and PPM increased as we moved further away from the most frequent reversals. While only 1% of Farmville’s trends continued for four months or more, 3% of PPM’s trends continued for the same length.
The difference suggests that over time PPM becomes less random--still random to a large degree, but at least less random.
You may recall that one selling point of PPM was that monthly PPM reports were more stable than diary quarterly reports, so we thought it would be interesting to compare the two.
We looked at a quarterly diary-based share data-set with 420 12+ full week trends published by the trades.
Quarterly diary trends reverse 77.8% of the time compared to PPM’s monthly 79.8% reversal. In other words, trend flip-flops are about the same in PPM and diary.
A diary trend lasting two reports occurs 14.8% of the time compared to PPM’s 9.9%. Three report trends are 5.5% of the total compared to PPM’s 7.1%.
Rare diary and PPM black swan trends where a trend continues for five or more months are highest for PPM and lowest for diaries. The diary has 0.2% 5+ month occurrences compared to our simulation where they occurred 0.3% of the time, and PPM at 0.8%.
What does it all mean?
Were you to flip a coin to predict the direction of ratings next month, you would have a 50:50 chance of being correct. Ironically, these odds are better than the probability that your ratings will continue to move in the same direction they did last month.
Flipping a coin turns out to be directionally more predictive than looking at this month’s trend.
Share estimates always have two components. The first is the actual number, the share you would find if you could ask everybody what they were listening to, instead of just a small number of panelists.
The second component is statistical noise. It is interference like static drowning out the signal from a distant AM station.
The noise is completely random and unpredictable. Sometimes it adds to the actual share, inflating it. Sometimes it subtracts from the actual share, deflating it.
Both monthly PPM share estimates and quarterly diary share estimates have so much statistical noise in them that it drowns out genuine changes in month to month ratings.
The only way to overcome statistical noise that makes monthly trends unreliable is to increase the the number of panelists. The larger the sample, the lower the noise, the greater the predictive value of trends.
For both PPM and the diary we have to go out three or more reports to find that either service becoming even slightly more predictive than flipping a coin.
Even then, long lasting trends can be a trap. There is a statistical phenomenon called reversion to mean, where a spurious trend simply nullifies a previous spurious trend.
It is like a winning streak in Las Vegas. No matter how much you win, play long enough and the house will win it all back.
We found many rating streaks that simply repeated themselves in the opposite direction months later (we hope after the PD had cashed his bonus check).
Keep in mind that this analysis was on full week 6+ numbers (12+ for diary). Randomness increases as one looks at smaller dayparts and narrower demos. Depending on the panel size, it could take six months or more for a predictive trend to emerge from the noise.
And you can guess how useful weekly trends let alone minute by minute are when the entire month is essentially meaningless noise.
What to do?
1. First, be skeptical of every trend, up or down. Know that whatever the month brings, it means nothing without the context of a much longer time frame.
2. Resist the urge to make programming changes based on a trend or two. You may pull the plug on something that is doing much better than the numbers suggest.
3. In critical situations, get a second opinion. Perceptual research is no substitute for Arbitron, but it can help determine whether you are seeing just statistical noise and randomness or something real.
4. Add Fooled by Randomness, by Nassim Nicholas Taleb, to your bookshelf. It is an easy read and a good explanation of how we tend to search for meaning in random things, like Arbitron ratings.