Thursday - August 07, 2008
On ESPN Part 2
I mentioned some time ago that I'd been looking into ESPN's foray into recruiting ratings, now four years old and presumably gaining some consistency. In part 1, I talked about my frustration at trying to find some consistency in the ESPN ratings that would allow me to create a mapping from ESPN's 100 pt scale to Rivals' 6.1 pt scale, something that would let me say "This kid was rated 79 by ESPN but 6.0 (high 4*) by Rivals, so Rivals clearly has him rated higher. Such a mapping was hard to create because ESPN's use of the 100 pt scale seems to vary widely from year to year.
But I said I'd keep trying. I said I'd report back in a few days. That was on June 19th.
In Part 2 (this), I'm looking at one very specific issue: Michigan.
That is, I've tracked the Rivals and Scout evaluations of Michigan's recruiting classes for a few years now, and while there are multiple cases each year where one is more bullish on a recruit than the other, in the end, it seems to be a crapshoot. That is, they have differences of opinion (which is why it's worth following both), but there seems to be no inherent bias. You can't say "Scout is always more bullish on Michigan recruits than Rivals is" or "Rivals always thinks more highly of Penn State recruits than Scout does."
I wanted to throw ESPN into the mix.
So here's what I did:
I added 2009 recruiting rankings from ESPN to the mapping and tried again. And I came up with this.
In the 2009 Rivals class, there are 25 5*s (6.1), 57 high 4*s (6.0), 75 mid 4*s (5.9) and 162 low 4*s (5.8). The mapping I came up with was
Rivals: ESPN (# of players / year per ESPN)
6.1: 84->100 (33)
6.0: 82->83 (50)
5.9: 80->81 (112)
Then I pulled in 3.5 years of recruiting classes for Michigan (Feb 2006 -> Feb 2008 and ther current, unfinished class). I also brought in the 3 other midwestern powers: Ohio State, Penn State and Notre Dame, then quickly dismissed Penn State because (no offense Penn State fans) they have not recruited at the same level as Michigan, OSU and ND and that affects the comparison.
And having established (though not published here, because it's a boring hypothesis to lay out evidence of) that Rivals and Scout seem fairly in line, I figure I am, as best as can be done, comparing ESPN to a consensus of 2.
I also allowed for minor differences in ranking. If one site says a player is 50th, a high 4*, and another says he is 25th, a 5*, that's not a difference worth noting. So translated to the Rivals 6.1 system, I noted the ratings as being significantly different if they were at least 0.2 apart - a 5* compared to a mid 4*, a mid 4* compared to a high 3*.
I also made some assumptions and threw out some data, and these are just my judgment, to be trusted or not trusted as you see fit:
1. I ignored players who were rated mid 3* or below by both services. I am not interested in whether Rivals and ESPN disagreed on just how mediocre the bottom of the class was, and there is too much noise in those ratings.
2. I ignored players where I did not have Rivals ratings or where there appeared to be good reason for not having ESPN ratings (Junior College transfers, foreign players).
3. However, if (to the best of my knowledge) a recruit was a conventional American high school recruit and was rated as a decent prospect by Rivals and simply not scouted by ESPN (usually carrying a grade of 40, which is ESPN-speak for "unrated), I think it is fair to say ESPN thinks less of this player. This player is marked as one that Rivals rated higher. I figure if Rivals thinks a kid is solidly Big 10 level, and ESPN simply chooses not to scout him (despite scouting over a thousand, maybe over two thousand kids), then ESPN doesn't think much of the recruit.
From there, I looked at one basic thing: how many players, by team, were rated higher by Rivals than by ESPN, and how many were rated higher by ESPN than by Rivals. If there's no bias, and with a decent sample of 68-78 comparable recruits for each team (3.5 recruiting classes) you'd expect it to be some high, some low, and a near washout.
School: Rivals is higher / ESPN is higher / Sample ... Effect
Michigan: 13 / 6 / 68 (-10%)
Notre Dame: 5 / 14 / 78 (+12%)
Ohio State: 14 / 8 / 75 (-8%)
Effect = (ESPN high - Rivals high) / Sample size
Positive = ESPN likes the classes more. Negative = Rivals likes the class more.
8%, 10%, 12% ...
Keep in mind, when I did this for Rivals vs. Scout, the numbers came out Michigan 2%, Notre Dame 4%, Ohio State 1%. By comparison, these are wild an enormous numbers, 8%, 10% and 12%. It does *not* mean that ESPN and Rivals agree more often than Rivals and Scout do. In fact, the # of matches is very similar. What it means is that while when Rivals and Scout disagree, it's as likely to be Rivals higher as it is to be Scout higher, when ESPN and Rivals disagree, it's usually to Notre Dame's benefit and Michigan's and Ohio State's detriment.
Bias? Not ready to say that.
First off, there could be a statistical bias here. Rivals and Scout have stable ratings systems. There may be 30 5* recruits one year and 36 the next, but it won't just from 50+ to mid 20s the way ESPN's did (for players rated 84 and higher). There won't be a 30-40% reduction in 4* players for one year, and then a jump back up, as there was between 2006, 2007 and 2008 for ESPN. Those wild swings could make a difference if, say one team had a small class and another had a large and blockbuster class in a year where ESPN was stingy. One team minimized the exposure to ESPN's stingy year, while another was fully exposed.
Another possibility is that it could be regional or positional. Maybe ND recruited a *ton* of offensive linemen and ESPN is just more generous with OL ratings than Rivals and Scout are. Those things (stripped of the school issue: just position and region) are things I will look at in part 3.
But before I get to part 3, one quick way of looking at the results of part 2.
If you just take the RR Rivals ratings and average them by team, what you get for this 3.5 year sample is
#1: Ohio State - 5.81
#2: Notre Dame - 5.78
#3: Michigan - 5.76
Not much of a difference. A difference of 0.03 over a class of 20 = trading 3 high 3* players for mid 4* players, noticable but not a huge difference.
If you average the mapped ESPN ratings, you get
#1: Notre Dame - 5.81
#2: Ohio State - 5.75
#3: Michigan - 5.72
The difference between that 5.81 and 5.72 is basically the same as if you took every single kid in Michigan's recruiting class and bumped them up a notch (from high 3* to low 4*, from low 4* to mid 4* ...). That's an enormous difference. And pwhat's driving it is that ESPN is just not impressed with the Michigan recruits that ESPN considers the elite among our classes. Michigan has 5 Rivals 5* players in the sample (Ryan Mallett, Brandon Graham, Donovan Warren, Steven Schilling and William Campbell) and only 2 of them carry even top 100 grades from ESPN (Mallett and Graham). 3 of Michigan's 9 "high 4*" players by Rivals are rated lower than that by ESPN (Justin Turner and Jonas Mouton as mid 4* and Boubacar Cissoko as a low 4*). And on the flipside, if that's the definition of elite, only 1 player has been named elite by ESPN that was not elite per Rivals (J.R. Hemingway).
On the flipside, there are 5 players that carry 5.9 or lower ratings by Rivals (mid 4* or lower, not top 100) that carry 5* / 6.1 grades from ESPN, and four more that are rated high 4* / 6.0 by ESPN.
Anyway, that's just data.
I want to dump things out by region, by position, by year and see what else turns up.
Basically, I want to look into this and see if there's consistency or bias in ESPN's ratings and whether I should continue paying attention. Of course, the best test will come on the field, when we can see the players that the services disagreed on and evaluate who was right and who was wrong.