Dr. Saturday - NCAAF

As recruiting takes over the landscape to an even greater extent than usual over the next week or so, we should also see the annual barrage of "snake oil" articles scoffing at the rankings that draw larger and larger audiences every year. In fact, they're already coming this year.

I was in this camp for a long time. One of the running ideas I had at my old site, actually, was to review old recruiting rankings and ruthlessly mock them for being so woefully off-base. Except that every time I tried, I ran into a wall: There were always as many (and usually more) players who "made it" among the top prospects as there were busts. Unlike some people, I find arguing "Recruiting rankings are stupid because Rhett Bomar" more than a little disingenuous; ditto "Recruiting rankings are stupid because Utah."

You can do anything you want anecdotally, especially with a subject as chicken-and-egg as recruiting. But I've become a believer over the last couple years because of the numbers -- that is, all of the numbers. Rivals, for example, uses a formula (no, I don't know the formula) to assign every team in the country a total score for its overall class every year, usually ranging from about 200 on the low end to just shy of 3,000 for the USCs and Floridas of the world. Here is each BCS conference team's total score for the last five years (the sum of the scores from 2004-2008). And here are those same five-year sums distributed according to overall winning percentage over the same period:

For all the hits and misses, the big trend is clear, and speaks for itself: The average winning percentage steadily increases in lockstep with increased recruiting points. The 15 or 20 teams that separate themselves on an annual basis have no chance of a losing record over any sustained period of time; the bottom fifth or so, obviously, is more likely to come out below .500.

Remember, though, that that graph is of overall records, including mid-major and I-AA patticakes that even the worst BCS conference teams typically out-recruit. Without that kind of context, the big picture can be a bit of a mess because of wildly varying schedules and other inconsistencies that wreck head-to-head comparisons; SEC teams, for example, are clustered at the top of the rankings every year, and Big East teams clustered at the bottom where the BCS conferences are concerned, and since these teams mainly play within their conferences, overachievers and underachievers are inevitable (somebody has to win the Big East). To get a more detailed of look, I took the same five-year point totals from Rivals and applied them to all 332 games last year between teams from BCS conferences, broken down here according to how far apart the two teams in each game were in the rankings:

Bottom line: Based on the recruiting rankings of the last five years, the "more talented" team according to the gurus won almost two-thirds of the time in 2008, by a little more than a touchdown per game. Just as importantly, the difference became more obvious as the gap widened, exactly as you'd expect if the rankings are worth anything at all.

There was virtually no difference between teams that recruited within 2,000 points of one another over the preceding five years (or less than 400 points apart per year); as you might expect, the rankings weren't very useful for parsing talent gaps that small with so many other factors in play, and teams that found themselves bunched closely together in the rankings were generally in the same situation when they went head-to-head on the field.

At that point, though, the class differences become too wide bridge, and the higher-ranked teams begin to dominate. Teams that brought in an annual 400-1,000-point advantage over their opponent on any given weekend won two-thirds of the time last year, by 10 points per game; teams that "out-recruited" the opposing sideline by at least 5,000 points from 2004-08 won a whopping three-fourths of the time, by more than two touchdowns. In other words, for every Oregon State over USC and Ole Miss over Florida, there were three cases of Oklahoma over Baylor, LSU over Mississippi State and Ohio State over Northwestern. But you knew that.

So the rankings are definitely not precise enough to predict the national championship (or, unless you're talking about USC, even most conference championships). But they are especially good at grouping programs into classes that tend to hold up over time. They establish the ceiling and floor of a program's potential: If your team isn't a top-10 recruiter over at least a three or four-year period, it's not going to be carrying off any crystal footballs, either.

A generalization, yes, but as far as generalizations go, it's solid enough to set an informed baseline for expectations until there are specific reasons to think otherwise. If you ask any prediction to do more than that, you probably lead a very disappointed life.

Wednesday: Recruiting stars and All-Americans.

digg delicious
more

17 Comments

Post a Comment
  1. kass0809@...
    1. Posted by kass0809@... Thu Sep 03, 2009 4:47 pm EDT

    Report Abuse

    Regarding the columns you refered to earlier, and I don't mean this to be insulting to the journalistic masses, but basically all journalists, esp. sports journalists, are very ignorant regarding statistics. I dont blame them, they are not trained to have an analytic mindset that can break problems down to their causes, they go by a simplistic cause-and-effect that is more often-than-not incorrect. So concepts like varability, honestly, reporters tend to not understand and therefore inaccurately report. Unfortunately, until journalism schools require a stats education, this will continue.
    Anyway, good article, my offseason is comprised of reading Dr. Saturday to use by brain cells and EDSBS to kill them.
  2. fmxda
    2. Posted by fmxda Thu Sep 03, 2009 4:01 pm EDT

    Report Abuse

    This just shows a CORRELATION between winning and recruiting rankings. It doesn't show that recruiting rankings CAUSE winnings.
    In fact, the opposite could be true in that winning causes a higher recruiting ranking; e.g. because the various sites get caught up in hype or even subconsciously rate winning programs' classes higher.
    Obviously that hypothesis can't be proved. So perhaps the answer is some analysis to see if recruiting numbers even out-predict prior year's record/ranking on a year-by-year basis, staggered so to account for when a recruiting class actually has impact. Using the same five-year time period for both record and recruiting is misleading for two reasons. It gives recruiting gurus time to possibly adjust rankings to prior success and also because recruits dont usually make an impact until 2 to 3 years with the team, probably peaking 3 to 4 years after.
    So for example you would test whether 2008 record/ranking is a better predictor of this season's (2009) success versus 2006 recruiting numbers.
    You could also test recruiting numbers versus the preseason BlogPoll or something of the like.
  3. PN
    3. Posted by PN Thu Sep 03, 2009 3:19 pm EDT

    Report Abuse

    @fmxda: Of course it's just a correlation. Using statistics, that's ALL you can ever prove. However, Doc's stat.s are accompanied by a convincing THEORY to go along with them. To disregard doc's thesis, you have to present a more convincing theory for the same correlation. Frankly, your theory of inter-causation just doesn't meet that standard. Over a very, very long period of time, I could buy the "program success=recruits being rated higher" thesis. Notre Dame is basically a living example of that. But the given correlation strength of win% and rankings over a very short period (when recruiting services would not have enough time to "update" their perceptions of each team and have their rankings adjust accordingly), Doc's theory seems best.
    (Finishing dissertation right now heavily involving stats.)
  4. James P
    4. Posted by James P Thu Sep 03, 2009 4:28 pm EDT

    Report Abuse

    I'm sorry Doc, but I don't buy your graph. Graphs can be very misleading, and I feel that yours is. The spread over the y-axis is about 50% of the graph for the first portion of the x-axis. That is quite a large margin. Do the dots correlate to teams? Then which team is which dot? And which is your independent and dependent variable? I feel that you could have charted that big red line as an up and down zigzag that becomes a gently rolling wave as it approaches infinity (15000).
    This is not to say that your fundamental premise is wrong; that recruiting rankings don't determine success. But there are only 2 teams above 80% win percentage, and both are above 9000. This to me implies diminishing returns on recruiting rankings. I think USC is a great example of this; so many top 10 classes and 5 start athletes, and every year they blow it in one game against a PAC-10 also ran.
  5. Simon M
    5. Posted by Simon M Thu Sep 03, 2009 3:05 pm EDT

    Report Abuse

    @fmxda
    Err, the correlation here is between winning percentage and Rivals.com ranking points. I suppose I cannot *prove* this at the moment, but it seems rather obvious that neither of these two factors is causing the other one. It is fairly preposterous to expect that being assigned a high recrutiment mark will *cause* your team to win, and it is slightly more feasibly yet still unlikely that you will be assigned a high recruitment mark because of your winning record.
    It's like to be a case of the 'third variable' phenomenon, where both correlated factors are related to another one which is causing the relationship. I would argue that this is an umeasurable 'true player talent' level. If this is true then Matt's graph indeed suggests a degree of validity to the rivals.com rankings, since if the valuation of a prospect (and therefore the points garnered from his commitment) was based on hype (high ranked schools recruit him -- he must be great), one couldn't reasonably expect for this correlation to occur.
  6. Double B
    6. Posted by Double B Thu Sep 03, 2009 3:03 pm EDT

    Report Abuse

    This is one of the best articles I've read on recruiting rankings. They tell us SOMETHING, they just don't tell us EVERYTHING. They do a great job of measuring overall athletic talent within a program. They do a poorer job of discerning the finer details of that talent (i.e. the difference between the 20th and 30th ranked classes in a given year).
    Looking at the table, you can see a "who's who" of college football in the top 10. Beyond that, however, it gets a lot cloudier. Is there a substantial difference between the 15th and 45th teams on the list in terms of recruits that hasn't been or can't be overcome by coaching and other factors? I don't think so. Look at it this way, the difference between #1 and #3 on that list (USC and Florida State) is greater than the difference between #21 and #41 on that list (Oregon and Wisconsin).
    Simplifying this a little bit, coaching and overall program administration will only take you so far. I think you can be a top 20 program with only top 40 recruiting classes (Texas Tech). You can be a top 10 program with top 20 recruiting classes (Virginia Tech). You can be a top 5 program (or even top 3) with top 10 recruiting classes (Ohio State/Texas). But you can't get to the top 5 program plateau with recruiting classes in the 20's, 30's or 40's. There's just not enough overall talent to sustain that level of play.
  7. Simon M
    7. Posted by Simon M Thu Sep 03, 2009 3:05 pm EDT

    Report Abuse

    @James P
    While graphs can indeed often be misleading, most of your arguments are preposterous.
    Most blatantly, you do realize that if you were to 'equal out' the 'lengths' of the axes (a dubious undertaking in the first place), the correlation would actually appear as if it was stronger? Which dot is which team is likewise completely irrelevant, and furthermore has no place in a statistical analysis. Neither are there any independent or dependent variables since this wasn't an experiment, which is the only scenario in which those terms apply.
    Also, you just as other fail to make a serious distinction between predicting success and causing it. No one in their right mind would claim that high recruitment rankings cause success, the question here is whether they can predict it to a reasonable degree.
  8. James P
    8. Posted by James P Thu Sep 03, 2009 4:28 pm EDT

    Report Abuse

    Simon M,
    I don't think that you read my argument correctly. Never did I say that one should "Equal out the lengths of the axes". I don't know how you even read that in my argument. Nor did I dispute the correlation between recruiting rankings and success.
    Also, you seem to be ignorant of independent and dependent variables. In this case the answer is recruiting rankings is the independent variable, and winning % is the dependent variable. Kudos to Dr. Saturday for getting the axis correct.
  9. Steve
    9. Posted by Steve Thu Sep 03, 2009 9:25 pm EDT

    Report Abuse

    @james p
    i'm pretty sure that red line is a standard ols, so the idea that it would be better as a zig zag is strange. certainly you could include that but i'm not sure what it really adds to the graph.
  10. James P
    10. Posted by James P Thu Sep 03, 2009 4:28 pm EDT

    Report Abuse

    Steve,
    You can see that at the left side of graph there is a wide deviation between recruiting and winning percentage. It tapers off towards the right. There are many possible conclusions one can draw from that. Perhaps high recruiting rankings don't really matter until you are in the top 60% percentile? If you redraw that line closer to the actual mean, the graph might say that.
    A team with a ranking of 4000 has the same winning percentage as a team with 13000. Knowing the teams might allow us to isolate exogenous factors (strength of schedule).
  11. LD
    11. Posted by LD Thu Sep 03, 2009 7:01 pm EDT

    Report Abuse

    There are 3 kinds of people in the world. Those who are good at math and those who aren't.
  12. Steve
    12. Posted by Steve Thu Sep 03, 2009 9:25 pm EDT

    Report Abuse

    We do know the teams. The worst winning percentage is Duke, and the second worst is Washington. Duke attracts a smattering of highly rated talent but for the most part is way overmatched by its conference. Washington has had one of the harder schedules the past few years (in addition to just not being very good to begin with).
    As for the 4000 vs 13000 comparison, I'm not sure exactly which dots you refer to, but I think its Florida and West Virginia. The r-squared of the ols is admittedly large at the left end of the graph, but I still think its safe to say that there is some positive correlation between "talent" as it is measured here and winning percentage.
    Of course, the graph itself is less valuable than the analysis of how teams with different point values match up.
  13. Zachary K
    13. Posted by Zachary K Thu Sep 03, 2009 8:00 pm EDT

    Report Abuse

    butbutbut but but butbut Uconn Uconn but look at Uconn
    /falls down sobbing and choking
  14. Zachary K
    14. Posted by Zachary K Thu Sep 03, 2009 8:00 pm EDT

    Report Abuse

    @fmxda, I *really* don't buy the argument that recruiting class rankings are directly influenced by winning percentage (or, in ND's case, historical prestige). Rivals et al. start looking at the kids long, long before most of the kids decide where they're headed. All the rankings do is tally up the individuals when all is said and done. As anecdotal evidence contrary to your claim, UNC landed a big class two years ago (off a 3-9 season) while Alabama drank everyone's milkshake last year off a .500 season.
  15. JP Girouard
    15. Posted by JP Girouard Thu Sep 03, 2009 3:37 pm EDT

    Report Abuse

    Also keep in mind that win-loss records is a trailing indicator of recruiting success - for that chart to have any meaning whatsoever, it needs to be tracking the rankings of recruiting classes of 2002-2006 (at a minimum) against the win-loss record of 2004-2008. If you do that, there's still a correlation, but it's decidely less obvious.
  16. martin v
    16. Posted by martin v Thu Sep 03, 2009 7:31 pm EDT

    Report Abuse

    James P, you said "Also, you seem to be ignorant of independent and dependent variables. In this case the answer is recruiting rankings is the independent variable, and winning % is the dependent variable. Kudos to Dr. Saturday for getting the axis correct. "
    You are incorrect and Simon M is correct. This is a correlation analysis. In a correlation analysis you do not have an independent and a dependent variable. This is a basic statistical concept that you can find in any Stat 101 text. If the author said he was doing a regression equation in which he was trying to predict winning percentage from recruiting rankings, then you would be correct in saying that recruiting rankings is the independent variable and winning % is the dependent variable. But the graph is simply looking at the relationship between the two and not making predictions.
  17. DevinR
    17. Posted by DevinR Thu Sep 03, 2009 3:45 pm EDT

    Report Abuse

    Seriously awesome graph. I love to see something semi-academic to back up your argument and it works perfectly

Dr. Saturday

Add to My Yahoo! RSS

Matt Hinton

Dr. Saturday is a college football blog edited by Matt Hinton. Email him tips and feedback.

Related Photo Gallery

Y! Sports Blogs

Dr. Saturday Recent Readers