Wednesday, June 26, 2013

A Simple Rating System to Judge Playoff Worthiness

(Photo Credit: The Express-Times)

You don't have to remind anyone connected with the Lehigh football program about the drama and subjectivity of the playoff selection process.

Last season, Lehigh went 10-1, their only loss coming to eventual Patriot League winners Colgate.  Despite the ten wins, the FCS Championships committee decided to deny the Mountain Hawks a playoff spot, thus making them the only school from an autobid conference with 10 or more Division I wins to be snubbed since the playoffs were expanded.

Lehigh fans might, then, see some of the news that "blindsighted" some of the FCS athletics directors with some interest.

For the upcoming season, there will be a new ratings system used by the FCS Championships committee members in that smoke-filled room.  The question is, though: will it prevent future snubs like the one Lehigh endured last season?


The news came out of North Dakota, where the two-time FCS national champion North Dakota State Bison had a scheduling dilemna: ten games, and little ability to get a final game that would "count" towards a possible at-large bid.

In seasons past, a school's wins against Division I schools were the only ones that "counted" for playoff inclusion.  For example, Delaware one season had a 7-4 record, but had one win against West Chester, which was a Division II school.  This clearly hurt their eligibility that season.

The Bison had 11 games, but late in the process Montana State abandoned their game with them in order to schedule a game against an FBS school instead.  That left North Dakota State with 10 games, and little to no chance to schedule a replacement.

This week, they announced their replacement game - Ferris State, a D-II school that would not "count" using the old system.

But buried in the release was something additional, and eye-opening.

In a surprise move that blindsided almost everybody including NDSU athletic director Gene Taylor, the FCS Championships Committee last week approved a new ratings system to gauge the quality of a team’s schedule. It will take into account games with Football Bowl Subdivision, FCS and Division II opponents. 
The expansion of the FCS playoff field from 20 to 24 teams was a reason cited for the change, Taylor said. 
“There will be levels of strength of schedule,” Taylor said. “For instance, a Division II win will not count as much as an FBS win. Home wins and road wins are also counted differently.”

While it was indeed a "blindsiding" move, in retrospect, maybe it shouldn't have been too surprising.  In a world of 16 qualifying schools, a minimum requirement for D-I wins might have been sustainable.  But in a world with 24 teams, suddenly there might be a dearth of schools that might have qualified for the 7 D-I win requirement - especially as some schools now have three games against FBS opponents on the docket.

So what will the new ratings system look like?  While it's not public knowledge just yet, there was a tantalizing hint in a WDAY report.

Gene Taylor, NDSU Athletic Director: “The FCS Championship Committee has been working on a new system going to be called the Simple Rating System and what this does is takes into account victories over FCS, FBS and Division II opponents so basically it's a strength of schedule being one tool in the selection process and that's brand new as of last week.”

While the strength of schedule system in totalis may not be publically known, what is known is the "Simple Rating System" is a system that has been used for years in the statistical realm.  Basically: it is a formula, used most famously in pro football, to determine a teams' strength of schedule.

You can read this wonky explanation of the rankings until your eyes bleed, but here's the dimestore version: It takes into account modified margin of victory, and a strength of schedule component that rates all the teams in the NFL.  As the season goes along, the data points converge - the overrated teams lose, the underrated teams win, and )most importantly) the ratings of every team gets recalculated every week to reflect this fact - and every week, the model is supposed to get stronger, as the ratings of the teams revert towards their actual "strength".

In the NFL, at least, the teams that played the strongest schedules are the ones that are most likely to win the Super Bowl.  Last year, Baltimore and San Francisco did.  The year before, the Giants did.

But in the NFL, you have 32 teams, and you know they're always going to play one of the other 32 teams.  So if you want to make this system for college football, where FCS teams play both the Alabama's and Ferris State's, what do you do?

In the Simple Ratings System, there is a formula for the margin of victory giving a small point boost for the away teams, and other measures in place to ensure that a team that wins 70-3 doesn't get credit for a 67 point win, which would skew the margin of victory data because they ran up the score once.

It seems logical to assume that there might be additional adjustments in the FCS version of the formula to make D-II games "worth less" - but count, which in a sense is better than the invisibility of these games before.

Also - in something that's less of a surprise - games against D-I teams would be adjusted to be "worth more", in a sense.  But there's an important factor to this that shouldn't be overlooked.  Close losses in this system to FBS schools could be counted as "wins".

Let's take as an example Liberty vs. Wake Forest last season.  Liberty lost that game 20-17.  In the published Simple Ranking System, the road team is given 3 points.  But what if, in the FCS flavor of the system, Liberty is given 5 points since it's an FBS vs. FCS matchup?  To the SRS, the final score is Liberty 22, Wake Forest 20.

Even if they can pull off a close loss, that's a huge boost to a teams' playoff chances.

Last year, I moaned publicly about "schedule strength" in regards to determining the playoff participants.  Not to pick on Wofford, but the 8-3 Terriers had as their out-of-conference teams one of the worst teams in FCS last season (3-8 Gardner-Webb), a terrible NAIA school (Lincoln, which they beat 82-0) and FBS South Carolina, whom they lost to, as expected, 24-7.  I felt they had artificially created "schedule strength" with South Carolina to go with their admittedly challenging conference slate, though their only win against a Top 25 school was against Appalachian State, 38-28.

It's important to say that the ultimate determination of the field will not be made in totalis by a formula.  Harris himself said that it's "a tool" the committee uses to evaluate teams, just as they've used other statistical tools in the past.

Harris also hinted that they would explain the new system at a later date.

But even it they don't, it seems like I could cook up a Google spreadsheet to determine my own version of the SRS starting in about, oh, week 6, and we could see how that plays into the committee's selection process.  I might not know exactly how they're weighting FBS and D-II wins, but I can come up with my own, public system, that anyone can see.

No comments :