Skip to main content

The Lack of Data in Rating the FCS Top 25

At the halfway point of the FCS college football season, chances are you're mad.

You're mad that the human pollsters, who comprise the voters in the Sports Network or Coaches' poll, don't properly value your team, or your team's opponents.

You're mad that the folks who program the computers, who either underrate your favorite team or overrate something in another team that you, yourself, don't value in your own rankings.

It's just like the picture above: everyone sees the statistics and stories, and part of a football helmet, without really seeing the accurate picture.

It begs the question: is there really, honestly, a way to get an accurate picture of how to rank the Top 25 teams in the country?

Or will arguments on the relative worth of one football team vs. another always be a part of the fabric of college football, never to be settled - even after the end of the regular season?


As you already know, my particular obsession with college football involves the Football Championship Subdivision, because that's the arena where my team, the Lehigh Mountain Hawks, compete.

I appreciate the importance of having the FCS community, on a national level, at least be aware of the exploits of the football team in order to gain some sort of national consideration.

But getting a total picture of the state of all the teams in the FCS is particularly difficult.

When I vote in a Top 25 poll, I like to think that I do a lot of research into my ballot and make a very good effort in trying to judge all the teams fairly.

But even when I pour over game recaps, highlights, press reports and sometimes entire football games online or on the DVR, there still isn't enough time to possibly watch, or experience, everything that a particular week, or season, has to offer.

Take for example, last week.

In week 7, there were 58 games that involved at least one FCS team.

Assuming you caught all three games on Thursday before heading into the weekend, assuming on average three hours per game, there were 165 hours of FCS football footage generated last week - literally seven full days of footage to go through, to really visually process and judge every single team.

(This is a typical week, but some weekends there can be even more.  In the first week of the season, there were 74 games, thanks to the large number of inter-division and intra-subdivision games.)

Since there's only about a 36 hour period to synthesize all the results and generate a Top 25 poll, by definition I had to come up with shortcuts to accurately judge all the teams.

Rather than watch every game, I chose which games to follow closely.

A matchup of nationally-ranked opponents, Towson/Villanova, was easily recorded on my DVR, and it didn't interrupt the Lehigh/Columbia game, so I focused on that game in particular.

I also DVRed Richmond/James Madison, which was on the NBC Sports Network and a cakewalk for me to record and watch.

I watched a little of Yale at Dartmouth on Fox College Sports - since my father, a Dartmouth grad, was up in Hanover for homecoming weekend, and I wanted to get a tiny glimpse of Yale and (maybe) my dad and mom in the stands, before taking care of my own family the rest of the afternoon.

On the computer I caught a little bit of Lafayette/Princeton while also sneaking a peek of Gardner-Webb/Coastal, which were on ESPN3 and therefore available to view a replay online after everyone had went to bed.  I even could forward to the highlights.

The Big Sky didn't have a lot of great matchups, but I did try to catch a little bit of Sacramento State/Northern Arizona, probably the best of the bunch that I could watch from home on my computer. With their free streaming solution on the America One network, I could get the stream at my house.

I didn't watch any of North Dakota State/Missouri State, since, by virtue of having watched them several times already this season and assuming it would be a blowout, I thought how their performance was unlikely to affect my overall ranking of the Bison after their win.  I did read the recap of their game, and I did get a bit of a sense of how the game went - Missouri State kept things surprisingly close for a while, but the Bison pulled away in the second half - but I didn't spend a lot of time on it because I had seen North Dakota State before.

There are a lot of games, too, that I couldn't watch in terms of game film.  Maybe I was too cheap to get the dish that would have given me the TV broadcast, or maybe it wasn't available to watch anywhere online.  Or maybe I wanted to spend time with my family.

But for the top teams I at least tried to check out a recap, or maybe some highlight video, to make some sort of decision.

Am I doing more, or less, than most people, in regards to shaping my knowledge of the world of the FCS?  I have no way of knowing for sure, but I do know that despite my large effort, there was a large portion of the FCS world I didn't get a chance to experience last weekend.

Though I try to fill the gap the best I can by catching video highlights and recaps, it's still not an easy process.

This certainly is also an issue in the FBS world, where the math concerning "hours in the day to watch college football" is similar, and the turnaround for coming up with a human poll even shorter.

But it's also highly unlikely, with ESPN, CBS, NBC and their countless sub-stations creating constant highlight reels of different outcomes, plays, recaps, and postgame interviews for voters to watch.  In a half hour, you can catch an awful lot of coverage, and pretty much all of the high points of all the games.

FCS still does not have anything approaching this level of coverage, meaning that any poll voter with the honor of determining an FCS Top 25 list needs to go some sort of extra mile in order to gather enough information to make some sort of decision.

Despite my best efforts, could I have done more?

Maybe I could have taken some time out on Saturday to catch, say, 4-2 Jackson State to play 1-5 Mississippi Valley State out of the SWAC.

I suppose I could have, but I didn't tune into Mississippi Valley State's free online stream to watch the Tigers go to 5-2, because I simply didn't have time or inclination.  After all, if Jackson State goes undefeated the rest of the way and goes 9-2, they will be obligated to play in the SWAC Championship Game instead of the FCS playoffs, which I care about more.

This leads to another problem with trying to determine a true Top 25.

Most of FCS competes for a spot in the FCS playoffs, but not everybody.

The teams of the SWAC, as I previously mentioned, does not.  (Neither does the Ivy League, for their own, illogical reasons.)

For those teams competing for the playoffs, though, some sort of Top 25 ranking is crucial for postseason consideration, and if I'm a voter in one of these polls, chances are I'm a representative that is rooting for some outcome that allows my team, or even just my conference, to have a squad in the Top 25.

For better or worse, I care about Lehigh and the teams of the Patriot League making the playoffs, because these teams deserve it.  I don't think set out to vote Lehigh higher than they deserve, I feel, but having seen every single one of their games this year I feel I have a pretty good grasp as to how good they are.  (For the record, I had them at No. 17 this week.)

But more importantly, what is the motivation for a multitude of fans of schools or conferences to rank, say, Harvard No. 1, or Jackson State No. 1?

First of all, they're less likely to have caught a game of theirs because they don't have much bearing on the playoffs - only indirectly.

Watching those games don't serve a direct purpose of seeding for the playoffs.  It doesn't help conferences get seeded better in the playoffs.  And it doesn't help the overall perception of my team.

Because the Ivy League and SWAC have opted-out of the playoffs, they've effectively opted out of the Top 25, too.

This is just one human element that definitely alters the outcome of these votes.

Personally, I like to think I rank Ivy League teams fairly accurately.  That's because, as a Patriot League fan, I'm a lot more exposed to them than the average national FCS voter.  In the past I have put Ivy League teams in my Top 10, while I haven't seen any Ivy League team ranked about 10 in a human poll for a very long time.

To me, Harvard is no-doubt-about-it a Top 20 team, and Princeton a Top 25 team, because I've seen them play and I know how impressive they are.  If both stay on a roll, I wouldn't hesitate to put them in my Top 10.  But I'm unusual in that respect.  Few others would.

Conversely, I'm not afraid to admit that I don't judge the SWAC as effectively as I could.

My experience with the SWAC is the occasional game on ESPNU on Thursdays, or sometimes a game on ESPN3 to catch.

But it's low on my personal priority list.  That's just the way it is: the Patriot League doesn't play any SWAC teams.  And while teams that play SWAC teams, like Sam Houston State who is at No. 2 in my poll, may not be properly ranked, it's a relatively small concern.  After all, it's tough enough to keep tabs on all the teams in the conference and Eastern area.

I guess that means my Top 25 is, by definition, imperfect.  I am not accurately judging the SWAC's relative strength in the overall fabric of FCS.  And the truth is, misrepresenting one or two leagues, whether through lack of data or otherwise, can affect a good portion of the Top 25.

The idea of a large poll of Top 25 voters from across the country is that, by virtue of all the different points of view, my lack of information about the SWAC gets counteracted by a series of voters that know the SWAC intimately, thus pushing Jackson State up in the rankings and in their proper place in the human polls.

This is the way polls have worked since the AP decided to make a writer's poll in the 1930s, largely to end debates as to who is really the nation's top team.

There's a lot to be said for this: while one person can't consume a week of game footage of every game in a week, a group of 100 people could collectively.

But it also means the construction of that human poll has to be made extremely carefully.  You can't have 40 Big Sky voters and 7 Patriot League voters in a poll - there'd be way too much tilt towards one direction, especially considering the importance in the Top 25 in helping shape the playoff field.

Finding a balance of thoughtful voters across the country is a difficult - some might say impossible - task.

Recently, an FCS Coaches' poll has been created, with the theory that nobody is better in creating a Top 25 than people who are paid to care about these things.  Again, there's a lot to be said for this: Villanova head coach Andy Talley or Albany head coach Bob Ford know a lot more than I ever will about college football, and their opinions matter.

But they not only suffer through the same information gaps as non-coaching pollsters, they also have next week's game to worry about.

Are Talley and Ford going to sit thoughtfully and mull over the relative worth of Jackson State and Harvard in the Top 25, or are they going to spend the time studying more game film on Towson?

This makes some people argue that any human polls worth the bits and bytes they're digitally printed on.

When I look at a human poll, I take it with a grain of salt.  I have to.  There are agendas to protect, and playoff positions to secure.   There might be regional bias, the polls might not balanced properly, or the might have the wrong voters.

And I know it's not because people are purposely being malicious or ill-informed - I know, because I'm one of these pollsters.  It's because to truly, objectively look at all the data and process it is a task impossible for humans.  Any human.

****

Perhaps it's these woolly elements that caused computers to be used to rank college football teams.

The idea behind a computer poll is that all those silly elements of "human bias" - vested interests in voting for your school, or conference, and incomplete data points - can be worked out of the system using a formula.

If you apply the same criteria to each game and each team, the theory goes, you'll get a better picture of the relative strength of a particular team than through human eyes.

It's this thinking that brought us the BCS formula in the FBS, a combination of human and computer polls that determined the placement of all the teams in the major bowls.

Whether it was intended to replace the human element or not, it's hard to say, but the truth is computer polls are a part of the whole fabric of college football.

Are the computers better than humans at discerning what's going on?

Personally, I don't think so.

The idea behind every single computer formula that ranks any teams, pro football, college football, college basketball or whatever, is that more data makes the rankings better.

Every game that's played is a "correct" answer that gets plugged into the computer and computes the relative strength of the rest of the teams.

In men's college basketball in particular, this works pretty well.

Let's take the Patriot League.  The typical college basketball season lasts about 28-30 games, let's say, not counting a postseason conference tournament.

The Patriot League consists of 10 teams, meaning every team plays 18 conference games, home and away.

That leaves 10-12 out-of-conference games, with most likely a good balance of easy home victories, games with teams of similar caliber, and "challenge games".

For a computer formula, that's a lot of data to process - which makes things like the RPI a fairly good, if not perfect, guide for picking a tournament field.

But college football is different in every possible way.

If Lehigh plays, say, Gonzaga in basketball, both teams will have a large body of work, and maybe even similar conference opponents, if not common opponents, to use as comparison down the line.

But let's say, in football, Lehigh plays Eastern Michigan.

How is it possible to get a truly fair measure of both teams' comparative strengths based on that one game?

Eastern Michigan is likely to only have one FCS school on the schedule, and Lehigh is likely to only play against one FBS school.  It virtually guarantees that there will be no common opponents.

Let's say Lehigh beats Eastern Michigan, and the Eagles go 1-10 the rest of the way, beating 0-12 Akron at home.

You can make two arguments, and neither can be disproven.

You can say "Lehigh's win over Eastern Michigan is a quality win, since they are FBS, and FBS is always better than an FCS team.  Jack up their schedule strength."

Or you can say, "Lehigh's win over Eastern Michigan is nothing special - heck, half the CAA would have laid them punishment, too.  It's the FCS equivalent of beating 1-10 Weber State."

At some point, the keeper of this computer formula has a question to answer - how can I mask this incomplete data to deliver a halfway-decent result?

Ideally, the formula keeper doesn't want to do this.  All keepers of computer formulas want data.

But there simply isn't enough of it to deliver any sort of reliable result.  As a response to this, they fudge.

Like human polls, the only way to try to reconcile this is through a shortcut - a shortcut that will almost always overweight a win over an FBS team - or even simply overweight the presence of an FBS team on the schedule.

Everyone can pretty much agree when Eastern Washington beat Oregon State it was a very big deal.  The Beavers haven't lost a single game since losing to the Eagles, and are only one game away from being bowl-eligible.  Though they have a tough, meaty schedule to go, they have an excellent chance to be playing in a bowl.

But where does, say, Fordham's upset of Temple rank in that pantheon?

Temple is 0-6, and it seems like a mortal lock they won't be playing in any bowls.  They've found new and exciting ways to lose every week.  When they face UConn the same day as Lehigh/Lafayette, it might be a battle of winless teams.

The only way to judge how Temple would do against any FCS team, so the computers say, would be to see how they've already done against FCS teams.

They've faced one - Fordham - and lost.  They won't face another this year.  That game ended 30-29, Fordham.

So what does that prove in the case of Temple's relative worth against other FCS schools?

Something, but not a lot.  That's the problem.

The only possible way to judge FCS vs. FBS games is through objective evaluations of both squads, and speculation.

Personally, I speculate that Lehigh would lose to Temple if they played head-to-head, but I can't prove that using any measurable data.  My only data is what I've seen of both teams, and perhaps some updated statistics.

I feel like it would be a close game, but I could be wrong.  Can I be proven wrong, or right, by a computer?

It's impossible.  There's not enough data.

Some try to minimize this overweighting of FBS teams by trying to rank all teams from NAIA to FBS in one humongous list, comparing every single team against another.

This doesn't fix the problem, though, because the problem is the same as it ever will be: not enough data points.

When a sample Simple Ratings System using all the 2012 data was published before this season, four of the top teams were from below Division I.  Some humans might value a win over Minnesota State-Mankato as more valuable than one over Montana State, but not this human.

To be fair, the Simple Ratings System, as well as every other computer formula, tries to be objective.  And the SRS in particular works extremely, surprisingly well for a closed system of teams that only play each other, like the NFL.  As the NFL season continues, the system has some anomalies, like any formula, but it really gets amazing once the playoffs roll around.

It may also work for college basketball, where there are a multitude of data points and data that can be compared.

But no formula can really work properly without enough data, and the college football season simply doesn't provide enough of it.

So when I read someone saying that "Sagarin ranks Lehigh at No. 40!" or "The Simple Ratings System Puts Sam Houston State at No. 2!", I take it all with a grain of salt.  I have to.

And I know It's not because the people keeping the formulas are malicious, or wrong.  There simply isn't enough data.

*****

About the only thing that seems certain is that the way things are now, and maybe even forever, the current web of human and computer polls seem to be destined for argument fodder across the country.

You can criticize either the human polls or the computers, and you'd be right.

And you'd hope, too, that the people picking the field for the playoffs would see both the computer and human polls the same way I do: with a grain of salt.

I hope they think that it's not because the people who create all this data are malicious, or wrong.  It's just that it never, by its nature, can be absolutely right.

Like the people who create the human polls or the computer formulas, all you can hope is that the people picking the field are have seen a lot of game film, are very objective, and bring a lot of data points, however imperfect they might be, into the room where they pick the field.

Just don't rely on them alone.

Comments

Popular posts from this blog

How The Ivy League Is Able To Break the NCAA's Scholarship Limits and Still Consider Themselves FCS

By now you've seen the results.  In 2018, the Ivy League has taken the FCS by storm. Perhaps it was Penn's 30-10 defeat of Lehigh a couple of weeks ago .  Or maybe it was Princeton's 50-9 drubbing of another team that made the FCS Playoffs last year, Monmouth.  Or maybe it was Yale's shockingly dominant 35-14 win over nationally-ranked Maine last weekend. The Ivy League has gone an astounding 12-4 so far in out-of-conference play, many of those wins coming against the Patriot League. But it's not just against the Patriot League where the Ivy League has excelled.  Every Ivy League school has at least one out-of-conference victory, which is remarkable since it is only three games into their football season.  The four losses - Rhode Island over Harvard, Holy Cross over Yale, Delaware over Cornell, and Cal Poly over Brown - were either close losses that could have gone either way or expected blowouts of teams picked to be at the bottom of the Ivy League....

UMass 21, Lafayette 14, halftime

Are you watching this game? UMass had this game under control until about 3 minutes in the second quarter, and then got an interception, converted for a TD. Then the Leopards forced a fumble off the return, and then converted THAT for a TD, making this a game. It's on CN8. You really should be watching this.

Examining A Figure Skating Rivalry: Tonya and Nancy

It must be very hard for a millennial to understand the fuss around the Nancy Kerrigan and Tonya Harding figure skating scandal in the run-up to the 1994 Olympics. If you're of a certain age, though - whether you're a figure skating fan or not, and I am decidedly no fan of figure skating - the Shakespearean story of Harding and Kerrigan still engages, and still grabs peoples' attention, twenty years later. Why, though?  Why, twenty years later, in a sport I care little, does the story still grab me?  Why did I spend time out of my life watching dueling NBC and ESPN documentaries on the subject, and Google multiple stories about Jeff Gilooly , idiot "bodyguards", and the whole sordid affair? I think it's because the story, even twenty years later, is like opium. The addictive story, even now, has everything.  Everything.  The woman that fought for everything, perhaps crossing over to the dark side to get her chance at Olypic Gold, vs. the woman who...