** edit 5/29, I'm releasing a series on these models, the next post (on the AFC East) is found here.
After reading Nate Silver's book I experienced a bit of professional jealousy. Not because Silver is more famous than I am, but because he gets to look at cooler data.
While I'm spending my days looking at customer and underwriting data, the guys at fivethirtyeight.com get to analyze political and baseball data all day. I like my paycheck, so this is all fine by me, but hey, I can at least look at least look at some sports data in my free time right?
Problem is: I don't really like baseball. Growing up in Kansas... Well, just look at the play of the Royals from 1986-2013 to figure out why I don't like baseball. But I do like football.
NERDINESSOpen Question: can we create effective very-early predictive models on NFL data? The data exists and it seems as though it would have some predictive power. What if I set out predict a team's overall record at the end of the NFL regular season (January 2016) before the season begins?
I can obviously create a model, and that model may be predictive, but will that forecast be any good? Some distinct challenges of NFL data:
- Small data set: NFL Teams only play 16 games a year. (versus 162 for MLB, 82 for NBA). Thus we don't have a ton of prior year data to predict with, and that data AND results are both somewhat volatile.
- Very early predictions: We're in late May right now, and the teams haven't even started pre-season games. Some players aren't even signed to their teams.
- Injuries: Injuries play a huge part in NFL performance week to week. Can I really make accurate predictions for the Denver Broncos if I think there's a >50% chance that Peyton Manning will miss 4 or more games?
I went forward creating a statistical model from game-level historical NFL data, going back to 2008. It's a logistic model that predicts the probability that a team will win a game.
MODEL PERFORMANCESo, first question, is the model predictive? Answer Yes. A couple of ways to look at this.
First when the model predicts a 20% chance of a team winning a game, does that team win about 20% of the time? The answer here is yes, and here's a graphical illustration:
|Win Probability on Bottom, Actual Outcome Vertical|
But this is game by game, is the model predictive by aggregated season? A couple of ways to evaluate, by two different base metrics:
- Average model. Does my model perform better than a model where we guess each team would just get the average 8 wins? Answer: yes, it's about twice as predictive as this model.
- Static model. Does my model perform better than a model where each team gets the same number of wins this year over last year? Yes, it's about 2.5 times more predictive than that model.
I know everyone is only interested in who will win the AFC West, but first a few observations from the model:
- Because the NFL season is relatively low N, and a few unprobable results can shift a teams seasonal win total significantly, we find that NFL results are fairly volatile.
- This is evidenced in the "Average Model" outperforming the "Static Model" in essence, in many cases it's better to guess that a team will "return to the mean" next season, rather than continue to perform as they did in prior seasons.
- The model reflects these volatile results. Specifically, total wins last year were much less predictive as a variable than metrics around football fundamentals. Specifically, some metrics that outperformed prior year wins (I specified these several different ways in models): Home/Away, average offensive yards, defensive yards allowed point differentials, turnover ratios, points scored, points allowed.
And finally, for the moment you've been waiting for, the AFC West results.
Basically, my Chiefs stay the same and don't win the division, all other teams return slightly towards the mean. The chiefs are a *bubble* playoff team. Personally, I don't think they'll make it.
I do have these statistics for all other divisions, which I may post at a future date, if they are of interest.
Addendum: R code for prediction versus actual probability segmentation: