Evan Green and Michael Menz
Fall, 2015
The New England Patriots have been the class of the NFL for the past 15 years. Part of their success has been attributed to Bill Belicheck’s aggresiveness on 4th downs. In the 2012 AFC championship game against the Baltimore Ravens, the Patriots faced a fourth and goal from the 1. Tom Brady took the snap and jumped over the middle of the line for a touchdown - a play called the quarterback sneak.
The following season, the Patriots faced a fourth and goal from the 1 against the Steelers. This time, Tom Brady handed the ball to Brandon Bolden who was tackled before crossing the goal line.
Did the Patriots make a better play call in the first case or did they just have better luck? There is a consensus in the sports statistics community that “going for it” on fourth and one is almost always the right call. The New York Times popular 4th down bot uses win probability and expected points models to live tweet its prescribed decision during NFL games to interested fans and is typically more aggressive than NFL coaches. However, there is not nearly as much research into what play should be run once a team decides to leave their offense on the field. Our analysis of over 3000 fourth and 1 attempts shows that the quarterback sneak is indeed far more effective than other play calls in obtaining a first down. Therefore, NFL coaches may be able to be even more aggressive in attempting to convert on 4th downs than previously thought by calling more successful play types.
We looked at all 4th and 1 down attempts since the 1998-1999 season, the earliest year data was available, until the 2015-2016 season. This data set includes 3181 plays, about 180 plays per season.
The table below summarizes part of our results. The overall conversion rate during this time frame was 65.7%. This relatively high success rate is a large reason why it is so often considered the smart decision to go for it in short yardage situations.
| Play Type | Conversion Rate |
|---|---|
| All Plays | 65.7% |
| Passing Plays | 58.2% |
| Running Plays | 67.9% |
However, we can look more granurally at conversion rate. Running plays, which include quarterback sneaks, are converted more frequently than passing plays. The true trends stand out even more emphatically with a more detailed investigation of play calling. In the table below, it is easy to see how much more successful quarterback sneaks are than all other play types. 82.8% is significantly higher than the 62.0% of all other plays and suggests they are underutilized. The lower success rate for running plays by non-quarterbacks illustrates the true distinction is between quarterback sneaks and all other plays, not between runnning and passing plays.
| Play Type | Conversion Rate |
|---|---|
| Quarterback Sneaks | 82.8% |
| Non-Quarterback Running plays | 63.4% |
| All Non-Sneaks | 62.0% |
Furthermore, in the graph below of the conversions rates by year, it is immediately evident how much higher the conversion rates are for quarterback sneaks. The consistency of this effect over every year in the sample is also apparent. This consistency suggests that this difference is not caused simply by chance but something fundamentally different about quarterback sneaks.
Conversion rate isn’t the only metric that is important to consider when evaluating plays. Different play types might also yield different yardage. To analyze plays more holistically, we used Estimated Points Added, with data from http://profootballreference.com. Estimated Points Added (EPA) is a good choice because EPA takes into account the value of gaining a first down, but also the additional value added by gaining additional yards and the negative value from turnovers. In general, positive EPA cooresponds to plays that are considered above average.
The table below shows the EPA of different play types. Again, the .98 EPA for all plays is relatively high and illustrates why it is recommended to attempt to convert in 4th and 1 situations.
| Play Type | EPA |
|---|---|
| All Plays | .98 |
| Running Plays | 1.01 |
| Passing Plays | .86 |
However, just like with the conversion rates, looking at more detail illuminates the trends more clearly. The following table illustrates how valuable quarterback sneaks are. Quarterback sneaks are produce almost twice as much EPA as other play types. The conversion to EPA also makes the importance of these difference more readily digestible than the difference in conversion rates. Quarterback sneaks produces an extra .77 points on average compared to other types of offensive plays on 4th and 1. These additional points could be the differences in winning or losing games.
| Play Type | EPA |
|---|---|
| Quarterback Sneaks | 1.61 |
| Non-Quarterback Running plays | .83 |
| All Non-Sneaks | .84 |
In the graph below of the average EPA by year, similar to the one produced above detailing conversion rates, we can observe a noisier sample than with conversion rates but can still see the striking and consistent benefits advantages of quarterback sneaks compared to other plays. The sample is noiser because EPA is affected by other factors such as field position and whether a touchdown was scored.
These differeneces in the conversion rates and value added are indicators that NFL coaches could improve their teams winning percentages by calling more quarterback sneaks.
In the optimal set of plays, the marginal value added by having another play of each play type should be equal. Otherwise a better set of plays could be created by choosing more of the play type with the highest marginal value. This theory for optimal sets is widely used in economics and game theory and is well-established. While this type of analysis of long run averages is not enough to determine marginal value, a task which would require control of an NFL team’s play calling (something the authors’ do not have presently), the vast advantages sneaks have both in conversion rate and EPA suggest that they are underutilized.
Furthermore, increasing the percentage of quarterback sneaks would cause the defense to shift toward defending them more, potentially increasing the chances that other play types are successful as well.
However, we would bring up some potential concerns as well. Play types are not called in a vacuum. Quarterbacks often have the ability to read the defense before the snap and audible between run and pass plays. Therefore, we expect that NFL teams are closer to an optimal set of plays than initially thought, adjusting play types intelligently to the defenses and their own skill set. While this definitely happens, given the numerous situations in which we know NFL teams do not act optimally, there is still room for improvement.
Quarterback sneaks might be particularly susceptible to selection bias. If there are no defenders over the center or it is particularly short yardage (inches instead of a full yard), sneaks may be particularly effective. In fact, on all 4th and 2 plays in the since the 1998-1999 season, quarterback sneaks were successful 89.7% of the time, greater than the 82.8% of the time for 4th and 1. Sneaks were rarer on 4th and 2 plays, happening only 29 times from between the 1998 and the 2015 season, but this shows that sneaks are not only successful for incredibly short yardage situations. Most importantly, quarterback sneaks are so much more valuable and convert so much more frequently than other play types that the effect is likely robust enough even after accounting for these selection biases.
Another consideration is that quarterback sneaks may put quarterbacks, the most important player of any team, at an injury risk but the nature of quarterback sneaks does not expose quarterbacks to blind side or open field hits where injury risks are much higher.