With the 2013-2014 NHL regular season underway today, I thought it would be a good time to look at how consistent NHL goalies have been since the 1997 regular season including data through last regular season. In order to do this, I looked at NHL goalies who started at least 10% of the regular season games for a given season for two consecutive seasons. That is to take care of issues where goalies are injured or were not considered by the team to be productive enough to be an integral part of the team. While the 10% cut-off may seem to high or too low, it is a good starting point to use so that I do include both the starting and back-up goalies for a given team in the analysis.
Thus, what I need is a measure for evaluating NHL goalie performance and a way of measuring consistency from one season to another.
In a paper published in the Journal of Sports Economics, David Berri and I measured NHL goalie performance. WAA is our measure of NHL goalie productivity
and is based on the absolute value (since a goal against has a negative
effect on team wins) of the marginal value of a goal against divided by
two (since each win is worth two standings points) times the number of
shots on goal that goalie faces times the difference in the save
percentage of the goalie and the average save percentage of all goalies
for that season.
To measure consistency, I ranked each goalie who had started at least 10% of the regular season games for that season and "graded" each goalie based on where they were in the ranking. Specifically, all goalies that were in the top 20% were "graded" as an A, between 21% and 40% were "graded" as a B, and so on. This is the same way we measured NFL QB's, MLB batters and NBA players in chapter 9 of The Wages of Wins.
Then I found out if that goalie started at least 10% of regular season games in consecutive regular seasons and evaluated whether they kept their same grade from one regular season to the next (which I called consistent), whether they moved up or down only one letter grade (which I call near consistent) or if they moved up or down more than two grades (which I call inconsistent).
From the 1997-98 regular season to the 2012-13 regular season, this happened 758 times (consecutive regular seasons starting at least 10% of the games for a goalie). Of those 758 observations, NHL goalies kept the same grade about 25% of the time, moved up or down one grade about 35% of the time and moved two or more grade from one regular season to the next about 40% of the time. Since all that I am looking at here is whether they kept the same grade (i.e. A's or F's in consecutive regular seasons), this seems to be a rather volatile amount of performance, and hence I would conclude that NHL goalies on the whole are rather inconsistent.
Of the 192 NHL goalies that kept their same grade, 34% were "A's", 13% were "B's", 18% were "C's", 14% were "D's" and 21% were "F's". Of those that moved up or down one grade 16% of the total (121 observations) moved down one grade and 19% (or 141) moved up one grade.
Over the last two regular seasons performing the same analysis results in 52 observations and NHL goalies keeping their same "grade" 21% of the time, moving up or down one "grade" 33% of the time and moving two or more grades 46% of the time. Again, lots of volatility from regular season to regular season.