Starting in the 2015-2016 season, the NHL instituted a new rule allowing coaches to challenge certain scoring plays. Specifically, coaches would be provided with one challenge that would allow them to challenge if a scoring play was offsides or involved goaltender interference. The penalty for a failed challenge was the loss of the team’s timeout, although this was later revised to a minor penalty for a failed offsides challenge prior to the 2017-2018 season. Finally, the league automatically reviews all goals scored in the final minute of regulation and overtime. With the challenge system now three-and-a-half years old, how successful are individual coaches and their video teams at the challenge system?
Methods
The NHL play-by-play was scraped using Emmanuel Perry’s NHL Scraper searching for events listed as “CHL” or “STOP” with an event description of a challenge. The final database was manually cross-referenced to eliminate duplicates. Finally, each individual event was manually reviewed with the play-by-play, recap, or video to confirm who initiated and won each challenge. The head coach for each game was determined based on the dates provided on Behind The Benches.
Findings
Through December 24th, 2018, there have been 900 challenges, with 290 (32.2%) overturning the call on the ice. Most challenges occur when the score is tied, although teams that are trailing, smartly, are more likely to utilize their challenge.
Interestingly, the success rate of challenges appears to decline significantly in the final fifteen minutes and overtime as coaches may be more apt to challenge any play that could alter the score in their favor.
From the get-go, the challenge system was rapidly adopted by coaching staffs as they challenged nearly any questionable play. This resulted in a number of plays being overturned only after reviewing frame-by-frame sequences for several minutes. As a result, the NHL instituted the delay-of-game penalty for a failed offsides challenge. Looking at the data below, we can see that the threat of a penalty proved to be a significant deterrent for coaches.
Season-by-Season Challenge Data
| Season | Interference Challenges | Interference Win% | Offsides Challenges | Offsides Win% | Total Challenges | Total Win% |
|---|---|---|---|---|---|---|
| 2015-2016 | 152 | 22.4% | 91 | 36.3% | 152 | 27.8% |
| 2016-2017 | 160 | 27.5% | 126 | 32.3% | 286 | 29.7% |
| 2017-2018 | 188 | 29.8% | 71 | 56.3% | 259 | 37.1% |
| 2018-2019 | 75* | 25.3% | 37* | 62.2% | 112* | 37.5% |
Instead of challenging any offsides play that appeared close, coaches picked their spots better, resulting in a substantially higher success rate. In terms of goaltender interference, it appears as if we can lump coaching staffs with the rest of us: people who can’t explain goaltender interference. Nearly three-quarters of goaltender interference challenges result in the loss of a timeout, but that hasn’t stopped coaches from challenging the play. The number of goaltender interference challenges has increased each season, with this season on pace for 169 challenges.
Evaluating challenge success at the coaching level is more difficult as the decision to challenge a play involves multiple people. Members of the video staff are quick to review a replay and provide important information to the head coach, who ultimately makes the decision. At the end of the day, the success or failure of a challenge comes down to a lot more than just the head coach’s intuition. Still, the head coach ultimately bears the responsibility for the challenge and thus I will split out success by individual head coaches rather than teams.
Looking at overall success, we see can see that most coaches have struggled mightily with the challenge. Only six coaches have won more challenges than they’ve lost, with Vancouver’s Travis Green leading the way (+3). On the other end of the spectrum, Detroit’s Jeff Blashill is the most frequent utilizer of the challenge system and also owns the worst differential at -24.
Given that the penalties for a failed goaltender interference challenge (loss of timeout) are far less severe than those of a failed offsides challenge, it’s important to split out which type of challenges coaches are losing. Starting with goaltender interference, we can see that Colorado’s Jared Bednar is by far the most successful head coach, going 9/12 (75%) on goaltender interference challenges.
On the other hand, Blashill is among the league’s worst at -16, followed closely by Toronto’s Mike Babcock (-17). However, is the loss of a timeout all that significant? Since the NHL’s rule change preventing teams from calling a timeout after an icing play, the timeout is generally only utilized to buy rest in a late-game situation,which is difficult to isolate and quantify.
Taking a closer look at the data, we see that the timeout often goes unused. From the start of the 2015-2016 season through 12/24/2018, coaches called 1561 timeouts and lost their timeout an additional 566. Put differently, coaches utilized just 19.5% of the available timeouts when accounting for timeouts lost due to failed challenges. On the other hand, a successful challenge has the potential to take away a goal or put one on the board. Some coaches have recognized this and have taken to use their challenge as a timeout following a goal against. Let’s revisit this plot showing goals either taken off the board or put on the board due to a successful challenge.
Rephrased as “net goals gained from interference challenges”, we see that Blashill shoots to the top of the chart. His challenges have resulted in 12 goals either prevented or added for Detroit. Since the loss of a timeout is a relatively benign penalty, (given that ~80% go unused) one could argue that Blashill has utilized the challenge system “effectively”, by challenging any play that appears remotely questionable. Blashill has challenged for goaltender interference 40 times, 15 more times than the second-place Babcock.
The other interesting aspect to analyze is how coaches have adjusted to the goaltender interference calls being reviewed by the NHL Situation Room in Toronto as opposed to the on-ice referees. Beginning March 28th, 2018, the NHL Situation Room now handles all goaltender interference and reviews the play with the on-ice referees. The change was made due to concerns of inconsistency in how goaltender interference was called. While there’s not enough data to analyze each coach individually (only 91 challenges since the change), we can look at how coaches have faired overall to see if consistency in the reviewer has helped.
Overall, it doesn’t look like the switch to the situation room has helped the coaches, although the sample is small. This will be an interesting development to follow as more data becomes available.
Switching to offsides challenges, we see that more coaches are successful with offsides challenges, although this is heavily influenced by the institution of a delay-of-game penalty for a failed challenge prior to the 2017-2018 season.
Gerard Gallant has been the league’s most successful coach, winning 9/13 challenges (69.2%) for offsides. On the other end of the spectrum, Calgary’s Bill Peters is by far the league’s least successful coach, going just 3/15 (20%). To properly analyze how the league’s offsides penalty influenced the utilization of the offsides challenge, we’ll split the data into pre-rule and post-rule buckets.
League-wide, the number of offsides challenges declined substantially from 127 in 2016-2017 to just 71 in 2017-2018. This year, the league is on pace for 83 offsides challenges. All but seven coaches saw their tendency to challenge for offsides decline after implementation of the penalty. Notably, John Tortorella went from challenging for offsides once every 12 games to challenging just three times since the new rule went into effect (116 games). The question becomes, how much should the threat of a penalty curb the desire to challenge?
Matt Cane of Hockey-Graphs looked at how certain a coach needs to be on an offsides challenge for it to be worthwhile to challenge. Interestingly, a team that’s trailing (score states -1 and -2) should challenge nearly every questionable call, as they need be just 20% certain to have a positive expected value. For a team that’s leading, coaches should exhibit more caution as the game approaches its end, but should still feel good about challenging even if they consider the play to be 50/50. Overall, Cane’s work suggests that coaches are more risk-averse than they need to be with their offsides challenge.
The challenge system was instituted as a potential solution to the growing frustration surrounding missed calls and controversial goaltender interference rulings. However, early use of the challenge system led to frivolous challenges and long delays prompting the league to re-evaluate the rules. The institution of the penalty for a failed offsides challenge has significantly curbed the use of the challenge (arguably far too much) but nothing has been done about goaltender interference. Given that coaches rarely use their timeouts and the true value of a timeout is marginal, coaches should feel empowered to challenge any questionable goaltender interference play. Red Wings’ head coach Jeff Blashill has done an excellent job of this as his frequent challenges for goaltender interference have resulted in a net gain of 12 goals for his team, the most in the NHL. While fans may be frustrated by him losing challenges, in reality, he’s giving his team more opportunities to have a call reversed with the only real penalty being loss of the opportunity to challenge for goaltender interference again later in the game. Future directions with this work would be to calculate a crude “win expectancy added” for coaches based on when they challenge and how successful they are with those challenges.
If anyone is interested in viewing the raw data, it can be accessed here
Special thanks to Emmanuel Perry for making his scraper available and to Shayna Goldman, Mike Murphy, and Alexa Goldman for making the information on coaching timelines available on Behind The Benches.
