On September 14, I was fortunate enough to have the opportunity to present at the RIT Sports Analytics Conference. The project was the culmination of over six months of work, and a year of working with Michael Lopez, the director of analytics for the NFL and my mentor through the Hockey Graphs Mentorship Program.

I wanted to write an article summarizing my presentation, as well as discussing improvements I plan on making. Additionally, I plan on diving deeper into the Detroit areas of the data for this site.

When I assembled my presentation for the conference, one of my main goals was to make it accessible to anyone, regardless of your background in analytics. I maintain that goal for this article, so even though it will be lengthy, I think that anyone can get something out of it.

An appendix at the end of the article will include the data used, as well as links to the slides shown at RITSAC and a video that includes my presentation, in addition to the rest of the presentations from the conference.

Overview

When I applied for the HG mentorship program, one of the requirements was to propose a long-term project you were interested in working on. Two seasons ago, I had worked on a way of examining the value players gave to their team compared to how much money they cost. I proposed an improved version of that for my long-term project, which became the presentation.

I started with the question “Which general managers are best at getting value for the contracts they sign?”

Method

Compiling Data

First, I put together a dataset with every contract signed in the cap era, from 2005-06 through last season (2018-19). Since the rules were so vastly different, pre-salary cap era contracts were not included

Next, I expanded the contract data so that each season of a contract had its own row. If a player signed a five year UFA contract, that contract would take up five rows. After that, I added the general manager who signed the contract, as well as the general manager for which the player played each season.

For the basic value metric, I used the Evolving Wild Goals Above Replacement (GAR) metric. No model is perfect at measuring the complete impact a player brings to his team, but the EW model is considered one of, if not, the best of the current models.

If you are not familiar with it, GAR measures the number of goals a player contributes above what a replacement level player would be expected to contribute. Two key factors to know about their model is 1) it’s descriptive, not predictive, and 2) it’s a counting statistic, so it’s influenced by how much someone plays.The Evolving Wild writeup of their WAR/GAR model is here: Part 1 | Part 2 | Part 3

The EW data starts at the 2007-08 season, meaning that player impact for the 05-06 and 06-07 seasons are not included in the final dataset I used for the presentation. If they add GAR data for those seasons, I will incorporate that.

If a player played for multiple teams in a season, there is a row for each team.

Since every player who played for a team in a season added positive or negative value, even if they only played in one game, I did not set a Time on Ice or Games Played cutoff.

I eliminated goalies for the time being. One of the next steps (which I’ll detail in another section) is to figure out the best way to incorporate them.

I took out information from the lockout season of 2012-13, since the season was much shorter than all the rest.

Lastly, I created a second dataset that combined seasons back into contracts, so each row was one contract. For contracts that included the lockout season, the TOTAL.SEASONS.PLAYED column is used to indicate how many complete seasons were played during that contract.

For the presentation at RITSAC, I used completed contracts. First, I used contracts over seasons because a player can have a good season or two, but the team signs the player for the entire contract. Secondly, because comparing, for example, two UFA contracts signed for six years doesn’t work very well if one contract has been completed and the other has only had two years played. You expect that on average, a player’s value will decline over the life of a UFA contract, so looking at the per-season value of years 1 and 2 will likely look very different than if you include years 3-6.

Assessing Value

To measure value, I used a metric that I referred to as “Efficiency.” It was simply GAR / cap hit percentage for that season. This provided a look at how much value a player provided based on how much that player was paid. Since it uses % of the cap, it can be easily compared across seasons.

The problem with this is that while it’s easy to understand in theory, it’s difficult to contextualize the differences between players. So if one player is at 0.65 and one player is at 0.70, the player with the higher number provides more value compared to what they are being paid that season, but how much more? It’s not very easy to use.

I went back to an article written by Gabriel Desjardins for Arctic Ice Hockey for an idea on how to figure out a better way. The short version is that he used the total salary paid in a season, combined with a “replacement level team” (a team consisting of 23 players each paid the league minimum) to estimate how much 1 Win Above Replacement (WAR) cost that season.

I recreated his process, using data from Capfriendly for the last four seasons in addition to re-running the data that he used for his article. One change since then is that adding the additional seasons from the time he wrote the article, the average points for an NHL team each season has increased from 91 to 91.5.

There were two seasons left over for which I didn’t have access to data. To estimate the total salaries spent for those seasons, I ran a linear regression using the total spent each season if each team spent to the cap. The r^2 value for that regression was 0.9925, meaning that the variation in “league cap max” explains 99.25% of the variable I was missing for those two seasons.

(If anyone has access to the actual total salaries for 2013-14 and 2014-15, feel free to send them my way, and I’ll update everything)

By converting WAR to GAR using Evolving Wild’s conversion factor for each season and changing out AAV for percentage of the cap each season, I created a metric that I called “Anticipated GAR.” I used GAR instead of WAR since the numbers are higher and it is therefore easier to see the difference between players. It varies from season to season, but for the EW model, it’s typically roughly 5.5 GAR = 1 WAR. I didn’t want to use “expected GAR” because to me that evokes a link to something like expected goals, which entails much more sophisticated calculations than what I’m doing here.

“Anticipated GAR” measures the amount of GAR a player should be expected to produce based on how much of the cap that player is paid that season.

The other term you’ll need to know to follow the rest of the article is “Excess GAR,” which is simply Actual GAR minus Anticipated GAR.

In short, Excess GAR measures how much extra value a player provided for his team compared to what you would anticipate from a player paid that much.

Findings

I built the presentation around the group of high-end RFAs who were holding out at the time. Mitch Marner, Patrick Laine, Mikko Rantanen, Matthew Tkachuk, and the player I would come to think of as the most valuable, Brayden Point.

The reason I made this choice is because since I started getting into the field of hockey analytics, I’ve been interested in contracts, specifically why it seems like GMs make what seem to be such obviously poor decisions around July 1.

In the few years I’ve been reading other people’s work, I kept coming back to the same idea, which is that not only is the system broken, it is designed to be broken. It’s designed to underpay players when they are in their prime, and has the effect of overpaying players when they are past their prime.

ELCs are basically boilerplate. There is a limit on salary and performance bonuses, and since teams only give the max ELCs to the top tier of prospects, it becomes very difficult to overpay these players. Any overpays are limited by the constraints of the contracts, so from a GM level, it’s very simple: Draft well, sign them, and get them in the lineup early enough in their career to take advantage of the years you get to pay them very little.

From the team side, it’s great! But for the players, it’s not great at all. Even the top players are limited to what they can earn at this point in their careers.

So then they become restricted free agents. They technically have some leverage, but it’s very little. Eventually when they gain arbitration rights, they have more leverage, but they still don’t have a lot of leverage to get paid what they are worth.

It’s not until unrestricted free agency when the leverage swings towards the player. At this point, according to aging curves, the average player is on the decline, since most are at least 27 years old at this point. So these players, many of whom have been underpaid during their prime years, now have leverage and are looking to make up for lost time.

Data

Some General Things First

Up first is Excess GAR per season played for all completed contracts in the dataset. For this chart and the next few, the blue line is the trend line. One thing that’s important to remember is that it’s harder to have an Excess GAR above 0 the further you go to the right of this chart.

Like I said above, it’s very hard to overpay on entry level contracts. Because of the limitations on what a team can pay for these contracts, these include nearly all the top contracts for Excess GAR per season played.

While the trend line was positive as the yearly cap percentage increased for ELCs (which makes perfect sense since the highest value entry level contracts go to the top tier of prospects), the trend line reverses when looking at RFA contracts. These are contracts signed when a player was an RFA, and they could extend into years the player would be a UFA if he was up for a new contract.

There’s still a lot of value to be had with this group, even as the yearly cap percentage increases, but the number of sub-zero contracts increases once you pass about 5% of the cap, with a notable increase just before 10%.

The downward trend-line unsurprisingly becomes more pronounced when looking at UFA completed contracts. The blue line is the UFA trend line, while the red line is the RFA trend line from the previous graph. I wonder if anyone can guess the player with the major outlier contract in the top right. I’ll tell you later.

This next graph puts the three types of contracts side by side for an easy comparison. As I said before, you can’t really compare the entry-level contracts to the rest in terms of GM performance because of the limitations on what they can offer to their players. The average UFA contract provides negative value to a team, and while it’s certainly possible to get good, even great value from these contracts, more RFA contracts have positive value, and more have very good value.

Before I get to the GM data, I wanted to take a look at what the data looks like if we break it down by the length of contracts. First is RFA contracts. You can see how much more popular 1-2 year deals have been than their longer counterparts. While the general potential for negative value contracts increases as you increase the length of the contract, you can see that there are still a good deal of positive value contracts with longer term.

The red line shows the average for that length of contract. Keep in mind these are completed contracts, so each season we’ll get more data to add.

Moving to the UFA contracts, you see the same general trend downwards, but going at a more substantial rate. This lines up with expectations, since you expect the average player’s performance to decline in his UFA years based on aging curves.

Note: The outlier with the 10 year contract is an error, please ignore that.

Earlier I asked if you could guess who the huge outlier was with a UFA contract over 10% of the cap. If you guessed Pavel Datsyuk, you’re correct. Since the cap era started, only eight completed contracts with an average cap percentage of 10 or higher had a positive Excess GAR per Season. Datsyuk had two of them.

GM Value

Now it’s time to get to what people have been waiting for. Which GMs are best at getting value from the contracts they sign?

I divided contracts based on the status of the player when they were signed: ELC, RFA, and UFA. If a GM worked for multiple teams, those are counted separately. For each group, I cut off the bottom group of GMs who signed the fewest completed contracts in the dataset. For ELCs, that was those who signed fewer than 10, while the cutoff was 15 for the other two groups. The data is presented per contract.

For each group, you’ll first see the overall picture for context, then I’ll zoom in on the top 10 and bottom 10.

ELC

Right around +3 Excess GAR per Contract is where the top GMs start to separate themselves from the pack. Like I said before, it’s not hard to get positive value from these contracts. Because of this, it’s interesting to note that two Vancouver GMs are at the bottom of the list.

Since ELCs are nearly always signed by the team drafting a player, this says more about drafting strength than do the other two groups of contracts.

One interesting thing to note about the bottom 10 group is that while there is no overlap between teams when comparing the top 10 to the bottom 10, Ray Shero appears on the top 10 for Pittsburgh and the bottom 10 for New Jersey.

Moving to RFA contracts, you can easily see that far fewer GMs get positive value on average in this area. The combination of the average player being at or close to his peak performance during at least some of the years on an RFA contract along with the team having leverage as described earlier means that it should be harder to overpay on these contracts.

The top group has some interesting names in it. While Ken Holland signed some bad (or very bad) UFA contracts in Detroit, he typically did very well with RFAs. It’s interesting to see that the new GM in Detroit, Steve Yzerman, did even better in Tampa Bay.

Looking at the bottom 10 GMs here, we see that just like Shero with ELCs, Jim Rutherford is in the top ten with Pittsburgh and the bottom 10 with Carolina. Also, similarly to seeing Vancouver have two GMs in the bottom 10 for ELCs, Edmonton shows up here twice in the bottom 10 for RFAs.

Based on the earlier graphs showing UFA contracts, the shape of this next graph shouldn’t be too surprising, although it is incredible that only two GMs have positive value per contract for UFA contracts. Also, notice the jump backwards for the bottom 4 GMs here: Bryan Murray, Darryl Sutter, Scott Howson, and Paul Holmgren (especially the bottom 2). It seems like the plan for GMs when it comes to UFAs is to try to limit their losses as much as they can.

One thing to point out with the top 10 GMs for UFA contracts is to remember that these are completed contracts. So if you are wondering how Peter Chiarelli is on this list for Edmonton, keep in mind that the Milan Lucic contract doesn’t expire until 2023.

In addition to Paul Holmgren being at the bottom of this group, he is also the only GM to appear on all three bottom 10 lists.

If you were wondering what GMs are on multiple top or bottom 10 lists, here you go.

While I don’t think it makes sense to do some sort of ranking system, the above chart does show a good approximation of a top group and bottom group.

When I presented, I closed by showing why Brayden Point had the best case of the group of RFA holdouts for a big payday. The graph below shows the 11 completed entry-level contracts in the cap era with an Excess GAR per season of over 15.

Below is the top 25 for context. When you look at the names that Point is above, it’s not hard to think that he’s going to earn his money, even at a high cap percentage. Comparing Point’s new contract to the one Mitch Marner signed, it’s pretty easy to say that Tampa Bay got great value, even taking taxes into consideration.

What’s Next?

There are a handful of things that I plan on doing to improve this data.

I previously mentioned that goalies are not included. I plan on working on including them in a future version.

One thing I ran out of time to figure out before the presentation was how to deal with trades. Right now, a player’s value is linked to the GM who signed the contract. I plan on working on a way to incorporate trades to the data.

Lastly, the ELC data currently uses AAV to calculate the cap hit, when in reality a team has to pay performance bonuses in some cases that count against the next year’s cap. Capfriendly lists the bonuses, but does not indicate if they were earned, so I need to find another way to figure that out.

For Red Wings fans, I plan on digging deeper into this data as it applies to Detroit in some future articles.

Here is a link to my datasets and slides from the presentation. If you notice any errors in the data, please let me know at the email address in my profile.

Thanks

While I put in a lot of hard work on this project, I had a lot of help. At the risk of leaving someone out, the following people or sites helped out along the way with this project, either directly or indirectly” Michael Lopez, Prashanth Iyer, CJ Turtoro, Josh and Luke from Evolving Wild, CapFriendly, Alison Lukan, Shayna Goldman, Matt Barlowe, Will Scouch, Meghan Hall, Dom Luszczyszyn, Matt Cane, Ryan Stimson, Micah Blake McCurdy, Chris Watkins, and JJ from Kansas.