All prospect rankings are bad... except mine
It is finally the moment, the amazing time where everyone releases their prospect rankings ahead of the NHL Draft. From analytical savants, to the public scouts, to the the people who just see themselves as fans. Whether you are a professional, or a simple hockey fans who just want to look smart in a hockey chat, it is the time of the year where you have the opportunity to give the product of months of scouting prospects summarized in the holy grail : your prospect ranking.
Let me give you the ingredients for a perfect prospect ranking that will earn you the praise from the nobles of the hockey scouting community on twitter :
Do what everyone does for the top-6 (something like : Bedard, Fantilli, Michkov, Carlsson, Benson, Smith)
Add some random european prospect top-10 because “he is one of your favourite”
Drop a North-American prospect to late 20’s because “he has bad skating and everyone overrates him”
Now, you have the perfect recipe to cook a perfect prospect rankings list.
If you did not understand the irony, I declare that the introduction was made on an ironic tone. The goal is not to laugh at people who make prospect rankings and shame their work. I rather focus on how to digest those lists that are appearing on a daily basis on our feeds. I did not ask everyone to show their lists, but I am still exposed to them, and it might have an influence on my personal opinions to be in touch with some kind of consensus. This phenomenon has a name in psychology : social influence. It affects people at different levels, but it has a lot of impact in hockey conversations, especially prospect conversations, mainly because people are talking about things they do not necessarily know about and fill the gaps with elements they have heard outside their own experience/research.
Prepare yourself for prospect assessment
I think that when it comes to prospects, we need a lot of humility. Aside of the major outliers — Generational players, prospect assessment is generally making betting that a 17 years old hockey player is a hit on his 0.0001%-10% chance of turning into a NHLer. Usually, these odds come a load of external factors that are not taken into consideration in models (character of the player, injuries, physical condition, etc.). So it is fair to say that you should start that feeling that, as much as you are confident in your top-5, there is a good chance that things do not go as planned. You also have to remember why you are assessing prospects. The reason why you are doing it will have an influence on your work. Watching prospect tapes just to have an idea of who they are is a very different task than working for a NHL team and providing reports based on what you have watched. I feel that you work in hockey more smartly when your work has real repercussions.
So, the article is not about complaining about Corey Pronman’s latest article… at all, and I suggest you to ignore his articles if you do not like them. I think the best draft ranking is the one you make yourself.
The Cons of NHLe
As someone who worked on tools to assess prospects, I feel that a lot of people who support our work are overly confident in the insights our models suggest. This is cool because people support our work, but the reality is that our tools are there to support scouting and not to take over or replace scouting as a whole. And NHLe is a perfect example of the beauty of analytics in prospecting… as well as how horrible it can be.
First, let me explain NHLe so that we are all on the same page. It is a measure created a while back by Gabe Desjardins (one of the Builders™ of Analytics in hockey) to put all point productions in one same unit of measure regardless of the league the player played in. It is amazing because we can now compare, with points, any player in the world. So we are all understanding that NHLe is simply an estimation of a player’s point production in NHL points? Right. The idea to put all players point production on the same unit of measure is really innovative and it helps to contextualize point production. For example, it shows how big is the difference between scoring 0.5 point per game in the NHL and in the AHL. The introduction of NHLe really changed scouting in hockey. Few public tools emerged of it and NHLe essentially became the poster stat for analytical research in prospecting.
Now, let’s do this exercise. We have 2 amazing prospects that had amazing DY seasons. Who would you rather select with a 1st overall pick if they were available in the same Draft?
Easy, right? Prospect A is having a better production in a much more difficult league while he’s significantly better than his teammates and the other players in the league. Prospect B plays in a easier league and we understand that he scored a lot more than anyone else in the league. The gab between the standard deviation between the scoring relative to the league and the team suggest that prospect B also had really good teammates. If you fall in the super easy trap I set for you, you should pick prospect A.
Now let’s see who those players are…
Now that you have the names, you would probably pick Connor McDavid first overall as you should. The goal of this exercise was to show that solely basing your evaluation on NHLe can be dangerous.
Here is an extensive list of potential flaws of NHLe — this is a list of the flaws of my first model (for the next time you want to dunk on it) :
Model does not account for variation of scoring rates from a year to another in a given league and what cause leagues scoring to vary.
When the model is applied to a player’s point per game, it does not give importance to the sum of game played and points. I think it does not correctly estimate the value of an additional point in a given league.
Even if we account for scoring variation in other leagues, the NHL (target league) is not exempt from scoring variations.
Model does not account for how better a player is from his peers.
Not enough emphasis on uncertainity
Only about point production
Here are the pros :
Easy to understand
Really good for complementing scouting work
If you are curious enough, it creates new questions to answer
Can be used in other models and is performing quite well
But do not worry, for most of these problems public scouts do not have answers for them. So make your own research. NHLe has its flaws (that we try to tackle on a daily basis), but it is still one of the best tools for prospecting publicly. Great hockey analysts like me work on additionnal tools using publicly available data to make better assessments of hockey prospects for future NHL Drafts, so keep your eye open for the next time talking about prospects will be cool again (in December for the World Junior Championship).
The Case of the Eye Test
You will happy. As much as Iike coding, my boss sent me scouting a few times this season. I watched a lot of film and I had the opportunity to attend a few games to provide insight on specific players to my team. The really fun moment was when we signed players I was sent to scout. Scouting is a real job. It is a lot of work when done properly. Most of the times, I had to do film or do scouting, I was paired with at least one other person. We all watched the same games, but we usually all came up with different reports. Elements in the game stood up to me, other things stood up to my collegues. We watched the same game. Same thing happenned when we were asked to watched specific players. My collegue came up with details that I did not necessarily noticed. We also had some disagreements on multiple topics. This is why I feel that reading one person giving his thoughts on all prospects is not my prefered way of learning about prospects. I need first to trust the skill level (in scouting) the person I am about to read. Then, I want to watch the prospects myself and I want to have a discussion and challenge ideas using data and video. Lastly, we need to stay away from the easy prospect coverage lines : "Good skating, comes from good family, great leadership, comparable to [random NHLer from a random team you've seen twice in your life]". For me, this is the optimal way of doing scouting publicly. If you work with a team, you want to assess team needs because at some point player talent will be roughly the same for everyone and you will need to make a choice on who you want to develop. This choice could be be influenced by few factors like how crowded your team is at a specific position. I think doing film and going to games is super important for prospect assessment, but we need to be really careful when reading the work other people did.
Without further ado…
I told you in the title that all prospect rankings are bad except for mine, so here is my top-5 :
Connor Bedard : You already know him
Adam Fantilli : He is good offensively
Matvei Michkov : Really good player that did not get as much exposure as his peers due to political situation
Leo Carlsson : Big and produced a lot in a league where players of his age usually struggle to get their name on the scoresheet
Zach Benson : Small excellent two-way player who led a team, with few 1st round picks, in scoring.
This is the only good list.
Make sure to follow me on Twitter because I will have good news to share soon.
Now you are ready for Drafting season. Do your own research. Subscribe to my Substack. Have fun.