Some FHM 'Draft Analysis'

tomkmb · 08-21-2024, 05:54 PM

I'm not sure if anyone else follows Byron Bader on Twitter? I enjoy his draft posts and discussions around hockey prospects et al - much like all Twitter analysts he posts cool visualisations and data driven insights. A thread I found particular interest in was this one. A chart displaying the 'hit rate' on draft picks by NHL teams over a 24 year sample set as defined by Bader's specific metrics.

Here is the summary in a simple infographic:

It got me thinking about FHM (not that I need much help in that respect, especially when it comes to Hockey prospect development and drafting) and I'm always on the lookout for projects to practice coding to tide me over, so I set out to produce something similar using in game data.

I wanted to include only (or at least mostly) game generated players, as there is always a large discussion around the qualities of draft pools that are generated by the game versus the real life classes that you get for the first 6 or so seasons. So the sample size I went for is from 2028 until 2052, then simmed on a further 10 years to keep in line with the duration of Bader's sample set (1990-2014 as presented in 2024).

This involved some painstaking simulation sessions, especially once you get into the 40's and 50's but eventually I had the data I needed. Some time later I managed to corral it all together and begin to figure out how I might go about making sense of all the info at my fingertips.

In the end it was pretty simple, the most complicated part is navigating some of the idiosyncrasies present in the csvs themselves, though I'm hopeful this will continue to be worked on and improved as they already have come so far since their first re-work a couple of iterations ago.

So when working on the final step for this, the presentation, it got me thinking about what this data is actually telling me...what was the point of getting this all together? Well, I simmed all of this as an observer, so there is no human influence in this universe, it's all AI drafting of game generated prospects, so I guess it's perhaps a little commentary on both of those things...or neither and I've just wasted my time but had a hell of a time doing it, I'll let you be the judge.

Anyway, without further rambling, here's a replication of the above infographic but using FHM data - for anyone who may find it interesting:

So there we have it! I welcome any insights or things you may notice from irl versus in game.

Some small footnotes I should say:

As far as I know Bader's total dataset is not publicly available, so when it came to comparing total prospect numbers I didn't have much to go on.

My total eligible pool of players to categorise was 2658, meaning ~110 players from each class of the 24 year sample set went on to make at least a single NHL appearance. That's 49.1%.

For anyone who may want to take a look at the final dataset, it's here.

If anyone wants to take a look at the save file or if you're a real masochist, the Jupyter Notebook, then let me know and I'll make those available too (disclaimer, I'm not a pro coder).

Enjoy!

Castleview · 08-22-2024, 03:26 AM

That's pretty interesting - the FHM figures are probably around 75% higher
across every draft round. I'm guessing that's mostly because FHM drafted players
will obviously sustain a hockey career regardless, while in real life most
players will give up and move on to doing something else relatively quickly.

It would be interesting to see a comparison using FHM historical
versus actual data.

#4BobbyOrr · 08-22-2024, 04:37 AM

FHM is naturally going to produce more stars by Bader's standards (which are incredibly flawed as he simply goes by PPG) as the scoring envitornment now is a lot higher than it was for those drafted between 1990 and 2014.

I'm curious to know how many franchise players are drafted outside of the top 15 in FHM. It feels like it's much harder to draft top 10 positional players outside of having a high pick. The Kucherov/Weber/Chara/Stone type of picks don't seem to exist, or at least not that I am noticing.

GMO123 · 08-23-2024, 07:19 PM

Why does left chart have 68.7% for an NHLer and right chart have 100%?

Are they measuring something different?

Sebastian Palkowski · 08-24-2024, 04:04 AM

The right table are separate picks from the first round, left side is total 1. Round. So 68.7% of all 1. Rounders become a NHLer but 100% of all 1 OA Picks becomes a NHLer.

Quote:

Originally Posted by GMO123

Why does left chart have 68.7% for an NHLer and right chart have 100%?

Are they measuring something different?

Sebastian Palkowski · 08-24-2024, 04:07 AM

Some time back I did a deep analyse of Draft Quality and it showed exactly the problems described here and in the many other threads (+ some additional problems in that area as well). I gave this info to Jeff and he told me he will make changes to player creation and development to improve the situation. I will report back when I get his stuff and run another test with the changes.

tomkmb · 08-24-2024, 11:24 AM

Quote:

Originally Posted by #4BobbyOrr

FHM is naturally going to produce more stars by Bader's standards (which are incredibly flawed as he simply goes by PPG) as the scoring envitornment now is a lot higher than it was for those drafted between 1990 and 2014.

I'm curious to know how many franchise players are drafted outside of the top 15 in FHM. It feels like it's much harder to draft top 10 positional players outside of having a high pick. The Kucherov/Weber/Chara/Stone type of picks don't seem to exist, or at least not that I am noticing.

Yeah, I agree. Hockey prospect evaluation is a lot more nuanced than can be accounted for but I still get tempted into it with things like this for fun. If you've got some criteria that you'd like to define for what constitutes a franchise player in FHM I can try to take a look and query the dataset to see if anyones in there that matches....just from memory one of the guys who was in the top 5 points scored was drafted at like 3.12 I think, that was the biggest surprise I noticed at the top.

tomkmb · 08-24-2024, 11:24 AM

Quote:

Originally Posted by Sebastian Palkowski

The right table are separate picks from the first round, left side is total 1. Round. So 68.7% of all 1. Rounders become a NHLer but 100% of all 1 OA Picks becomes a NHLer.

Yep, thanks for clarifying Sebastian.

tomkmb · 08-24-2024, 11:26 AM

Quote:

Originally Posted by Sebastian Palkowski

Some time back I did a deep analyse of Draft Quality and it showed exactly the problems described here and in the many other threads (+ some additional problems in that area as well). I gave this info to Jeff and he told me he will make changes to player creation and development to improve the situation. I will report back when I get his stuff and run another test with the changes.

This is cool - glad to know these sorts (and I'm sure more complex sorts) of analysis go on behind the scenes too - hopefully Jeff is cooking up something for the player generation/development algorithm for the next installment(s) of the game.

artlan · 01-22-2026, 03:17 PM

Hi Sebastian, Jeff -

Sorry to bring up an old thread, but I was wondering if/how this was addressed/improved in the latest iteration. Thanks!

artlan · 01-22-2026, 03:17 PM

Hi Sebastian, Jeff -

Sorry to bring up an old thread, but I was wondering if/how this was addressed/improved in the latest iteration. Thanks!

08-21-2024, 05:54 PM	#1
tomkmb All Star Starter Join Date: Jan 2021 Posts: 1,674	Some FHM 'Draft Analysis' I'm not sure if anyone else follows Byron Bader on Twitter? I enjoy his draft posts and discussions around hockey prospects et al - much like all Twitter analysts he posts cool visualisations and data driven insights. A thread I found particular interest in was this one. A chart displaying the 'hit rate' on draft picks by NHL teams over a 24 year sample set as defined by Bader's specific metrics. Here is the summary in a simple infographic: It got me thinking about FHM (not that I need much help in that respect, especially when it comes to Hockey prospect development and drafting) and I'm always on the lookout for projects to practice coding to tide me over, so I set out to produce something similar using in game data. I wanted to include only (or at least mostly) game generated players, as there is always a large discussion around the qualities of draft pools that are generated by the game versus the real life classes that you get for the first 6 or so seasons. So the sample size I went for is from 2028 until 2052, then simmed on a further 10 years to keep in line with the duration of Bader's sample set (1990-2014 as presented in 2024). This involved some painstaking simulation sessions, especially once you get into the 40's and 50's but eventually I had the data I needed. Some time later I managed to corral it all together and begin to figure out how I might go about making sense of all the info at my fingertips. In the end it was pretty simple, the most complicated part is navigating some of the idiosyncrasies present in the csvs themselves, though I'm hopeful this will continue to be worked on and improved as they already have come so far since their first re-work a couple of iterations ago. So when working on the final step for this, the presentation, it got me thinking about what this data is actually telling me...what was the point of getting this all together? Well, I simmed all of this as an observer, so there is no human influence in this universe, it's all AI drafting of game generated prospects, so I guess it's perhaps a little commentary on both of those things...or neither and I've just wasted my time but had a hell of a time doing it, I'll let you be the judge. Anyway, without further rambling, here's a replication of the above infographic but using FHM data - for anyone who may find it interesting: So there we have it! I welcome any insights or things you may notice from irl versus in game. Some small footnotes I should say: As far as I know Bader's total dataset is not publicly available, so when it came to comparing total prospect numbers I didn't have much to go on. My total eligible pool of players to categorise was 2658, meaning ~110 players from each class of the 24 year sample set went on to make at least a single NHL appearance. That's 49.1%. For anyone who may want to take a look at the final dataset, it's here. If anyone wants to take a look at the save file or if you're a real masochist, the Jupyter Notebook, then let me know and I'll make those available too (disclaimer, I'm not a pro coder). Enjoy! Last edited by tomkmb; 08-22-2024 at 05:19 AM. Reason: Slight amendment to figures as I'd forgotten to put an end date on the sample size...d'oh!

08-24-2024, 04:07 AM	#6
Sebastian Palkowski Developer Join Date: Mar 2002 Location: Hamburg, Germany Posts: 5,430	Some time back I did a deep analyse of Draft Quality and it showed exactly the problems described here and in the many other threads (+ some additional problems in that area as well). I gave this info to Jeff and he told me he will make changes to player creation and development to improve the situation. I will report back when I get his stuff and run another test with the changes. __________________ Buy Franchise Hockey Manager

08-22-2024, 03:26 AM	#2
Castleview All Star Reserve Join Date: May 2020 Location: Scotland Posts: 921	That's pretty interesting - the FHM figures are probably around 75% higher across every draft round. I'm guessing that's mostly because FHM drafted players will obviously sustain a hockey career regardless, while in real life most players will give up and move on to doing something else relatively quickly. It would be interesting to see a comparison using FHM historical versus actual data.

08-22-2024, 04:37 AM	#3
#4BobbyOrr Minors (Double A) Join Date: Nov 2015 Posts: 168	FHM is naturally going to produce more stars by Bader's standards (which are incredibly flawed as he simply goes by PPG) as the scoring envitornment now is a lot higher than it was for those drafted between 1990 and 2014. I'm curious to know how many franchise players are drafted outside of the top 15 in FHM. It feels like it's much harder to draft top 10 positional players outside of having a high pick. The Kucherov/Weber/Chara/Stone type of picks don't seem to exist, or at least not that I am noticing.

08-23-2024, 07:19 PM	#4
GMO123 Major Leagues Join Date: Nov 2011 Posts: 421	Why does left chart have 68.7% for an NHLer and right chart have 100%? Are they measuring something different?

01-22-2026, 03:17 PM	#10
artlan Minors (Single A) Join Date: Oct 2015 Posts: 83	Hi Sebastian, Jeff - Sorry to bring up an old thread, but I was wondering if/how this was addressed/improved in the latest iteration. Thanks!