Theorems, Formulas, and Evaluations

[chrismeglan]

This isn't necessarily related to OOTP (at least not yet anyway) but I'm looking for a few individuals in the community who have somewhat of an interest in advanced metrics.

If you do, I'm curious of a few things. Has anyone here come up with, or attempted to come up with, their own formula using advanced stats to evaluate players in the game? If so, along what lines were you thinking? What phenomenon did you notice? How successful have you been?

I've basically looking for players who have come up with hypotheses based on statistical data. The main reason being that I've grown more and more interested in the amateur draft, and I believe that amateur drafts, in any sport, do not necessarily have to be crapshoots.

No need to be specific. Any contribution will help me accomplish my goal here. Thanks in advance.

[Cryomaniac]

I haven't personally come up with anything, although I too would be interested if anyone has.

[Hammercraft]

Everything in the game is quantified by nature, because it's a computer game, so the metric that I place the most emphasis on is Work Ethic. In real life it is qualified, not quantified, and isn't as easy to measure. But since the work is already done here I rely on it. I will draft players who have at least passable potential and have a high Work Ethic. They don't always turn out, but players with poor Work Ethic seem to never develop well.

[drhay53]

I don't have my own formula or anything, as I figure the advanced metrics people have spent years coming up with are probably pretty good (I do maintain some spreadsheets that I use to evaluate players based on their ratings in the game though).

I'll just say that if I had to boil players down to the smallest number of stats possible, this is how I evaluate them:

Pitchers:
FIP-

Position players:
wRC+ (offense)
ZR (defense)

edit: basically, pitcher war seems to very strongly correlate with FIP and position player war is some combination of offense and defense. I think of Zone Rating as basically every +- 10 is +- 1 WAR. I'm not sure if that's accurate but it follows from the general rule that 10 runs is 1 win. The units of ZR are runs so I think it's reasonably accurate. How I balance ZR vs wRC+ when I evaluate a position depends on how highly I think defense should be valued there. Generally I won't allow my CF, SS, or 2B to have a negative or even near 0 ZR unless they're a superstar offensive player. If they are a superstar with average or worse defense I'll probably try to DH them or change their position. Just rambling on but these are some of my thoughts.

[RchW]

Originally Posted by Hammercraft

Everything in the game is quantified by nature, because it's a computer game, so the metric that I place the most emphasis on is Work Ethic. In real life it is qualified, not quantified, and isn't as easy to measure. But since the work is already done here I rely on it. I will draft players who have at least passable potential and have a high Work Ethic. They don't always turn out, but players with poor Work Ethic seem to never develop well.

My experience is very different and I'm so glad it is. If WE was as influential as you suggest the game would be a joke.

In real life a lot of this stuff is a manufactured narrative. A quid pro quo between the player the team and the media. After all nobody wants to read about a guy who doesn't work hard and succeeds even though this happens more than people want to admit.

[chrismeglan]

Originally Posted by Hammercraft

Everything in the game is quantified by nature, because it's a computer game, so the metric that I place the most emphasis on is Work Ethic. In real life it is qualified, not quantified, and isn't as easy to measure. But since the work is already done here I rely on it. I will draft players who have at least passable potential and have a high Work Ethic. They don't always turn out, but players with poor Work Ethic seem to never develop well.

I tend to do the same in OOTP, and I do notice a correlation between work ethic and development. Of course, you have to set scouting accuracy to 100% to notice every instance. Let's say this were something you could dive deeper into. How do you think you would measure work ethic?

[chrismeglan]

Originally Posted by drhay53

I don't have my own formula or anything, as I figure the advanced metrics people have spent years coming up with are probably pretty good (I do maintain some spreadsheets that I use to evaluate players based on their ratings in the game though).

I'll just say that if I had to boil players down to the smallest number of stats possible, this is how I evaluate them:

Pitchers:
FIP-

Position players:
wRC+ (offense)
ZR (defense)

edit: basically, pitcher war seems to very strongly correlate with FIP and position player war is some combination of offense and defense. I think of Zone Rating as basically every +- 10 is +- 1 WAR. I'm not sure if that's accurate but it follows from the general rule that 10 runs is 1 win. The units of ZR are runs so I think it's reasonably accurate. How I balance ZR vs wRC+ when I evaluate a position depends on how highly I think defense should be valued there. Generally I won't allow my CF, SS, or 2B to have a negative or even near 0 ZR unless they're a superstar offensive player. If they are a superstar with average or worse defense I'll probably try to DH them or change their position. Just rambling on but these are some of my thoughts.

I'll make note of this. Thanks for your contribution Dr. Hay.

[Hammercraft]

Originally Posted by chrismeglan

I tend to do the same in OOTP, and I do notice a correlation between work ethic and development. Of course, you have to set scouting accuracy to 100% to notice every instance. Let's say this were something you could dive deeper into. How do you think you would measure work ethic?

Time & effort spent working on skills, probably. Intellect probably indicates how effective that effort was towards improving skills. But intellect is misleading, because what you need is baseball smarts, not book-reading and bean-counting ability. You can have guys that are practically illiterate, yet excel because they are bodily-kinesthetic geniuses (Howard Gardner's theory of multiple intelligences, etc.).

[Cobby]

I once did a study of this. It was intended mainly to feed my intuition about the effect of personality traits, not to be scientifically rigorous.

I can't remember now how many years I simulated - I think it was 40 or 50 years. I looked at the amateur draft class for each year and the question was: what percent of the draftees eventually made it to the big leagues, and how does that percentage change for each of the personality categories.

The basic result was that about 14.5% of amateurs in the draft eventually make it to the big leagues. Here were the results for each of the personality categories:

Work Ethic:
Very Low: 5.5%
Low: 13.3%
Normal: 17.6%
High: 19.7%
Very High: 20.5%

Desire to Win
Very Low: 8.5%
Low: 12.7%
Normal: 18.7%
High: 19.2 %
Very High: 16.1%

Intelligence:
Very Low: 13.2%
Low: 13.2%
Normal: 14.8%
High: 15.4%
Very High 15.4%

Greed:
Very Low: 15.8%
Low: 15.9%
Normal: 13.2%
High: 15.8%
Very High 14.7%

Loyalty:
Very Low: 15.4%
Low: 14.7%
Normal: 14.4%
High: 15.3%
Very High: 12.5%

Leadership:
Very Low: 16.0%
Low: 14.3%
Normal: 14.4%
High: 14.5%
Very High 15.7%

Injury:
Fragile: 14.0%
Normal: 13.8%
Durable: 16.0%

Work ethic had by far the biggest effect. The effect of intelligence was modest. I don't hear "desire to win" discussed much, but a Very Low desire to win (lack of drive perhaps, or lack of the "fire in the belly"?) had the second strongest negative effect after "Very Low" work ethic.

On the positive side, "Very High" work ethic had the highest positive effect followed by "High" Work Ethic and "High" (but not "Very High") Desire to Win was next. (I guess maybe someone with "Very High" desire to win might be too intense for his own good...)

One needs to be careful interpreting these results though. For one thing, the sample size was probably not large enough. If each of the traits are distributed normally there will be relatively few at the extremes (Very High and Very Low) so these numbers are bound to be less accurate.

Also I didn't take into account the initial talent assessment at all. Most of the players in these drafts are never going to make it anyway no matter what their personality. And conversely plenty of talented players will make it despite their personalities. I still think talent assessment (how ever you might want to accomplish that) would still be much more important than personality.

The other thing is, my criterion was just whether or not the player made it to the big leagues. By that criterion, a career minor leaguer with a single September cup of coffee is treated the same as a future hall-of-famer.

So I'd probably perform some sort of talent assessment first, then give the players a little boost up or down based on personality traits - perhaps a little more than just a tie-breaker, but definitely not the main criterion... (imho)

[drhay53]

My guess is that the reason you're seeing an inversion between high and very high desire to win is that the parameter is unimportant and it's just a statistical fluctuation.

More importantly, though, I wonder what is the correlation between each personality trait? For instance, if you assign a numeric value to each possibility, and calculate the correlation matrix between all of the parameters.

i.e., desire for win may be correlated with work ethic, and while desire for win on it's own is unimportant, it may be correlated with making it to the big leagues only because it's correlated with work ethic.

Also, do you know the full sample size of players?

I realize you said it was an old test and you probably don't have these numbers, but just throwing it out there.

[Cobby]

Originally Posted by drhay53

My guess is that the reason you're seeing an inversion between high and very high desire to win is that the parameter is unimportant and it's just a statistical fluctuation.

More importantly, though, I wonder what is the correlation between each personality trait? For instance, if you assign a numeric value to each possibility, and calculate the correlation matrix between all of the parameters.

i.e., desire for win may be correlated with work ethic, and while desire for win on it's own is unimportant, it may be correlated with making it to the big leagues only because it's correlated with work ethic.

Also, do you know the full sample size of players?

I realize you said it was an old test and you probably don't have these numbers, but just throwing it out there.

I went searching through the hard drive and found the original file. I did this almost exactly 1 year ago - which means it was with OOTP16. Not sure if anything changed or was tweaked in OOTP17.

There were 10,603 players.

Yes, it would be interesting to see if any of the traits are correlated. So, did as you suggested and assigned numeric values to each trait and applied Excel's CORREL function. Here are the results:

_____LEA__ LOY........WIN__ GRE__ WE__ INT
LEA....1.000....0.150...-0.138....-0.140...0.003...0.002
LOY....0.150...1.000....-0.155....-0.242...0.101...0.006
WIN.. -0.138...-0.155...1.000.....0.210....0.043..-0.006
GRE..-0.140...-0.242...0.210.....1.000...-0.171..-0.009
WE.....0.003...0.101....0.043.....-0.171...1.000....0.131
INT.....0.002...0.006....-0.006....-0.009....0.131...1.000

There are some correlations. Greed and Loyalty are negatively correlated, which makes sense. Work Ethic is negatively correlated with greed and mildly correlated with intelligence and loyalty. Desire to win is correlated with greed (that kinda makes sense too).

I think it would also be interesting to see how the traits might combine with each other. Would the positive effect of high work ethic counteract a negative effect of low desire to win? Or would one take precedence over the other? I might take a look at that some day.

There were a couple interesting things I noticed when I looked at the data again. Just eyeballing it, Leadership, Intelligence, and Loyalty appear to be normally distributed, each with about the same standard deviation. Greed looks to be normally distributed, but with a larger standard deviation than the others. Work Ethic and Desire to Win are definitely not normally distributed. The are both skewed so that there are considerably more players with "Very Low" for these traits than "Very High" (about 4 times as many).

Also, I hadn't remembered that I did include potential rating for all of the players (with scouting accuracy at 100%). It's a crude and as many have pointed out, flawed, measure of talent or skill, but despite this, it was far better at predicting who made it to the major leagues than personality. I graphed this in Excel and came up with a trend line. The best fit was a logarithmic with this formula:

y = 0.623* LN(x) - 1.8178

x is the potential rating (20-80 scale) and y is the probability of making it to the big leagues.

Plug 80 into this formula and you get 0.912
Plug in 20 and you get 0.049

So an 80 potential player makes it over 90% of the time, whereas a 20 potential player only makes it about 5% of the time. Of course, it won't quite be so clear-cut when you don't have scouting accuracy set at 100%...

Another thing I included was age. This one looks like a step function. College age players (20 and older) make it to the majors more often than high school players (19 and younger)

And so there you have it. Put it all together and ya got yourself a draft strategy!

[drhay53]

Thanks for quickly addressing my questions! Now here's an interesting thought. Can you add potential rating to the correlation matrix with the personality traits? I have always wondered if the game considered this in the potential rating, and you already have all of this data lying around

To my original point, I don't think anything in the correlation matrix explains why the highest desire for win players have a decreased rate. It would seem odd to me if that were true, but maybe they get unhappy too much in bad minor league teams and end up not developing or being traded around. Who knows.

[Cobby]

Originally Posted by drhay53

To my original point, I don't think anything in the correlation matrix explains why the highest desire for win players have a decreased rate. It would seem odd to me if that were true, but maybe they get unhappy too much in bad minor league teams and end up not developing or being traded around. Who knows.

Yes I don't see anything in the correlation matrix either that explains it (although the little boost and dip for Intelligence looks like it could be completely explained by its correlation with work ethic). As for the "desire to win" dip for "very high" - maybe it's real, but maybe it's just due to a smaller sample. Just 677 of the 10,603 players had Very High desire to win. I'm willing to put it down to that.

Originally Posted by drhay53

Can you add potential rating to the correlation matrix with the personality traits? I have always wondered if the game considered this in the potential rating, and you already have all of this data lying around

This is an interesting question. Here are the results:

..............POT
LEA....-0.040
LOY....-0.026
WIN.....0.176
GRE....0.014
WE......0.179
INT.......0.011
AGE.....0.131

The answer appears to be "Yes" (at least to some extent) for Work Ethic and Desire to WIn (and Age too, I thought I'd throw that in there). "No" for the others.

Even before I looked at this, I was beginning to come to the conclusion that the only personality traits that that really matter for player development were Work Ethic and Desire to Win. (and not Intelligence, as others seem to think intuitively). The fact that they are ones that correlate the most with POT seems to back up that conclusion.

It's not a huge correlation, but it might be enough that you don't really need to consider personality separately.

My own main takeaway from this is that "Desire to Win" is important, and "Intelligence", not so much. "Work Ethic" is important too, but most people already knew that. But, talent or potential matters the most...

[baseballonly]

I have always written off work ethic, but....that might change my mind.

[injury log]

This is interesting work, I like studies like this. But if you'd done your studies in a different sequence, I think you might have reframed your conclusions.

- if first you had found that initial Potential is the best predictor of future success

- then you found that initial Potential was correlated with Desire for Winner and Work Ethic in your sample

and you then formulated a hypothesis about whether Work Ethic and Desire for Winner would be correlated with future success, naturally you'd hypothesize they will be, in your sample, because they're correlated with initial Potential. So that's not a surprising finding, and doesn't say anything about whether Work Ethic or Desire for Winner contribute to future success (independent of their correlation with Potential).

What I think you'd want to do is do a multiple regression where you regress on the log of initial Potential and on WE and Desire for Win (or some transformation of those), and check your p-values to see if WE or Desire for Win have any provable predictive value independent of Potential.

Based on what I know of the game, Desire for Winner shouldn't be part of the player development model at all (though Work Ethic is), which is what makes me think this correlation between Desire for Win and Potential is the only explanation for your findings (unless there's some bug in the dev code no one has noticed).

[jpeters1734]

I think your personality study has a slight flaw. You don't take into account the ratio of initial creation. Meaning, what percentage of players were created with a high WE? If 19% of all amateurs are created with a high WE and 10 years later 19% of those amateurs made the ML, you can conclude that high WE has no effect at all.

The question shouldn't be how many ML players are there with high WE. It should be, what percent of players created with high WE make the ML, and what percentage of players created with a low WE make it.

[injury log]

Originally Posted by jpeters1734

I think your personality study has a slight flaw. You don't take into account the ratio of initial creation. Meaning, what percentage of players were created with a high WE? If 19% of all amateurs are created with a high WE and 10 years later 19% of those amateurs made the ML, you can conclude that high WE has no effect at all.

The question shouldn't be how many ML players are there with high WE. It should be, what percent of players created with high WE make the ML, and what percentage of players created with a low WE make it.

I think you've misunderstood his data. When he says, for example, "Intelligence: Very Low: 13.2%", he is saying that 13.2% of players with Very Low Intelligence made the Majors.

[Cobby]

Originally Posted by injury log

I think you've misunderstood his data. When he says, for example, "Intelligence: Very Low: 13.2%", he is saying that 13.2% of players with Very Low Intelligence made the Majors.

Yes, this is correct.

[Cobby]

Originally Posted by injury log

This is interesting work, I like studies like this. But if you'd done your studies in a different sequence, I think you might have reframed your conclusions.

- if first you had found that initial Potential is the best predictor of future success

- then you found that initial Potential was correlated with Desire for Winner and Work Ethic in your sample

and you then formulated a hypothesis about whether Work Ethic and Desire for Winner would be correlated with future success, naturally you'd hypothesize they will be, in your sample, because they're correlated with initial Potential. So that's not a surprising finding, and doesn't say anything about whether Work Ethic or Desire for Winner contribute to future success (independent of their correlation with Potential).

What I think you'd want to do is do a multiple regression where you regress on the log of initial Potential and on WE and Desire for Win (or some transformation of those), and check your p-values to see if WE or Desire for Win have any provable predictive value independent of Potential.

Based on what I know of the game, Desire for Winner shouldn't be part of the player development model at all (though Work Ethic is), which is what makes me think this correlation between Desire for Win and Potential is the only explanation for your findings (unless there's some bug in the dev code no one has noticed).

Yes, I agree at the moment that we don't really know for sure if WE and WIN really bring anything more to the table than just POT alone.

But, I did learn something interesting today by playing around in the editor. I can't get the potential rating to change all by editing the personality traits. That indicates to me that the program does not incorporate personality when calculating potential (contrary to what I was thinking earlier). So. In order to explain the correlation between WIN and POT and WE and POT, it has to be that this happens during player creation. In other words there's probably a slightly greater chance that better players get assigned these positive personality traits than worse players (or vice versa - more chance for worse players to get the negative versions of these traits) (or both). From then on though, whether or not the development engine uses them is an open question.

I do think the answer is probably yes. The reason is that about half of all players are created with a POT of 20. This allows me to examine a large enough sample of players in my dataset (5290 to be exact) all having the same POT to see if there are any statistically significant differences in their chances of reaching the majors based on personality. (I used the cumulative binomial distribution for the significance test)

Using p= 0.05 as the test of significance, here are the traits that mattered for players with POT = 20:

WE..............Very Low...p = 0.046
WE.............. High.........p = 0.043
Greed.......... Low..........p = 0.014 (hmmmm?)
Leadership..Very High..p = 0.018 (hmmmm?)

A few others were close to the significance line and I think a larger sample size might have pushed them over the limit.

These include INT Very High at 0.062, and WIN High at 0.067

But the point is, these differences in development are with players of exactly the same overall Potential, so we can rule out correlation with potential as the cause in these cases at least.

[jpeters1734]

Originally Posted by injury log

I think you've misunderstood his data. When he says, for example, "Intelligence: Very Low: 13.2%", he is saying that 13.2% of players with Very Low Intelligence made the Majors.

Yes I understand his data. Perhaps you didn't understand my point. If the game only creates 13.2% of players with very low int in the first place and 13.2% make the majors, then you can't conclude anything from it