Theorems, Formulas, and Evaluations

[injury log]

Originally Posted by jpeters1734

Yes I understand his data. Perhaps you didn't understand my point. If the game only creates 13.2% of players with very low int in the first place and 13.2% make the majors, then you can't conclude anything from it

I think what you mean to say is: if 13.2% were created 'very low', and 13.2% of Major Leaguers were 'very low', that would not tell you anything. That is true (or at least it would tell you that a 'very low' rating doesn't matter much). I understand the point you're trying to make perfectly.

But I'll say again: you have not correctly understood the meaning of Cobby's data. He is not saying "13.2% of Major Leaguers have a 'very low' rating". He is saying "13.2% of people created with a 'very low' rating make the Majors".

[jpeters1734]

my example literally says "make the majors". Regardless, we need the initial creation numbers to get any value from it

[injury log]

Originally Posted by Cobby

Using p= 0.05 as the test of significance, here are the traits that mattered for players with POT = 20:

WE..............Very Low...p = 0.046
WE.............. High.........p = 0.043
Greed.......... Low..........p = 0.014 (hmmmm?)
Leadership..Very High..p = 0.018 (hmmmm?)

A few others were close to the significance line and I think a larger sample size might have pushed them over the limit.

These include INT Very High at 0.062, and WIN High at 0.067

But the point is, these differences in development are with players of exactly the same overall Potential, so we can rule out correlation with potential as the cause in these cases at least.

I'd caution against reaching that conclusion, for two reasons:

- if you look at 30 sets of data that have no correlation with anything, and use a 0.05 significance level to judge if they're correlated with something, you'd naturally expect 5% of your sets of data (so one or two of them) would seem significant, just because of random variance and the definition of a p-value. Finding spurious correlations in that way is what's called "p-hacking". That's very likely the explanation for at least some of your findings, since there's no logical reason they should be true otherwise - for example, there's no logical reason a "Low" Greed rating would be beneficial for development, but a "Very Low" Greed rating would not be. If the extremes of a rating scale are not more obviously beneficial than the non-extremes, that's at least a reason to question the importance of that rating;

- probably more importantly, you pointed out earlier that certain personality ratings are correlated with Potential in your sample. If that is a genuine correlation, it probably persists when you remove the top players from the sample. That is, among the 20 Potential guys, the players with good Personality may be better, in general, than the guys with bad Personality. Guys with 20 Potential are not all alike; some might be rated 50/50/50 and some 40/40/40. The 50/50/50 guy is more likely to reach the Majors. And if that is true, your results might only be indicating which players had the higher initial potential, not how personality ratings influence development after player creation.

Because of that latter issue, I think the correct way to investigate whether Personality ratings have an influence independent of correlations with potential ratings is to do a multiple regression only on the > 20 POT group, with log(potential) and personality ratings as your regressors. Excel will do that for you easily and will give you p-values for the coefficients for each personality variable, and that would tell you if they have an influence independent of Potential.

[injury log]

Originally Posted by jpeters1734

my example literally says "make the majors". Regardless, we need the initial creation numbers to get any value from it

This is not true, as you'll see immediately by imagining a simpler example. If I do a test on a huge sample of players, and find that 10% of players from Tuvalu make the Majors, it is completely irrelevant whether 10% of players were originally from Tuvalu, or 0.1%, or 50%. My best estimate of the probability of a Tuvaluan making the Majors is 10%. That probability has absolutely nothing to do with the proportion of players who are from Tuvalu.

[jpeters1734]

Originally Posted by injury log

This is not true, as you'll see immediately by imagining a simpler example. If I do a test on a huge sample of players, and find that 10% of players from Tuvalu make the Majors, it is completely irrelevant whether 10% of players were originally from Tuvalu, or 0.1%, or 50%. My best estimate of the probability of a Tuvaluan making the Majors is 10%. That probability has absolutely nothing to do with the proportion of players who are from Tuvalu.

I really don't understand how and why you've come to that conclusion.

If the 2017 draft has 10% of players with a very high WE and 10 years later out of the players that made the ML from that draft, 20% have very high WE, you can conclude that WE does have an effect on development.

But if after 10 years, only 10% of those players that made it have very high WE, then you can conclude that WE has no effect at all.

[jpeters1734]

Also if .1% of players start from Tuvalu but 10% of all major league players are from Tuvalu, you can bet your ass I'm signing a player from Tuvalu since there's a great chance that player will make the ML

[injury log]

These two sentences mean something very different:

#1: 10% of players from Tuvalu make the Majors

#2: 10% of Major League players are from Tuvalu

Every time someone has said a sentence like #1, you have mistakenly thought they were saying something like sentence #2. No one (besides you) has once said something like #2 in this thread. If you understand the distinction between #1 and #2 and go back and read over the previous posts, what Cobby and I have been saying will make sense to you.

[Cobby]

Originally Posted by injury log

Because of that latter issue, I think the correct way to investigate whether Personality ratings have an influence independent of correlations with potential ratings is to do a multiple regression only on the > 20 POT group, with log(potential) and personality ratings as your regressors. Excel will do that for you easily and will give you p-values for the coefficients for each personality variable, and that would tell you if they have an influence independent of Potential.

OK! Now we're getting somewhere! This is my first foray into multiple regression analysis - but I think I've got it figured out. I did it twice, once on the whole dataset and once discarding the POT 20 players as you suggested. The only overall difference between the two was that AGE was significant when you discarded the POT 20 players and wasn't when you included them (for some reason). Here are the p values for the POT=20 excluded case:

...............P-value
Age.........0.014
LEA.........0.027
LOY.........0.865
WIN.........0.698
GRE........0.880
WE..........0.022
INT...........0.116
LN(POT)..0.000

The big surprise is that Leadership is significant. Age and Work Ethic too, but not the other personality traits. (although Intelligence looks like it's trying hard....)

The R^2 was .339 and the ANOVA significance F was 0. I think that means that we've got some predictive value here....

So. I suppose at some point, I'll toss out the irrelevant personality traits, run it again and use results to come up with an equation.

Then we'll have a formula for evaluating draftees and this long seemingly OT diversion will finally come back around to answering the OP's original question...

[injury log]

Interesting work, thanks for running that. Potential is so strongly predictive that your model stats (F stat) will automatically look great no matter what other variables you include. One result I find reassuring is that there's no evidence Desire for Winner is meaningful - if it were, that would contradict everything I know about the player development engine.

As you suggest, the best thing to do next would be to discard the variables that don't appear to be relevant to get a simpler model, and see what the regression equation looks like. If you were writing this up for a stats journal or something like that, there are quite a few other things you'd want to do to confirm the model is sound, but I'm guessing you probably don't want to go to all that trouble! But if it's correct to assume that your "make the Majors" probability is linearly affected by Work Ethic and other personality traits, then the regression coefficient for Work Ethic will tell you how much the probability of making the Majors increases when you increase Work Ethic by one point (on the 1-5 scale I think you're using), if everything else is held constant. It will be interesting to see how big those coefficients are.

[baseballonly]

Originally Posted by Cobby

OK! Now we're getting somewhere! This is my first foray into multiple regression analysis - but I think I've got it figured out. I did it twice, once on the whole dataset and once discarding the POT 20 players as you suggested. The only overall difference between the two was that AGE was significant when you discarded the POT 20 players and wasn't when you included them (for some reason). Here are the p values for the POT=20 excluded case:

...............P-value
Age.........0.014
LEA.........0.027
LOY.........0.865
WIN.........0.698
GRE........0.880
WE..........0.022
INT...........0.116
LN(POT)..0.000

The big surprise is that Leadership is significant. Age and Work Ethic too, but not the other personality traits. (although Intelligence looks like it's trying hard....)

The R^2 was .339 and the ANOVA significance F was 0. I think that means that we've got some predictive value here....

So. I suppose at some point, I'll toss out the irrelevant personality traits, run it again and use results to come up with an equation.

Then we'll have a formula for evaluating draftees and this long seemingly OT diversion will finally come back around to answering the OP's original question...

I'm not understanding this...why are you comparing personalities with potential ratings? Shouldn't you be comparing them with whether or not the players actually reach their potential? Or am I missing something?

[Cobby]

Originally Posted by baseballonly

I'm not understanding this...why are you comparing personalities with potential ratings? Shouldn't you be comparing them with whether or not the players actually reach their potential? Or am I missing something?

Well, the thing we're trying to understand is what factors contribute to a player's chance of making the major leagues. (if I ever did this again I'd probably use Career WAR or something like that as the dependent variable)

My original data is a big table of 10,000+ simulated players with their age at creation, their personality traits, their original Overall potential rating and a bit - yes or no - indicating whether the player eventually made it to the major leagues.

Originally, I just reported the basic results for each personality category (see my first post on this above). They show the percent of players for each of the categories that made the major leagues (keeping in mind that 14.5% of players overall made it). This original table was intriguing, especially since it seemed to show that WIN (desire to win) had a prominent effect on percent making the majors.

But there are a lot of reasons why this could be misleading. So with guidance from drhay53 and especially from injury log, we began to explore this further.

We were trying to figure what really has predictive value and what doesn't. That's what the latest results from the multiple regression are telling us. The p (p stands for probability) values show how confident we are that each of these factors makes an independent contribution toward predicting whether or not a player makes it to the majors. The lower the number, the more confident we are that this is the case.

So yes, in a way we are comparing these to POT because POT is such a powerful predictor of whether or not a player makes the majors that maybe none of the others make any real difference. The table of p values above indicate that there are some personality traits that make a difference independent of POT.

When I get time to work on this a little more, we'll be able to come up with an equation that will give an idea of how much these traits matter. Even though things like work ethic have a contribution independent of POT it may be that the contribution is small enough to be ignored. Or not. That will be the next step.

[Cobby]

OK...(drum roll please).... here are the equations:

For players with POT > 20, the relevant factors are POT, AGE, WE, LEA and INJ.

POT is the overall potential on a 21-80 scale
AGE in my sample ranged from 16 to 25
WE is work ethic score on a scale from 0 to 4, 0 = Very Low, 1= Low, 2 = Normal, 3 = High and 4 = Very High
LEA is leadership on the same scale as WE
INJ is the injury-proneness on a scale of 1 to 3. 1 = Fragile, 2=Normal, 3= Durable (none of the players are created with Wrecked, or Iron Man ratings)

The formula is:

ML% = 0.6196 * LN(POT) + .0061 * AGE + .0113 * LEA + .0125 * WE + 0.0274 * INJ - 2.0454

If I set POT to 42.2 and all of the others to the middle of their ranges, we get a player with a 50-50 chance of making the majors.

If I then vary age from 16 to 25, the probability goes from 47.3 to 52.7
If I vary leadership from 0 to 4, the range is 47.7 to 52.3
Varying WE, we get 47.5 to 52.5
Varying INJ, we get 47.3 to 52.7

That gives you an idea of the magnitude of the effect of these; roughly +/- 2.7% chance of making the majors for each.

Going back to neutral, and varying POT, this range is equivalent to roughly 40.5 to 44 - so somewhere in the neighborhood of plus/minus 2 points worth of potential. So if you didn't want to export into a spreadsheet and do calculations, your rule of thumb could be to first order the players by potential then add one point for High and 2 points for Very High each on Leadership and work ethic, and subtract equivalently for Low and Very Low. Then give two points for Durable and subtract 2 points for Fragile. And then, give an adjustment for AGE, with 20.5 as the midpoint.

If I set all of factors to their min and max settings, the probability ranges from 39.7% to 60.2% - that's quite a difference for two players with the same potential.

For players with POT = 20, the two relevant factors are Age and WE. (but age is backwards from above, younger is better for these players). The formula for these players would be:

ML% = 0.0058 * WE - 0.00237 * AGE + 0.06975

This gives values from about 1% to 5.5%. So when you get to the point in the draft you could just start sort by AGE and draft the young High WE players, or sort by WE and draft the youngest ones first.

The biggest surprise for me in all of this is the effect of Leadership. The manual does say that high leadership players have an effect on the development of other players around them. If that's the case, then the value of leadership may even be understated by this study. I think I'm going to start paying more attention to this and try to have high leadership players hanging around at all levels of my organization just for all those positive vibes they seem to be emitting...

[drhay53]

A lot of very interesting stuff has gone on in here the last few days. Awesome!

I just wanted to point out something about intelligence. I've always had it in my head that intelligence mainly affects the rate at which players gain experience playing different positions. It is interesting though that one of the personality blurbs from your scout is "slow to absorb instruction from coaches". That to me implies that intelligence *should* affect development.

Another random thought is that I was surprised your R^2 is only 0.339. The model is really not explaining very much of the variance in the data. In other words, there's still just a ton of randomness in whether or not a player makes it to the majors. I rather like that, to be honest. It makes the game harder. Still, that explained variance could really help you get an advantage over someone who wasn't accounting for it at all in their drafting.

[baseballonly]

Nice work, but you should now incorporate position and individual ratings. Are hitters more likely to make it than pitchers? Defensive guys or all-offense? Does stuff develop more often than control? That would be quite interesting and useful.

[Cobby]

Originally Posted by baseballonly

Nice work, but you should now incorporate position and individual ratings. Are hitters more likely to make it than pitchers? Defensive guys or all-offense? Does stuff develop more often than control? That would be quite interesting and useful.

Thanks. Yes, I agree all of that would be very interesting.

It also would be interesting to find out if there is more uncertainty in projecting pitchers vs. position players. Maybe pitchers make the majors at about the same rate, but it's harder to tell ahead of time which ones.

Also it would be interesting to look at longevity. Maybe high stuff pitchers don't last as long once they do make it. (I suspect they probably flame out earlier than control pitchers...)

But. At the moment I'm having too much fun playing the game to start another study. Perhaps the spirit will move me at some point...

[Cobby]

Originally Posted by drhay53

A lot of very interesting stuff has gone on in here the last few days. Awesome!

I just wanted to point out something about intelligence. I've always had it in my head that intelligence mainly affects the rate at which players gain experience playing different positions. It is interesting though that one of the personality blurbs from your scout is "slow to absorb instruction from coaches". That to me implies that intelligence *should* affect development.

Yeah. The p value for Intelligence was fairly low compared to some of the others that were ruled out; 0.11, I think. It doesn't meet the p < 0.05 criterion for statistical significance, but that doesn't mean that it definitely doesn't affect development either. I think it's more likely that it does than doesn't, but who knows?